Verified Rigor RecordRigor Reviewed

Rigor-Reviewed Manuscript

Journal of Open Source Software (JOSS): design and first-year review

Name: Verified Rigor Record — Journal of Open Source Software (JOSS): design and first-year review
Item: Journal of Open Source Software (JOSS): design and first-year review
Rating: 100
Author: Αlpha¹

Paper source

Open DOI

Αlpha¹ Editorial

This rigor review was examined and confirmed by Αlpha¹ Editorial · July 5, 2026

Αlpha¹ Rigor Review

Rigor Reviewed

PASS

Rigor holds up · Other Study Type

Eight dimensions — click a segment for details

Critical

High-priority

Model-reported confidence: highReviewer agreement: highEngine: 7.5.0Source: full text

Not verified in this report

No reported statistical tests were found to recompute.

This Αlpha¹ Rigor Review uses AI Rigor Reviewers trained on a curated corpus of high-fidelity and retracted papers, with expert supervision and curation. It can still make mistakes; verify each finding against the source before relying on it.

The paper is a descriptive account of the Journal of Open Source Software (JOSS) and its first-year operations, not a hypothesis-testing study. It is methodologically sound for its purpose: the motivation is well-supported, key resources (software tools) are identified, data and code are publicly available, and reporting is transparent. No inferential statistics or biological/ethical concerns apply.

Three independent reviewers evaluated all eight dimensions; they converged on most dimensions. A minor divergence occurred on key resources: two reviewers considered it not applicable, but the dimension's checklist includes software tools identified, which is adequately reported, so pass was assigned. Verification components checked citations (18 references not found in registry, none retracted) and reproducibility (1 dead link).

12 major claims checked against the paper's own evidence: all adequately supported.

Rigor

Eight Dimensions3 pass

12 major claims1 partial11 supported

The introduction and background sections cite surveys and literature establishing that software is critical to research but lacks scholarly credit. It discusses challenges of publishing software and acknowledges prior attempts (e.g., other software journals). The premise that a new journal could improve software quality and provide career credit is logically developed from these cited sources.

Sub-criteria

prior work citedADEQUATE

premise rationaleADEQUATE

limitations addressedADEQUATE

Per reviewer (3) — the votes behind the verdict

PASSReviewer 1· 95% conf

The paper clearly establishes the need for a software journal, cites relevant prior work on software citation and publishing, and explains how JOSS addresses limitations in existing mechanisms.

Evidence

direct quote[Background and Motivation, paragraph 1]

“A 2014 study of UK Russell Group Universities ... reports that ∼90% of academics surveyed said they use software in their research, while more than 70% said their research would be impractical without it.”

direct quote[Introduction, paragraph 1]

“The Journal of Open Source Software (JOSS) was founded in May 2016 to offer a solution within the existing publishing mechanisms of science.”

direct quote[Challenges of publishing software]

“Despite being a critical part of modern research, software lacks support across the scholarly ecosystem for its publication, acknowledgement, and citation.”

PASSReviewer 2· 95% conf

The scientific premise is well-established with ample citations and a clear rationale for JOSS.

Evidence

direct quote[Background and Motivation]

“A 2014 study of UK Russell Group Universities reports that ~90% of academics surveyed said they use software in their research... Despite being a critical part of modern research, software lacks support across the scholarly ecosystem for its publication, acknowledgement, and citation.”

direct quote[The Journal of Open Source Software, Goals and principles]

“To tackle the challenges mentioned above, the Journal of Open Source Software (JOSS) launched in May 2016 with the goal of drastically reducing the overhead of publishing software articles.”

direct quote[Background and Motivation, Challenges of publishing software]

“Unfortunately, software citation is in its infancy... Even when software (and data) are published so that they can be cited, we do not have a standard culture of peer review for them.”

PASSReviewer 3· 95% conf

The paper clearly establishes the scientific foundation by citing prior research on software use and citation, and provides a logical rationale for creating JOSS.

Evidence

direct quote[Introduction, paragraph 1 and Background and Motivation section]

“A 2014 study of UK Russell Group Universities (Hettrick et al., 2014) reports that ∼90% of academics surveyed said they use software in their research, while more than 70% said their research would be impractical without it.”

direct quote[Introduction, paragraph 2]

“The Journal of Open Source Software (JOSS) was founded in May 2016 to offer a solution within the existing publishing mechanisms of science.”

direct quote[Background and Motivation, Challenges of publishing software]

“Even when software (and data) are published so that they can be cited, we do not have a standard culture of peer review for them.”

The paper describes the JOSS journal's review process, infrastructure, and first-year publication statistics. There is no randomized comparison, no control group, no intervention allocation, and no formal hypothesis testing. The sub-criteria for study design (randomization, blinding, power analysis, etc.) are not applicable to this type of manuscript.

Sub-criteria

controlsN/A

power analysisN/A

blinding levelsN/A

outlier handlingN/A

randomization unitN/A

inclusion exclusionN/A

randomization methodN/A

replicate distinctionN/A

independent replicationN/A

Per reviewer (3) — the votes behind the verdict

N/AReviewer 1· 100% conf

This paper does not involve an experimental or observational study design; it is a descriptive report on a journal's operation and first-year statistics. Therefore, standard study design rigor criteria do not apply.

Evidence

absence[Methods/Results are not structured as a formal study]

No experimental or observational study design is present; the paper is a descriptive account.

N/AReviewer 2· 95% conf

No experimental or quasi-experimental study design is present; the paper is a descriptive overview of a journal platform.

N/AReviewer 3· 100% conf

The paper does not describe a study design with experimental elements; it is a descriptive report of a journal's operations.

Evidence

absence[Entire paper]

No study design elements are present.

All sub-criteria (sex, age, species, housing, demographics) are irrelevant because the study involves no biological systems or human participants. The analysis concerns publication statistics of a journal.

Sub-criteria

demographicsN/A

sex reportedN/A

sex justifiedN/A

age weight healthN/A

housing conditionsN/A

species strain sourceN/A

Per reviewer (3) — the votes behind the verdict

N/AReviewer 1· 100% conf

No biological organisms, cell lines, or human subjects are involved. The paper discusses software and journal infrastructure only.

Evidence

absence[Entire paper]

No biological variables are reported or applicable.

N/AReviewer 2· 95% conf

No biological variables (e.g., sex, age, species) are relevant; the study involves no live subjects or cell lines.

N/AReviewer 3· 100% conf

The paper does not involve biological organisms, cell lines, or human subjects, so biological variables are not applicable.

Evidence

absence[Entire paper]

No biological variables are reported.

The study is a descriptive analysis of JOSS's first year, using data from the journal's own records. No new data were collected from human participants or animals. The data availability statement provides public access to the data. Therefore, all ethics sub-criteria are not applicable.

Sub-criteria

iacuc statementN/A

informed consentN/A

irb ethics statementN/A

regulatory complianceN/A

Per reviewer (3) — the votes behind the verdict

N/AReviewer 1· 100% conf

The paper does not involve human subjects, animal subjects, or any non-public identifiable data; it describes a journal's operation and uses only publicly available data (publication records). No ethics approval is required.

Evidence

absence[Entire paper]

No human or animal subjects research is reported.

N/AReviewer 2· 95% conf

No ethics approvals needed because the paper does not involve human subjects, animals, or non-public identifiable data.

N/AReviewer 3· 100% conf

The paper does not involve human or animal subjects, so ethical approvals are not applicable.

Evidence

absence[Entire paper]

No ethics statement is present.

The paper describes a software journal and does not involve biological or chemical resources. The software tools mentioned (GitHub, Zenodo, etc.) are not considered key biological/chemical resources under this dimension.

Sub-criteria

mycoplasma testingN/A

reagents identifiedN/A

organisms identifiedN/A

antibodies identifiedN/A

cell line authenticationN/A

software tools identifiedN/A

Per reviewer (3) — the votes behind the verdict

PASSReviewer 1· 90% conf

The paper identifies the software tools and infrastructure used for JOSS (GitHub, Zenodo, Figshare, Ruby on Rails, Whedon) with adequate detail. No biological/chemical resources are used, so those criteria are not applicable.

Evidence

direct quote[How JOSS works]

“The JOSS web application and submission tool is hosted at http://joss.theoj.org. It is a simple Ruby on Rails web application”

direct quote[Peer review in JOSS, step 4]

“the editor asked the submitting author to make a permanent archive of the software ... with a service such as Zenodo or Figshare”

N/AReviewer 2· 95% conf

No key biological or chemical resources (antibodies, cell lines, reagents) are used.

N/AReviewer 3· 100% conf

The paper does not use antibodies, cell lines, organisms, or traditional reagents; it is not a clinical trial with an investigational product.

Evidence

absence[Entire paper]

No biological or chemical resources are used.

The paper presents counts, means, medians, and percentages of submission data, but does not conduct hypothesis tests, effect size estimation, or confidence intervals. Therefore, the statistical analysis dimension is not applicable.

Sub-criteria

tests namedN/A

exact p valuesN/A

effect sizes ciN/A

data presentationN/A

software identifiedN/A

assumptions verifiedN/A

mathematical plausibilityN/A

Per reviewer (3) — the votes behind the verdict

N/AReviewer 1· 85% conf

The paper presents only descriptive statistics (counts, proportions, means) from the journal's first-year data. No inferential statistical tests are performed. Therefore, most sub-criteria for statistical analysis are not applicable.

Evidence

direct quote[A Review of the First Year]

“articles spent on average 45.5 days between submission and publication (median 32 days, interquartile range 52.3 days)”

direct quote[Figure captions]

“Data, plotting script, and figure files available ()”

N/AReviewer 2· 95% conf

No inferential statistical tests are performed; the paper reports only descriptive statistics.

N/AReviewer 3· 95% conf

The paper reports only descriptive statistics and does not use inferential statistics, so statistical analysis criteria are not applicable.

Evidence

absence[Entire paper]

No inferential statistics are reported.

The paper states: 'The following information was supplied regarding data availability: Niemeyer, Kyle (2017): JOSS first-year publication data and figures. figshare. https://doi.org/10.6084/m9.figshare.5147722. Source code of JOSS, on GitHub: https://github.com/openjournals/joss.' The data are deposited in a public repository with a persistent identifier. The code for the journal platform is in a public repository. The plotting scripts are included in the Figshare archive. No sequencing data or clinical data access procedures are applicable. All three applicable criteria (data statement, repository deposit, code sharing) are met adequately.

Sub-criteria

code sharingADEQUATE

accession numbersN/A

repository depositADEQUATE

data availability statementADEQUATE

Per reviewer (3) — the votes behind the verdict

PASSReviewer 1· 95% conf

A clear data availability statement provides access to the first-year data via Figshare with DOI. The source code for JOSS is available on GitHub. These meet the criteria for adequate data and code sharing.

Evidence

direct quote[Data Availability Statement]

“Niemeyer, Kyle (2017): JOSS first-year publication data and figures. figshare. https://doi.org/10.6084/m9.figshare.5147722”

direct quote[Data Availability Statement]

“Source code of JOSS, on GitHub: https://github.com/openjournals/joss”

PASSReviewer 2· 95% conf

Data and code are available with clear accessibility routes: data on figshare, code on GitHub.

Evidence

direct quote[Data Availability Statement]

“The following information was supplied regarding data availability: Niemeyer, Kyle (2017): JOSS first-year publication data and figures. figshare. https://doi.org/10.6084/m9.figshare.5147722.”

direct quote[Data Availability Statement]

“Source code of JOSS, on GitHub: https://github.com/openjournals/joss.”

PASSReviewer 3· 95% conf

The paper provides a concrete data availability statement with a Figshare DOI and GitHub repository, meeting all applicable criteria.

Evidence

direct quote[Data Availability Statement after the main text]

“Niemeyer, Kyle (2017): JOSS first-year publication data and figures. figshare. https://doi.org/10.6084/m9.figshare.5147722 . Source code of JOSS, on GitHub: https://github.com/openjournals/joss .”

The paper describes the JOSS process in sufficient detail for understanding, discusses limitations (e.g., not fully indexed by Google Scholar), draws conclusions commensurate with the evidence, and declares funding sources and competing interests. Trial registration and reporting guideline are not applicable.

Sub-criteria

funding coiADEQUATE

trial registrationN/A

reporting guidelineN/A

methods completenessADEQUATE

all outcomes reportedADEQUATE

limitations discussedADEQUATE

conclusions proportionalADEQUATE

Per reviewer (3) — the votes behind the verdict

PASSReviewer 1· 90% conf

The paper is transparently reported: methods for the journal process are detailed, limitations are discussed (e.g., reviewer delays, indexing issues), conclusions are proportional to the descriptive data, and funding/competing interests are declared.

Evidence

direct quote[Peer review in JOSS, Section 'JOSS review criteria']

“The specific items in the reviewer checklist are: • Conflict of interest ... • Code of Conduct ... • General checks ... • Functionality ... • Documentation ... • Software paper”

direct quote[Conclusions, paragraph 2]

“JOSS is currently not fully indexed by Google Scholar, despite the fact that JOSS articles include adequate metadata and that we made an explicit request for inclusion in March 2017”

direct quote[Funding Statement and Additional Information and Declarations]

“This work was supported in part by the Alfred P. Sloan Foundation. ... The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

PASSReviewer 2· 95% conf

Reporting is transparent: methods are detailed, limitations discussed, conclusions proportional, and funding/COI provided.

Evidence

direct quote[Peer review in JOSS]

“The JOSS process shows a typical JOSS submission and review process, described here in more detail using the hdbscan package as an example:”

direct quote[Conclusions]

“JOSS is currently not fully indexed by Google Scholar, despite the fact that JOSS articles include adequate metadata and that we made an explicit request for inclusion in March 2017...”

direct quote[Funding Statement and Competing Interests]

“This work was supported in part by the Alfred P. Sloan Foundation... Competing Interests: Daniel S. Katz is an Academic Editor for PeerJ CS.”

PASSReviewer 3· 95% conf

The paper is transparent and complete, with detailed methods, limitations, funding, and competing interests reported.

Evidence

direct quote[Peer review in JOSS section]

“The JOSS process shows a typical JOSS submission and review process, described here in more detail using the hdbscan package (McInnes, Healy & Astels, 2017) as an example:”

direct quote[Conclusions, paragraph 2]

“JOSS is currently not fully indexed by Google Scholar, despite the fact that JOSS articles include adequate metadata and that we made an explicit request for inclusion in March 2017 (see GitHub issue #130).”

direct quote[Funding Statement and Additional Information and Declarations]

“This work was supported in part by the Alfred P. Sloan Foundation.”

5 copyedit issues flagged: mostly consistency, clarity, other.

Checked 38 references: 20 verified — 18 unresolved.

Reproducibility

8 data/code links checked; 7 live, 1 dead.

Registration stated in text, but no registry ID was detected. No reporting guideline cited.

Αlpha¹ Rigor Action ItemsFinal pass before journal submission

The published paper is robust for its descriptive purpose, but an informed reader should weigh the 18 references that could not be verified in citation registries and the one dead link reported by the reproducibility check. These do not undermine the core descriptive findings but warrant correction via an erratum or independent re-verification of the cited sources and link.

1.
HIGHdata code
Fix or replace the dead link found in the reproducibility check (1 of 8 checked links was dead).
A dead link undermines the data/code availability statement and may prevent readers from accessing supporting materials.
2.
HIGHreporting
Verify or correct each of the 18 references flagged as 'not found in registry' (e.g., 'In which journals should I publish my software?', 'Contributor covenant code of conduct', 'Hettrick et al. (2014)', and others listed in the citation verification).
Unresolved references may be fabricated or contain errors; authors should confirm they exist and are correctly cited, and consider supplying DOIs where missing.
3.
MEDIUMreporting
In the 'Review of the First Year' section, explicitly name the software used to generate figures and descriptive statistics (e.g., 'R version X with ggplot2' or 'Python with matplotlib').
Even for descriptive analyses, naming the analysis software enhances reproducibility and is considered best practice.
4.
MEDIUMcopyedit
Add a period at the end of the Figure 2 caption for consistency with other figure captions.
Minor punctuation inconsistency that detracts from professional presentation.
5.
MEDIUMcopyedit
Ensure consistent spacing before periods in command examples (e.g., '@whedon assign @danielskatz as editor .' should not have a space before the period).
Inconsistent spacing reduces readability and suggests lack of polish.
6.
MEDIUMcopyedit
Revise 'About half of these UK academics said they develop their own software while in the course of doing research' to 'while conducting research' for conciseness.
Minor phrasing improvement for clarity.
7.
MEDIUMcopyedit
Uniformly apply a consistent citation style (e.g., include DOIs for all references where available) to the reference list.
Inconsistent formatting (some with DOI, some without) may create confusion and is not in line with publishing standards.
8.
MEDIUMcopyedit
In the Figure 4 caption, replace 'Cumulative sum' with 'Cumulative number' for clarity.
'Cumulative number' is a more standard and clear term than 'cumulative sum'.
9.
LOWreporting
Include a formal data citation in the references section for the Figshare dataset (Niemeyer, 2017) to improve discoverability.
Adding a formal reference for the dataset makes it easier for readers to find and cite the underlying data.
10.
LOWreporting
Consider adding version numbers for all software tools mentioned (e.g., Ruby on Rails version, Whedon version) in the Methods section.
Version information helps readers understand the exact platform configuration at the time of the study.

Full text (57,222 chars)Analyzed: Jul 5, 2026

Αlpha¹ assesses methodological rigor, not the importance of the findings. See how we calibrate →