PASSReviewer 1· 70% conf
Methods are detailed, null results are reported transparently, limitations are discussed, and funding/COI are addressed; no explicit reporting checklist (e.g., CONSORT) is referenced.
Evidence
direct quote[Methods, first paragraph]
“We preregistered our study design, data collection strategy and analysis plan for the human subject experiment ( https://osf.io/dt2p3 ).”direct quote[Results, Experimental performance]
“Participants using LLMs did not have statistically significant differences in disposition accuracy from the control group”direct quote[Discussion, paragraph 4]
“Our scenarios focused on common conditions where users may be familiar with the symptoms, and results might differ on rare conditions or less typical presentations.”PASSReviewer 2· 85% conf
The study is preregistered, methods are detailed, limitations are discussed, and conclusions are proportional; a reporting guideline is not explicitly mentioned.
Evidence
direct quote[Methods, first paragraph]
“We preregistered our study design, data collection strategy and analysis plan for the human subject experiment ( https://osf.io/dt2p3 ).”direct quote[Discussion, paragraph 4]
“Our work can only provide a lower bound on performance: newer models, models that make use of advanced techniques from chain of thought to reasoning tokens, or fine-tuned specialized models, are likely to provide higher performance on medical benchmarks.”direct quote[Acknowledgements]
“The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.”PASSReviewer 3· 90% conf
Methods are detailed, the study is preregistered, outcomes are reported, limitations are discussed, and conclusions are proportional. However, a reporting guideline is not mentioned and a COI statement is not present in the main text.
Evidence
direct quote[Methods, paragraph 1]
“We preregistered our study design, data collection strategy and analysis plan for the human subject experiment ( https://osf.io/dt2p3 ).”paraphrase[Discussion, paragraph 4]
“We used clinical vignettes for the interactions to limit risks to participants. This allowed us to focus on the challenges of LLMs interacting with members of the general public, without taking into account participants’ feelings of urgency and stress...”paraphrase[End of main text]
“The paper states 'Competing interests; and statements of data and code availability are available at 10.1038/s41591-025-04074-y' but does not include a COI statement in the main text.”