Kaimen Rigor reviewed

Rigor-Reviewed Manuscript

Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.

Name: Kaimen Rigor review — Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.
Item: Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.
Rating: 100
Author: Alpha1

Bean AM, Payne RE, Parsons G, Kirk HR, Ciro J, Mosquera-Gómez R, Hincapié M S, Ekanayaka AS, Tarassenko L, Rocher L, Mahdi A

Paper source

Read full text (PMC)

Embed this Rigor badge

Kaimen Rigor review — 3.75★ (Adequate), 0 critical, 2 high issues for “Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.”

Links back to this Kaimen Rigor review.

Alpha1 Editorial

This rigor review was examined and confirmed by Alpha1 Editorial · August 15, 2026

Read alongside

Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.

Bean AM, Payne RE, Parsons G, Kirk HR, Ciro J, Mosquera-Gómez R, Hincapié M S, Ekanayaka AS, Tarassenko L, Rocher L, Mahdi A

Read full text (PMC)

DOI

10.1038/s41591-025-04074-y

Share this Kaimen Rigor review

Post on X Share on LinkedIn Share via Email

Embed this Rigor badge

<a href="https://alpha1science.com/verify/3f379946-161c-4171-801d-a37a22bb1e00" target="_blank" rel="noopener">
  <img src="https://alpha1science.com/verify/3f379946-161c-4171-801d-a37a22bb1e00/badge" alt="Kaimen Rigor review — 3.75★ (Adequate), 0 critical, 2 high issues for “Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.”" width="440" height="104" style="max-width:100%;height:auto" />
</a>

Every embed links back to this Kaimen Rigor review.

Record fingerprint

d05f4833dffd39c566a9b149fc3c04ee573ff3ad2e036b022011181d11f90e7e

Verify via API

Issued Aug 15, 2026Engine 7.39.0

Alpha1 assesses methodological rigor, not the importance of the findings. See how we calibrate →

Alpha¹ by Kaimen

Fetching the latest research

Kaimen Rigor reviewed

Rigor-Reviewed Manuscript

Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.

Bean AM, Payne RE, Parsons G, Kirk HR, Ciro J, Mosquera-Gómez R, Hincapié M S, Ekanayaka AS, Tarassenko L, Rocher L, Mahdi A

Paper source

Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.

Read full text (PMC)

Embed this Rigor badge

Links back to this Kaimen Rigor review.

Alpha1 Editorial

This rigor review was examined and confirmed by Alpha1 Editorial · August 15, 2026

Kaimen Rigor review

Kaimen Rigor reviewed

3.75 / 5

Adequate — real issues to address

Interventional Study

Critical

High-priority

How this rating was calculated▸

Started at 5★ — deductions

CitationsUnresolved reference ×3−0.75★
IntegrityIntegrity concern−0.5★

Model-reported confidence: highReviewer agreement: highEngine: 7.39.0Source: full text

Not verified in this report

Statistics were not checked: no recomputable values were found in this text — no test statistic reported with its degrees of freedom, no effect estimate printed with both a 95% CI and a p-value, and no percentage printed with both its count and its denominator.

Findings by category

101ArithmeticNumerical inconsistencies 102OverreachOverstated conclusions 03AuthenticityNone found 04DisclosureNone found 105AvailabilityBroken references and links 06Copyedit5 minor suggestions

This Kaimen Rigor review uses Kaimen Rigor reviewers trained on a curated corpus of high-fidelity and retracted papers, with expert supervision and curation. It can still make mistakes; verify each finding against the source before relying on it.

The paper is a well-designed randomized controlled trial with clear ethical approvals, preregistration, and public data/code. Minor reporting gaps include lack of detailed power analysis, incomplete inclusion/exclusion criteria, and no explicit reporting guideline. A few copyedit issues (typos, inconsistent percentage) and three references not found in registries warrant attention.

Both reviewers classified the study as interventional (RCT), which is adopted. Non-applicable sub-criteria (e.g., animal housing, cell lines) were excluded. The statistics verification component recomputed 0 tests due to coverage limits, so statistical correctness is not independently confirmed. The reviewers diverged slightly on limitations addressed and exact p_values, but the combined evidence supports pass.

Numerical inconsistencies

1 finding · worst low

Values that contradict each other or are impossible for the stated sample: recomputed p-values and test statistics, GRIM/GRIMMER checks on summary numbers, percentages against their own counts, totals against their parts, and estimates against their own confidence intervals.

Internal contradictions in the reported numbersAssessed

Overstated conclusions

1 finding · worst low

Conclusions that reach past what the paper's own results support — including a significance claim that no longer holds when the statistic is recomputed, and efficacy resting on an unvalidated surrogate endpoint.

Conclusions only partially backed by the presented evidenceAssessed

5 major claims checked against the paper's own evidence: 1 only partially supported (evidence backs part of the claim; gaps or caveats remain); the rest adequately supported.

Data authenticity concerns

None found

An adversarial read for patterns associated with data that may not be genuine: results that look too clean, implausibly large effects, duplicated data or images, and methods that do not match the results reported.

Checked — nothing surfaced.

Reporting gaps

None found

Required detail the manuscript never states — study design, biological variables, ethics approval and consent, key resources, statistical reporting, data and code availability, and overall transparency.

Checked — nothing surfaced.

Eight Dimensions8 pass

The introduction cites prior studies showing LLMs excel on benchmarks but fail in real clinical settings, and notes that physician-LLM collaboration underperforms. The rationale linking this to testing LLMs with the general public is clear. However, the paper does not explicitly discuss limitations of prior research that it aims to address, though it implicitly does by focusing on human-LLM interaction.

Sub-criteria

prior work citedADEQUATE

premise rationaleADEQUATE

limitations addressedINADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The paper establishes a clear premise with prior work and a logical rationale, though it does not explicitly address limitations of prior research.

Evidence

paraphrase[Introduction, paragraph 2]

“For instance, one study showed that radiologists assisted by AI did not perform better at reading chest X-rays than without AI assistance, and both performed worse than AI alone.”

paraphrase[Introduction, paragraph 3]

“More promising and easier to deploy, LLM-powered chatbots have instead been suggested as a ‘new front door’ to healthcare for patients who lack medical expertise.”

absence[Introduction]

Limitations of prior research are not explicitly discussed.

PASSReviewer 2· 90% conf

The paper establishes a clear scientific premise with prior work and a logical rationale.

Evidence

“On the one hand, LLM scores on medical knowledge benchmarks are now commensurate with passing the US Medical Licensing Exam . LLM-generated clinical documents are rated as equivalent to or better than those written by doctors , . On the other hand, excelling at medical tasks in silico does not translate to accurate performance in clinical settings under physician guidance.”

Introduction ¶2Find in source

“To understand whether LLMs can reliably support the general public and bring care closer to patients, we conducted a study with 1,298 UK participants.”

Introduction ¶4Find in source

paraphrase[Discussion, paragraph 1]

“Paraphrase: The authors note that previous work focused on physicians and benchmarks, and they extend testing to the general public with realistic scenarios.”

Randomization method is described as stratified random sampling via Prolific, and the unit is individual participants. Blinding is partial: participants were blinded to which LLM they used, but the control group was aware they were not using an LLM. A power analysis is mentioned but not detailed in the main text. Inclusion/exclusion criteria are implied (age, English speaking) but not pre-specified in detail. Outlier handling is addressed through exclusions for technical issues and attrition analysis. Controls are appropriate (control group using usual methods). Independent replication is not applicable for a single trial.

Sub-criteria

controlsADEQUATE

power analysisINADEQUATE

blinding levelsADEQUATE

outlier handlingADEQUATE

randomization unitADEQUATE

inclusion exclusionINADEQUATE

randomization methodADEQUATE

replicate distinctionN/A

independent replicationN/A

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 85% conf

The study design is rigorous with randomization, stratification, and blinding of participants to model assignment, though power analysis is not detailed in the main text.

Evidence

“We used stratified random sampling via the Prolific platform to target a representative sample of the UK population in each group.”

Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.

Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.

Numerical inconsistencies

Overstated conclusions

Data authenticity concerns

Reporting gaps

Broken references and links

Copyediting

Numerical inconsistencies

Overstated conclusions

Data authenticity concerns

Reporting gaps

Broken references and links

Copyediting