Rigor Reviewed

Rigor-Reviewed Manuscript

Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.

Name: Αlpha¹ Rigor Record — Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.
Item: Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.
Rating: 100
Author: Αlpha¹

Bean AM, Payne RE, Parsons G, Kirk HR, Ciro J, Mosquera-Gómez R, Hincapié M S, Ekanayaka AS, Tarassenko L, Rocher L, Mahdi A

Paper source

Read full text (PMC)

Embed this Rigor badge

Αlpha¹ Rigor Review — 3.75★ (Adequate), 0 critical, 2 high issues for “Reliability of LLMs as medical assistants for the general public: a randomized preregistered study.”

Links back to this Rigor Record.

Αlpha¹ Editorial

This rigor review was examined and confirmed by Αlpha¹ Editorial · August 15, 2026

Αlpha¹ Rigor Review

Rigor Reviewed

3.75 / 5

Adequate — real issues to address

Interventional Study

Critical

High-priority

How this rating was calculated▸

Started at 5★ — deductions

CitationsUnresolved reference ×3−0.75★
IntegrityIntegrity concern−0.5★

Model-reported confidence: highReviewer agreement: highEngine: 7.39.0Source: full text

Not verified in this report

Statistics were not checked: no recomputable values were found in this text — no test statistic reported with its degrees of freedom, no effect estimate printed with both a 95% CI and a p-value, and no percentage printed with both its count and its denominator.

Problems surfaced3 types

Numerical inconsistencies1

Values that contradict each other or are impossible for the stated sample: recomputed p-values and test statistics, GRIM/GRIMMER checks on summary numbers, percentages against their own counts, totals against their parts, and estimates against their own confidence intervals.

Internal contradictions in the reported numbersAssessed

Overstated conclusions1

Conclusions that reach past what the paper's own results support — including a significance claim that no longer holds when the statistic is recomputed, and efficacy resting on an unvalidated surrogate endpoint.

Conclusions only partially backed by the presented evidenceAssessed

Broken references and links1

References checked against Crossref, OpenAlex and Retraction Watch for retractions and resolvability, plus declared data and code links probed for whether they resolve to content matching the paper.

References not resolvable to a published paperRecomputed

This Αlpha¹ Rigor Review uses AI Rigor Reviewers trained on a curated corpus of high-fidelity and retracted papers, with expert supervision and curation. It can still make mistakes; verify each finding against the source before relying on it.

The paper is a well-designed randomized controlled trial with clear ethical approvals, preregistration, and public data/code. Minor reporting gaps include lack of detailed power analysis, incomplete inclusion/exclusion criteria, and no explicit reporting guideline. A few copyedit issues (typos, inconsistent percentage) and three references not found in registries warrant attention.

Both reviewers classified the study as interventional (RCT), which is adopted. Non-applicable sub-criteria (e.g., animal housing, cell lines) were excluded. The statistics verification component recomputed 0 tests due to coverage limits, so statistical correctness is not independently confirmed. The reviewers diverged slightly on limitations addressed and exact p_values, but the combined evidence supports pass.

5 major claims checked against the paper's own evidence: 1 only partially supported (evidence backs part of the claim; gaps or caveats remain); the rest adequately supported.

1 integrity concern flagged (0 high).

Reporting

Eight Dimensions8 pass

The introduction cites prior studies showing LLMs excel on benchmarks but fail in real clinical settings, and notes that physician-LLM collaboration underperforms. The rationale linking this to testing LLMs with the general public is clear. However, the paper does not explicitly discuss limitations of prior research that it aims to address, though it implicitly does by focusing on human-LLM interaction.

Sub-criteria

prior work citedADEQUATE

premise rationaleADEQUATE

limitations addressedINADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The paper establishes a clear premise with prior work and a logical rationale, though it does not explicitly address limitations of prior research.

Evidence

paraphrase[Introduction, paragraph 2]

“For instance, one study showed that radiologists assisted by AI did not perform better at reading chest X-rays than without AI assistance, and both performed worse than AI alone.”

paraphrase[Introduction, paragraph 3]

“More promising and easier to deploy, LLM-powered chatbots have instead been suggested as a ‘new front door’ to healthcare for patients who lack medical expertise.”

absence[Introduction]

Limitations of prior research are not explicitly discussed.

PASSReviewer 2· 90% conf

The paper establishes a clear scientific premise with prior work and a logical rationale.

Evidence

direct quote[Introduction, paragraph 2]

“On the one hand, LLM scores on medical knowledge benchmarks are now commensurate with passing the US Medical Licensing Exam . LLM-generated clinical documents are rated as equivalent to or better than those written by doctors , . On the other hand, excelling at medical tasks in silico does not translate to accurate performance in clinical settings under physician guidance.”