Rigor Reviewed

Rigor-Reviewed Manuscript

An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.

Name: Αlpha¹ Rigor Record — An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.
Item: An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.
Rating: 100
Author: Αlpha¹

Tao X, Zhou S, Ding K, Li S, Li Y, Wu B, Huang Q, Chen W, Shen M, Meng E, Chen X, Hu H, Zhang J, Zhou J, Zou L, Ma L, Han S

Paper source

Read full text (PMC)

Embed this Rigor badge

Αlpha¹ Rigor Review — 2.25★ (Weak), 0 critical, 2 high issues for “An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.”

Links back to this Rigor Record.

Αlpha¹ Editorial

This rigor review was examined and confirmed by Αlpha¹ Editorial · August 15, 2026

Αlpha¹ Rigor Review

Rigor Reviewed

2.25 / 5

Weak — serious problems found

Interventional Study

Critical

High-priority

How this rating was calculated▸

Started at 5★ — deductions

IntegrityIntegrity concern ×3−1.5★
ClaimsEfficacy rests on an unvalidated surrogate endpoint−0.5★
ClaimsTreatment effect not shown to be clinically meaningful−0.5★
CitationsUnresolved reference−0.25★

Model-reported confidence: highReviewer agreement: highEngine: 7.39.0Source: full text

Not verified in this report

Statistics were not checked: no recomputable values were found in this text — no test statistic reported with its degrees of freedom, no effect estimate printed with both a 95% CI and a p-value, and no percentage printed with both its count and its denominator.
The numeric-impossibility checks (GRIM/GRIMMER/DEBIT/SPRITE) did not run: 35 reported means were read, and their group size is not stated where the values are printed (this source has no machine-readable table structure). These checks need the count the mean was averaged over, so none was performed.

Problems surfaced6 types

Numerical inconsistencies1

Values that contradict each other or are impossible for the stated sample: recomputed p-values and test statistics, GRIM/GRIMMER checks on summary numbers, percentages against their own counts, totals against their parts, and estimates against their own confidence intervals.

Internal contradictions in the reported numbersAssessed

Overstated conclusions3

Conclusions that reach past what the paper's own results support — including a significance claim that no longer holds when the statistic is recomputed, and efficacy resting on an unvalidated surrogate endpoint.

Efficacy rests on an unvalidated surrogate endpointAssessed
Treatment effect not shown to be clinically meaningfulAssessed
Conclusions only partially backed by the presented evidenceAssessed

Data authenticity concerns1

An adversarial read for patterns associated with data that may not be genuine: results that look too clean, implausibly large effects, duplicated data or images, and methods that do not match the results reported.

Data look implausibly cleanAssessed

Broken references and links1

References checked against Crossref, OpenAlex and Retraction Watch for retractions and resolvability, plus declared data and code links probed for whether they resolve to content matching the paper.

References not resolvable to a published paperRecomputed

This Αlpha¹ Rigor Review uses AI Rigor Reviewers trained on a curated corpus of high-fidelity and retracted papers, with expert supervision and curation. It can still make mistakes; verify each finding against the source before relying on it.

This is a well-conducted and transparently reported pragmatic RCT evaluating an LLM chatbot for care transitions. The study design is rigorous with clear randomization, blinding, and power analysis, and the manuscript thoroughly documents ethics, data/code availability, and limitations. Minor reporting gaps (e.g., missing CONSORT flow diagram, unclear handling of missing physician notes) and a single unresolved reference do not undermine the overall integrity.

Both reviewers independently scored all eight dimensions and agreed on every status; no divergence required reconciliation. The statistics verification component found no recomputable tests (coverage limited to tests with test statistic + df or effect + CI), so reported statistics remain unverified. The citation check flagged one reference not found in any registry (a GitHub/DOI entry), which is a potential fabrication signal but not confirmed.

6 major claims checked against the paper's own evidence: 1 only partially supported (evidence backs part of the claim; gaps or caveats remain); the rest adequately supported.

Premise concern: surrogate not validated for clinical benefit; effect size not shown to be clinically meaningful.

3 integrity concerns flagged (0 high).

Reporting

Eight Dimensions8 pass

The introduction cites numerous studies on healthcare system inefficiencies, LLM applications, and co-design approaches, acknowledging both strengths and limitations of prior work. The rationale linking the premise to the study objectives is explicit, and the paper addresses limitations of prior research by proposing co-design as a novel strategy.

Sub-criteria

prior work citedADEQUATE

premise rationaleADEQUATE

limitations addressedADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The paper establishes a clear scientific premise with extensive prior work and a logical rationale for the study.

Evidence

paraphrase[Introduction, paragraphs 1-3]

“Large language models (LLMs) possess transformative potential to re-engineer hospital workflows... However, current applications remain largely confined to support healthcare professionals in controlled settings... Critically, evidence is lacking for LLM chatbots that directly interact with socioeconomically diverse patient populations.”

paraphrase[Introduction, paragraph 4]

“The answer is critical for global health equity, as passively collected local dialogues may codify and even scale systemic inequities.”

paraphrase[Introduction, paragraph 4]

“To bridge the gap between the potential of LLMs and their practical impact in resource-limited settings, we developed PreA... using a multistakeholder participatory co-design approach.”

PASSReviewer 2· 90% conf

The paper establishes a clear scientific premise with extensive prior work and a logical rationale for the study.

Evidence

paraphrase[Introduction, paragraphs 1-3]

“Large language models (LLMs) possess transformative potential to re-engineer hospital workflows and address the systemic inefficiencies amplified by escalating demand. However, current applications remain largely confined to support healthcare professionals in controlled settings...”

direct quote[Introduction, paragraph 4]

“Yet, the relative utility of co-design versus passive data collecting for meeting clinical needs remains unknown.”