Kaimen Rigor reviewed

Rigor-Reviewed Manuscript

An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.

Name: Kaimen Rigor review — An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.
Item: An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.
Rating: 100
Author: Alpha1

Tao X, Zhou S, Ding K, Li S, Li Y, Wu B, Huang Q, Chen W, Shen M, Meng E, Chen X, Hu H, Zhang J, Zhou J, Zou L, Ma L, Han S

Paper source

Read full text (PMC)

Embed this Rigor badge

Kaimen Rigor review — 2.25★ (Weak), 0 critical, 2 high issues for “An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.”

Links back to this Kaimen Rigor review.

Alpha1 Editorial

This rigor review was examined and confirmed by Alpha1 Editorial · August 15, 2026

Read alongside

An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.

Tao X, Zhou S, Ding K, Li S, Li Y, Wu B, Huang Q, Chen W, Shen M, Meng E, Chen X, Hu H, Zhang J, Zhou J, Zou L, Ma L, Han S

Read full text (PMC)

DOI

10.1038/s41591-025-04176-7

Share this Kaimen Rigor review

Post on X Share on LinkedIn Share via Email

Embed this Rigor badge

<a href="https://alpha1science.com/verify/75a6bc70-86c5-48a2-b062-749083cfda62" target="_blank" rel="noopener">
  <img src="https://alpha1science.com/verify/75a6bc70-86c5-48a2-b062-749083cfda62/badge" alt="Kaimen Rigor review — 2.25★ (Weak), 0 critical, 2 high issues for “An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.”" width="440" height="104" style="max-width:100%;height:auto" />
</a>

Every embed links back to this Kaimen Rigor review.

Record fingerprint

3fc0bb5fab38346a23525f1b27473bd13f83a786300e19c865499836b8fd16f6

Verify via API

Issued Aug 15, 2026Engine 7.39.0

Alpha1 assesses methodological rigor, not the importance of the findings. See how we calibrate →

Alpha¹ by Kaimen

Fetching the latest research

Kaimen Rigor reviewed

Rigor-Reviewed Manuscript

An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.

Tao X, Zhou S, Ding K, Li S, Li Y, Wu B, Huang Q, Chen W, Shen M, Meng E, Chen X, Hu H, Zhang J, Zhou J, Zou L, Ma L, Han S

Paper source

An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.

Read full text (PMC)

Embed this Rigor badge

Links back to this Kaimen Rigor review.

Alpha1 Editorial

This rigor review was examined and confirmed by Alpha1 Editorial · August 15, 2026

Kaimen Rigor review

Kaimen Rigor reviewed

2.25 / 5

Weak — serious problems found

Interventional Study

Critical

High-priority

How this rating was calculated▸

Started at 5★ — deductions

IntegrityIntegrity concern ×3−1.5★
ClaimsEfficacy rests on an unvalidated surrogate endpoint−0.5★
ClaimsTreatment effect not shown to be clinically meaningful−0.5★
CitationsUnresolved reference−0.25★

Model-reported confidence: highReviewer agreement: highEngine: 7.39.0Source: full text

Not verified in this report

Statistics were not checked: no recomputable values were found in this text — no test statistic reported with its degrees of freedom, no effect estimate printed with both a 95% CI and a p-value, and no percentage printed with both its count and its denominator.
The numeric-impossibility checks (GRIM/GRIMMER/DEBIT/SPRITE) did not run: 35 reported means were read, and their group size is not stated where the values are printed (this source has no machine-readable table structure). These checks need the count the mean was averaged over, so none was performed.

Biggest issues

01Efficacy rests on an unvalidated surrogate endpoint
highOverreachreviewer-assessedFull detail
The primary outcomes are consultation duration, physician-rated care coordination, and patient-rated ease of communication. These are process and patient-reported measures, not hard clinical outcomes. The paper does not establish a validated link between these surrogates and long-term clinical outcomes such as mortality or morbidity. Target engagement is not demonstrated in terms of dose-exposure or PK/PD.
“The primary outcomes were consultation duration, physician-rated care coordination, and patient-rated ease of communication. These metrics were selected based on co-design feedback, which identified time efficiency and care coordination as critical for…”
Find in source
02Treatment effect not shown to be clinically meaningful
highOverreachreviewer-assessedFull detail
The reported effects are relative reductions/increases in consultation duration and patient-reported scores. For example, consultation duration reduced by 28.7% (from 4.41 to 3.14 minutes), and care coordination score increased by 113.1% (from 1.73 to 3.69 on a 5-point scale). However, these are not anchored to a minimal clinically important difference or a clear biological/clinical meaningfulness threshold. The absolute changes are small (e.g., ~1.3 minutes reduction in consultation time) and may not be clinically meaningful.
“The PreA-only group had a significantly shorter consultation duration compared to the No-PreA group (PreA-only 3.14 ± 2.25 versus No-PreA 4.41 ± 2.77 min; P < 0.001; Fig. ), corresponding to a 28.7% (95% CI 22.7–34.8) relative reduction.”
Find in source

Findings by category

101ArithmeticNumerical inconsistencies 302OverreachOverstated conclusions 103AuthenticityData authenticity concerns 04DisclosureNone found 105AvailabilityBroken references and links 06Copyedit5 minor suggestions

This Kaimen Rigor review uses Kaimen Rigor reviewers trained on a curated corpus of high-fidelity and retracted papers, with expert supervision and curation. It can still make mistakes; verify each finding against the source before relying on it.

This is a well-conducted and transparently reported pragmatic RCT evaluating an LLM chatbot for care transitions. The study design is rigorous with clear randomization, blinding, and power analysis, and the manuscript thoroughly documents ethics, data/code availability, and limitations. Minor reporting gaps (e.g., missing CONSORT flow diagram, unclear handling of missing physician notes) and a single unresolved reference do not undermine the overall integrity.

Both reviewers independently scored all eight dimensions and agreed on every status; no divergence required reconciliation. The statistics verification component found no recomputable tests (coverage limited to tests with test statistic + df or effect + CI), so reported statistics remain unverified. The citation check flagged one reference not found in any registry (a GitHub/DOI entry), which is a potential fabrication signal but not confirmed.

Numerical inconsistencies

1 finding · worst low

Values that contradict each other or are impossible for the stated sample: recomputed p-values and test statistics, GRIM/GRIMMER checks on summary numbers, percentages against their own counts, totals against their parts, and estimates against their own confidence intervals.

Internal contradictions in the reported numbersAssessed

Overstated conclusions

3 findings · worst high

Conclusions that reach past what the paper's own results support — including a significance claim that no longer holds when the statistic is recomputed, and efficacy resting on an unvalidated surrogate endpoint.

Efficacy rests on an unvalidated surrogate endpointAssessed
Treatment effect not shown to be clinically meaningfulAssessed
Conclusions only partially backed by the presented evidenceAssessed

6 major claims checked against the paper's own evidence: 1 only partially supported (evidence backs part of the claim; gaps or caveats remain); the rest adequately supported.

Premise concern: surrogate not validated for clinical benefit; effect size not shown to be clinically meaningful.

Data authenticity concerns

1 finding · worst low

An adversarial read for patterns associated with data that may not be genuine: results that look too clean, implausibly large effects, duplicated data or images, and methods that do not match the results reported.

Data look implausibly cleanAssessed

3 integrity concerns flagged (0 high).

Reporting gaps

None found

Required detail the manuscript never states — study design, biological variables, ethics approval and consent, key resources, statistical reporting, data and code availability, and overall transparency.

Checked — nothing surfaced.

Eight Dimensions8 pass

The introduction cites numerous studies on healthcare system inefficiencies, LLM applications, and co-design approaches, acknowledging both strengths and limitations of prior work. The rationale linking the premise to the study objectives is explicit, and the paper addresses limitations of prior research by proposing co-design as a novel strategy.

Sub-criteria

prior work citedADEQUATE

premise rationaleADEQUATE

limitations addressedADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The paper establishes a clear scientific premise with extensive prior work and a logical rationale for the study.

Evidence

paraphrase[Introduction, paragraphs 1-3]

“Large language models (LLMs) possess transformative potential to re-engineer hospital workflows... However, current applications remain largely confined to support healthcare professionals in controlled settings... Critically, evidence is lacking for LLM chatbots that directly interact with socioeconomically diverse patient populations.”

paraphrase[Introduction, paragraph 4]

“The answer is critical for global health equity, as passively collected local dialogues may codify and even scale systemic inequities.”

paraphrase[Introduction, paragraph 4]

“To bridge the gap between the potential of LLMs and their practical impact in resource-limited settings, we developed PreA... using a multistakeholder participatory co-design approach.”

PASSReviewer 2· 90% conf

The paper establishes a clear scientific premise with extensive prior work and a logical rationale for the study.

Evidence

paraphrase[Introduction, paragraphs 1-3]

“Large language models (LLMs) possess transformative potential to re-engineer hospital workflows and address the systemic inefficiencies amplified by escalating demand. However, current applications remain largely confined to support healthcare professionals in controlled settings...”

“Yet, the relative utility of co-design versus passive data collecting for meeting clinical needs remains unknown.”

Introduction ¶4Find in source

paraphrase[Introduction, paragraph 4]

“To bridge the gap between the potential of LLMs and their practical impact in resource-limited settings, we developed PreA... using a multistakeholder participatory co-design approach.”

Randomization used a computer-generated sequence without stratification, allocation was concealed, and the trial was single-blinded with physicians and data analysts blinded. A priori power analysis determined a target sample size of 2,010 participants. Inclusion/exclusion criteria were pre-specified. Outlier handling is addressed through the analysis population and missing-data approach.

Sub-criteria

controlsN/A

power analysisADEQUATE

blinding levelsADEQUATE

outlier handlingADEQUATE

randomization unitADEQUATE

inclusion exclusionADEQUATE

randomization methodADEQUATE

replicate distinctionN/A

independent replicationN/A

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The RCT design is rigorous with clear randomization, blinding, power analysis, and inclusion/exclusion criteria.

Evidence

“Participants were allocated to one of the three groups using an individual-level, computer-generated randomization sequence without stratification.”

MethodsFind in source

paraphrase[Methods, Randomization and blinding]

“This trial was single-blinded: while the patients knew their group assignments... the physicians were uninformed about the PreA-intervention groups... Furthermore, research staff involved in data analysis remained blinded to group assignments throughout the study.”

“The target minimum sample size of 2,010 participants (670 per study arm) was prespecified based on a power analysis using preliminary data from the pilot study of 90 patients. This minimum target sample size ensured sufficient power (>80%) for the primary outcome at a significance level of 0.05.”

MethodsFind in source

PASSReviewer 2· 85% conf

The RCT design is rigorous with clear randomization, blinding, power analysis, and inclusion/exclusion criteria.

Evidence

“Participants were allocated to one of the three groups using an individual-level, computer-generated randomization sequence without stratification.”

MethodsFind in source

“This trial was single-blinded: while the patients knew their group assignments (PreA-only, PreA-human, or No-PreA), the physicians were uninformed about the PreA-intervention groups (PreA-only or PreA-human).”

MethodsFind in source

“The target minimum sample size of 2,010 participants (670 per study arm) was prespecified based on a power analysis using preliminary data from the pilot study of 90 patients.”

MethodsFind in source

The paper reports age, sex, ethnicity, education, work status, income, and medical discipline for all participants. Since both sexes are enrolled, sex justification is not applicable. Age and health status are reported via inclusion criteria and baseline characteristics.

Sub-criteria

demographicsADEQUATE

sex reportedADEQUATE

sex justifiedN/A

age weight healthADEQUATE

housing conditionsN/A

species strain sourceN/A

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

Demographic and clinical characteristics are comprehensively reported and balanced across groups.

Evidence

“Participants had a mean age of 47.6 ± 14.6 years and included 1,141 women (55.1%) and 928 men (44.9%).”

ResultsFind in source

paraphrase[Table 1]

“These baseline covariates were well balanced across the three trial arms, with no significant differences in distribution (Table 1).”

PASSReviewer 2· 90% conf

Demographic and clinical characteristics are comprehensively reported and balanced across groups.

Evidence

“Participants had a mean age of 47.6 ± 14.6 years and included 1,141 women (55.1%) and 928 men (44.9%).”

ResultsFind in source

Table 1Find in source

The paper names the ethics committees that approved the study, states that informed consent was obtained from all participants, and declares adherence to the Declaration of Helsinki and ICH-GCP guidelines.

Sub-criteria

iacuc statementN/A

informed consentADEQUATE

irb ethics statementADEQUATE

regulatory complianceADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 95% conf

Ethics approval, informed consent, and regulatory compliance are clearly documented.

Evidence

“The Chinese Academy of Medical Sciences and Peking Union Medical College and the local medical ethics committee of the First Affiliated Hospital of Guilin Medical University approved the study.”

EthicsFind in source

“We obtained informed consent from all participants in this study.”

EthicsFind in source

“The trial followed the Declaration of Helsinki and the International Conference of Harmonization Guidelines for Good Clinical Practice.”

EthicsFind in source

PASSReviewer 2· 95% conf

Ethics approval, informed consent, and regulatory compliance are clearly documented.

Evidence

“The Chinese Academy of Medical Sciences and Peking Union Medical College and the local medical ethics committee of the First Affiliated Hospital of Guilin Medical University approved the study.”

EthicsFind in source

“We obtained informed consent from all participants in this study.”

EthicsFind in source

“The trial followed the Declaration of Helsinki and the International Conference of Harmonization Guidelines for Good Clinical Practice.”

EthicsFind in source

The PreA chatbot is described as based on OpenAI's GPT-4.0 mini, and the software used for analysis (Python v.3.7, R v.4.3.0) is identified. Since this is a clinical trial of a software intervention, bench resources like antibodies and cell lines are not applicable.

Sub-criteria

mycoplasma testingN/A

reagents identifiedADEQUATE

organisms identifiedN/A

antibodies identifiedN/A

cell line authenticationN/A

software tools identifiedADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 85% conf

The investigational product (PreA chatbot) and software tools are adequately identified.

Evidence

“we developed PreA (Pre-Assessment), an LLM chatbot (OpenAI; GPT-4.0 mini) for primary-to-specialist care transitions”

Introduction ¶4Find in source

“Python v.3.7 and R v.4.3.0 were used to perform the statistical analyses and present the results.”

Statistical analysisFind in source

PASSReviewer 2· 80% conf

The investigational product (PreA chatbot) and software tools are identified, though some details are limited.

Evidence

“we developed PreA (Pre-Assessment), an LLM chatbot (OpenAI; GPT-4.0 mini)”

Introduction ¶4Find in source

“Python v.3.7 and R v.4.3.0 were used to perform the statistical analyses and present the results.”

Statistical analysisFind in source

The paper names the statistical tests used (t-tests, Mann-Whitney U, chi-squared, ANOVA, Wilcoxon signed-rank), reports exact p-values, and provides effect sizes with 95% CIs. Software is identified. Data presentation includes box plots and per-group n. Mathematical plausibility checks were not possible for all values due to continuous data, but no obvious errors were found.

Sub-criteria

tests namedADEQUATE

exact p valuesADEQUATE

effect sizes ciADEQUATE

data presentationADEQUATE

software identifiedADEQUATE

assumptions verifiedADEQUATE

mathematical plausibilityN/A

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 85% conf

Statistical methods are appropriate and well-described, with exact p-values and effect sizes with confidence intervals.

Evidence

paraphrase[Methods, Statistical analysis]

“two-sample Student’s t-tests with unequal variances were used for approximately normal data, while a nonparametric Mann–Whitney U-test was applied to skewed distributions.”

paraphrase[Results, Outpatient workflow]

“The PreA-only group had a significantly shorter consultation duration compared to the No-PreA group (PreA-only 3.14 ± 2.25 versus No-PreA 4.41 ± 2.77 min; P < 0.001; Fig. 2)”

“corresponding to a 28.7% (95% CI 22.7–34.8) relative reduction.”

ResultsFind in source

PASSReviewer 2· 85% conf

Statistical methods are appropriate and well-described, with exact p-values and effect sizes with confidence intervals.

Evidence

paraphrase[Methods, Statistical analysis]

“two-sample Student’s t-tests with unequal variances were used for approximately normal data, while a nonparametric Mann–Whitney U-test was applied to skewed distributions.”

“The PreA-only group had a significantly shorter consultation duration compared to the No-PreA group (PreA-only 3.14 ± 2.25 versus No-PreA 4.41 ± 2.77 min; P < 0.001; Fig. ), corresponding to a 28.7% (95% CI 22.7–34.8) relative reduction.”

ResultsFind in source

“corresponding to a 28.7% (95% CI 22.7–34.8) relative reduction.”

ResultsFind in source

The paper provides a data availability statement with a clear mechanism for requesting anonymized data, and code is available in a public GitHub repository. Raw conversation data are not shared due to privacy, which is appropriate.

Sub-criteria

code sharingADEQUATE

accession numbersN/A

repository depositADEQUATE

data availability statementADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

Data and code availability are well-documented with a concrete access route.

Evidence

“Anonymized, nondialogue individual-level data underlying the results can be requested by qualified researchers for academic use. Requests should include a research proposal, statistical analysis plan and justification for data use, and can be submitted via email to S.H.”

Data availabilityFind in source

“Code for classification analysis and data visualization can be found at https://github.com/ShashaHan-collab/PreA-OutpatientRCT”

Data availabilityFind in source

PASSReviewer 2· 90% conf

Data and code availability are well-documented with a concrete access route.

Evidence

“Source data are provided in Tables and Extended Data Tables and can be accessed via the code repository ( https://github.com/ShashaHan-collab/PreA-OutpatientRCT ) .”

Data availabilityFind in source

“Code for classification analysis and data visualization can be found at https://github.com/ShashaHan-collab/PreA-OutpatientRCT (ref. ).”

Data availabilityFind in source

The trial is registered (ChiCTR2400094159), methods are detailed, limitations are discussed, and conclusions are proportional. Funding and competing interests are declared. A reporting summary is mentioned.

Sub-criteria

funding coiADEQUATE

trial registrationADEQUATE

reporting guidelineADEQUATE

methods completenessADEQUATE

all outcomes reportedADEQUATE

limitations discussedADEQUATE

conclusions proportionalADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The manuscript is transparent with complete methods, trial registration, and discussion of limitations.

Evidence

paraphrase[Methods, Ethics approval]

“This trial is registered at the Chinese Clinical Trial Registry (identifier ChiCTR2400094159)”

paraphrase[Discussion, paragraph 5]

“Several limitations warrant consideration when interpreting our findings. The generalizability of our time-reduction findings may be context-dependent...”

paraphrase[Acknowledgements and Competing interests]

“The funders had no role in study design and conduct of the study... J. Zha. and J. Zho. are employees of WeChat AI, Tencent, which provided computational resources and technical support for this research.”

PASSReviewer 2· 90% conf

The manuscript is transparent with complete methods, trial registration, and discussion of limitations.

Evidence

“Chinese Clinical Trial Registry identifier: ChiCTR2400094159”

AbstractFind in source

“Several limitations warrant consideration when interpreting our findings.”

Discussion ¶5Find in source

paraphrase[Acknowledgements and Competing interests]

“S.H. is supported by the National Natural Science Foundation of China (no. 82304269)... The other authors declare no competing interests.”

Registered (1 ID: Chinese Clinical Trial Registry). Reporting guideline cited: CONSORT.

Broken references and links

1 finding · worst low

References checked against Crossref, OpenAlex and Retraction Watch for retractions and resolvability, plus declared data and code links probed for whether they resolve to content matching the paper.

References not resolvable to a published paperRecomputed

Checked 57 references by DOI: 50 verified — 1 DOI unresolved, 6 no DOI (shown, not verified).

2 data/code links checked; 2 live.

Copyediting

5 minor

Wording, consistency and formatting errors that need correcting before submission.

No major wording or formatting errors. 5 minor suggestions below.

5 copyedit issues flagged: mostly clarity, grammar, consistency.

Kaimen Rigor Action ItemsFinal pass before journal submission

The published work is robust and well-reported; an informed reader should weigh the minor reporting gaps (missing CONSORT flow diagram, unclear handling of missing physician notes) and the single unresolved reference. No erratum is warranted for the main findings, but the authors should verify the flagged reference and consider adding the missing flow diagram in any revision.

1.
HIGHreporting
Verify the reference 'ShashaHan-colab/PreA-Outpatient RCT' (DOI 10.5281/zenodo.17330614) that was not found in any registry; correct or remove it if it is erroneous or fabricated.
An unresolved reference is a potential fabrication signal that must be resolved before the paper can be fully trusted.
2.
HIGHreporting
Add a CONSORT-style flow diagram to the main text to clearly show participant flow from screening to analysis, including the 69 patients excluded after randomization.
A flow diagram is a CONSORT requirement and would clarify the timing and reasons for exclusions, improving transparency.
3.
MEDIUMreporting
Clarify in the Methods how missing or 'blank' physician notes were handled in the referral quality agreement analysis (e.g., whether they were excluded from the denominator).
Unclear handling of missing data in a key analysis could affect the validity of the agreement results.
4.
MEDIUMreporting
State explicitly whether the trial was conducted in accordance with CONSORT guidelines, and provide the full statistical analysis plan as a supplementary file.
Explicit adherence to reporting guidelines and a linked SAP would strengthen reproducibility and transparency.
5.
MEDIUMdata code
Provide more details on the data access request process, such as expected response time and criteria for approval, in the Data Availability statement.
A more concrete access process would facilitate independent validation and reuse of the data.
6.
MEDIUMdata code
Consider making the PreA chatbot available for research purposes through a controlled access mechanism, and document its architecture and versioning in the Methods.
Sharing the intervention would enable independent replication and validation, which is currently limited by commercial licensing.
7.
MEDIUMstatistics
Report effect sizes with confidence intervals for secondary outcomes, not just p-values.
Effect sizes with CIs provide a more informative measure of the magnitude and precision of effects than p-values alone.
8.
MEDIUMreporting
Add a note on the generalizability of the findings to other healthcare settings beyond the two hospitals in western China.
The study's context-specific nature warrants a caution to readers about external validity.
9.
LOWcopyedit
Change 'compared to' to 'compared with' in the Results section for formal academic style.
Minor grammatical preference that improves academic tone.
10.
LOWcopyedit
Fix the missing possessive apostrophe in the Discussion: change 'The data-tuned model replication' to 'The data-tuned model's replication'.
Corrects a grammatical error that could confuse readers.
11.
LOWcopyedit
Clarify in the Results that the 28.7% relative reduction refers specifically to consultation duration.
Improves clarity and prevents misinterpretation of the effect size.
12.
LOWcopyedit
Ensure consistency in the Abstract and Table 1 for sex reporting (e.g., use parentheses consistently for counts and percentages).
Minor style consistency improves readability.

Full text (68,971 chars)Analyzed: Aug 15, 2026

Start the discussion on JRNLClub

Community discussion on jrnlclub.com

Read alongside

An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.

Tao X, Zhou S, Ding K, Li S, Li Y, Wu B, Huang Q, Chen W, Shen M, Meng E, Chen X, Hu H, Zhang J, Zhou J, Zou L, Ma L, Han S

Read full text (PMC)

DOI

10.1038/s41591-025-04176-7

Share this Kaimen Rigor review

Post on X Share on LinkedIn Share via Email

Embed this Rigor badge

<a href="https://alpha1science.com/verify/75a6bc70-86c5-48a2-b062-749083cfda62" target="_blank" rel="noopener">
  <img src="https://alpha1science.com/verify/75a6bc70-86c5-48a2-b062-749083cfda62/badge" alt="Kaimen Rigor review — 2.25★ (Weak), 0 critical, 2 high issues for “An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.”" width="440" height="104" style="max-width:100%;height:auto" />
</a>

Every embed links back to this Kaimen Rigor review.

Record fingerprint

3fc0bb5fab38346a23525f1b27473bd13f83a786300e19c865499836b8fd16f6

Verify via API

Issued Aug 15, 2026Engine 7.39.0

Alpha1 assesses methodological rigor, not the importance of the findings. See how we calibrate →

Kaimen Rigor review

Kaimen Rigor reviewed

2.25 / 5

Weak — serious problems found

Interventional Study

Critical

High-priority

How this rating was calculated▸

Started at 5★ — deductions

IntegrityIntegrity concern ×3−1.5★
ClaimsEfficacy rests on an unvalidated surrogate endpoint−0.5★
ClaimsTreatment effect not shown to be clinically meaningful−0.5★
CitationsUnresolved reference−0.25★

Model-reported confidence: highReviewer agreement: highEngine: 7.39.0Source: full text

Not verified in this report

Statistics were not checked: no recomputable values were found in this text — no test statistic reported with its degrees of freedom, no effect estimate printed with both a 95% CI and a p-value, and no percentage printed with both its count and its denominator.
The numeric-impossibility checks (GRIM/GRIMMER/DEBIT/SPRITE) did not run: 35 reported means were read, and their group size is not stated where the values are printed (this source has no machine-readable table structure). These checks need the count the mean was averaged over, so none was performed.

Biggest issues

01Efficacy rests on an unvalidated surrogate endpoint
highOverreachreviewer-assessedFull detail
The primary outcomes are consultation duration, physician-rated care coordination, and patient-rated ease of communication. These are process and patient-reported measures, not hard clinical outcomes. The paper does not establish a validated link between these surrogates and long-term clinical outcomes such as mortality or morbidity. Target engagement is not demonstrated in terms of dose-exposure or PK/PD.
“The primary outcomes were consultation duration, physician-rated care coordination, and patient-rated ease of communication. These metrics were selected based on co-design feedback, which identified time efficiency and care coordination as critical for…”
Find in source
02Treatment effect not shown to be clinically meaningful
highOverreachreviewer-assessedFull detail
The reported effects are relative reductions/increases in consultation duration and patient-reported scores. For example, consultation duration reduced by 28.7% (from 4.41 to 3.14 minutes), and care coordination score increased by 113.1% (from 1.73 to 3.69 on a 5-point scale). However, these are not anchored to a minimal clinically important difference or a clear biological/clinical meaningfulness threshold. The absolute changes are small (e.g., ~1.3 minutes reduction in consultation time) and may not be clinically meaningful.
“The PreA-only group had a significantly shorter consultation duration compared to the No-PreA group (PreA-only 3.14 ± 2.25 versus No-PreA 4.41 ± 2.77 min; P < 0.001; Fig. ), corresponding to a 28.7% (95% CI 22.7–34.8) relative reduction.”
Find in source

Findings by category

Numerical inconsistencies

1 finding · worst low

Internal contradictions in the reported numbersAssessed

Overstated conclusions

3 findings · worst high

Efficacy rests on an unvalidated surrogate endpointAssessed
Treatment effect not shown to be clinically meaningfulAssessed
Conclusions only partially backed by the presented evidenceAssessed

6 major claims checked against the paper's own evidence: 1 only partially supported (evidence backs part of the claim; gaps or caveats remain); the rest adequately supported.

Premise concern: surrogate not validated for clinical benefit; effect size not shown to be clinically meaningful.

Data authenticity concerns

1 finding · worst low

Data look implausibly cleanAssessed

3 integrity concerns flagged (0 high).

Reporting gaps

None found

Checked — nothing surfaced.

Eight Dimensions8 pass

Sub-criteria

prior work citedADEQUATE

premise rationaleADEQUATE

limitations addressedADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The paper establishes a clear scientific premise with extensive prior work and a logical rationale for the study.

Evidence

paraphrase[Introduction, paragraphs 1-3]

paraphrase[Introduction, paragraph 4]

“The answer is critical for global health equity, as passively collected local dialogues may codify and even scale systemic inequities.”

paraphrase[Introduction, paragraph 4]

“To bridge the gap between the potential of LLMs and their practical impact in resource-limited settings, we developed PreA... using a multistakeholder participatory co-design approach.”

PASSReviewer 2· 90% conf

The paper establishes a clear scientific premise with extensive prior work and a logical rationale for the study.

Evidence

paraphrase[Introduction, paragraphs 1-3]

“Yet, the relative utility of co-design versus passive data collecting for meeting clinical needs remains unknown.”

Introduction ¶4Find in source

paraphrase[Introduction, paragraph 4]

“To bridge the gap between the potential of LLMs and their practical impact in resource-limited settings, we developed PreA... using a multistakeholder participatory co-design approach.”

Sub-criteria

controlsN/A

power analysisADEQUATE

blinding levelsADEQUATE

outlier handlingADEQUATE

randomization unitADEQUATE

inclusion exclusionADEQUATE

randomization methodADEQUATE

replicate distinctionN/A

independent replicationN/A

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The RCT design is rigorous with clear randomization, blinding, power analysis, and inclusion/exclusion criteria.

Evidence

“Participants were allocated to one of the three groups using an individual-level, computer-generated randomization sequence without stratification.”

MethodsFind in source

paraphrase[Methods, Randomization and blinding]

MethodsFind in source

PASSReviewer 2· 85% conf

The RCT design is rigorous with clear randomization, blinding, power analysis, and inclusion/exclusion criteria.

Evidence

“Participants were allocated to one of the three groups using an individual-level, computer-generated randomization sequence without stratification.”

MethodsFind in source

“The target minimum sample size of 2,010 participants (670 per study arm) was prespecified based on a power analysis using preliminary data from the pilot study of 90 patients.”

MethodsFind in source

Sub-criteria

demographicsADEQUATE

sex reportedADEQUATE

sex justifiedN/A

age weight healthADEQUATE

housing conditionsN/A

species strain sourceN/A

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

Demographic and clinical characteristics are comprehensively reported and balanced across groups.

Evidence

“Participants had a mean age of 47.6 ± 14.6 years and included 1,141 women (55.1%) and 928 men (44.9%).”

ResultsFind in source

paraphrase[Table 1]

“These baseline covariates were well balanced across the three trial arms, with no significant differences in distribution (Table 1).”

PASSReviewer 2· 90% conf

Demographic and clinical characteristics are comprehensively reported and balanced across groups.

Evidence

“Participants had a mean age of 47.6 ± 14.6 years and included 1,141 women (55.1%) and 928 men (44.9%).”

ResultsFind in source

Table 1Find in source

Sub-criteria

iacuc statementN/A

informed consentADEQUATE

irb ethics statementADEQUATE

regulatory complianceADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 95% conf

Ethics approval, informed consent, and regulatory compliance are clearly documented.

Evidence

“The Chinese Academy of Medical Sciences and Peking Union Medical College and the local medical ethics committee of the First Affiliated Hospital of Guilin Medical University approved the study.”

EthicsFind in source

“We obtained informed consent from all participants in this study.”

EthicsFind in source

“The trial followed the Declaration of Helsinki and the International Conference of Harmonization Guidelines for Good Clinical Practice.”

EthicsFind in source

PASSReviewer 2· 95% conf

Ethics approval, informed consent, and regulatory compliance are clearly documented.

Evidence

“The Chinese Academy of Medical Sciences and Peking Union Medical College and the local medical ethics committee of the First Affiliated Hospital of Guilin Medical University approved the study.”

EthicsFind in source

“We obtained informed consent from all participants in this study.”

EthicsFind in source

“The trial followed the Declaration of Helsinki and the International Conference of Harmonization Guidelines for Good Clinical Practice.”

EthicsFind in source

Sub-criteria

mycoplasma testingN/A

reagents identifiedADEQUATE

organisms identifiedN/A

antibodies identifiedN/A

cell line authenticationN/A

software tools identifiedADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 85% conf

The investigational product (PreA chatbot) and software tools are adequately identified.

Evidence

“we developed PreA (Pre-Assessment), an LLM chatbot (OpenAI; GPT-4.0 mini) for primary-to-specialist care transitions”

Introduction ¶4Find in source

“Python v.3.7 and R v.4.3.0 were used to perform the statistical analyses and present the results.”

Statistical analysisFind in source

PASSReviewer 2· 80% conf

The investigational product (PreA chatbot) and software tools are identified, though some details are limited.

Evidence

“we developed PreA (Pre-Assessment), an LLM chatbot (OpenAI; GPT-4.0 mini)”

Introduction ¶4Find in source

“Python v.3.7 and R v.4.3.0 were used to perform the statistical analyses and present the results.”

Statistical analysisFind in source

Sub-criteria

tests namedADEQUATE

exact p valuesADEQUATE

effect sizes ciADEQUATE

data presentationADEQUATE

software identifiedADEQUATE

assumptions verifiedADEQUATE

mathematical plausibilityN/A

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 85% conf

Statistical methods are appropriate and well-described, with exact p-values and effect sizes with confidence intervals.

Evidence

paraphrase[Methods, Statistical analysis]

“two-sample Student’s t-tests with unequal variances were used for approximately normal data, while a nonparametric Mann–Whitney U-test was applied to skewed distributions.”

paraphrase[Results, Outpatient workflow]

“The PreA-only group had a significantly shorter consultation duration compared to the No-PreA group (PreA-only 3.14 ± 2.25 versus No-PreA 4.41 ± 2.77 min; P < 0.001; Fig. 2)”

“corresponding to a 28.7% (95% CI 22.7–34.8) relative reduction.”

ResultsFind in source

PASSReviewer 2· 85% conf

Statistical methods are appropriate and well-described, with exact p-values and effect sizes with confidence intervals.

Evidence

paraphrase[Methods, Statistical analysis]

“two-sample Student’s t-tests with unequal variances were used for approximately normal data, while a nonparametric Mann–Whitney U-test was applied to skewed distributions.”

ResultsFind in source

“corresponding to a 28.7% (95% CI 22.7–34.8) relative reduction.”

ResultsFind in source

Sub-criteria

code sharingADEQUATE

accession numbersN/A

repository depositADEQUATE

data availability statementADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

Data and code availability are well-documented with a concrete access route.

Evidence

Data availabilityFind in source

“Code for classification analysis and data visualization can be found at https://github.com/ShashaHan-collab/PreA-OutpatientRCT”

Data availabilityFind in source

PASSReviewer 2· 90% conf

Data and code availability are well-documented with a concrete access route.

Evidence

“Source data are provided in Tables and Extended Data Tables and can be accessed via the code repository ( https://github.com/ShashaHan-collab/PreA-OutpatientRCT ) .”

Data availabilityFind in source

“Code for classification analysis and data visualization can be found at https://github.com/ShashaHan-collab/PreA-OutpatientRCT (ref. ).”

Data availabilityFind in source

Sub-criteria

funding coiADEQUATE

trial registrationADEQUATE

reporting guidelineADEQUATE

methods completenessADEQUATE

all outcomes reportedADEQUATE

limitations discussedADEQUATE

conclusions proportionalADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The manuscript is transparent with complete methods, trial registration, and discussion of limitations.

Evidence

paraphrase[Methods, Ethics approval]

“This trial is registered at the Chinese Clinical Trial Registry (identifier ChiCTR2400094159)”

paraphrase[Discussion, paragraph 5]

“Several limitations warrant consideration when interpreting our findings. The generalizability of our time-reduction findings may be context-dependent...”

paraphrase[Acknowledgements and Competing interests]

PASSReviewer 2· 90% conf

The manuscript is transparent with complete methods, trial registration, and discussion of limitations.

Evidence

“Chinese Clinical Trial Registry identifier: ChiCTR2400094159”

AbstractFind in source

“Several limitations warrant consideration when interpreting our findings.”

Discussion ¶5Find in source

paraphrase[Acknowledgements and Competing interests]

“S.H. is supported by the National Natural Science Foundation of China (no. 82304269)... The other authors declare no competing interests.”

Registered (1 ID: Chinese Clinical Trial Registry). Reporting guideline cited: CONSORT.

Broken references and links

1 finding · worst low

References checked against Crossref, OpenAlex and Retraction Watch for retractions and resolvability, plus declared data and code links probed for whether they resolve to content matching the paper.

References not resolvable to a published paperRecomputed

Checked 57 references by DOI: 50 verified — 1 DOI unresolved, 6 no DOI (shown, not verified).

2 data/code links checked; 2 live.

Copyediting

5 minor

Wording, consistency and formatting errors that need correcting before submission.

No major wording or formatting errors. 5 minor suggestions below.

5 copyedit issues flagged: mostly clarity, grammar, consistency.

Kaimen Rigor Action ItemsFinal pass before journal submission

1.
HIGHreporting
Verify the reference 'ShashaHan-colab/PreA-Outpatient RCT' (DOI 10.5281/zenodo.17330614) that was not found in any registry; correct or remove it if it is erroneous or fabricated.
An unresolved reference is a potential fabrication signal that must be resolved before the paper can be fully trusted.
2.
HIGHreporting
Add a CONSORT-style flow diagram to the main text to clearly show participant flow from screening to analysis, including the 69 patients excluded after randomization.
A flow diagram is a CONSORT requirement and would clarify the timing and reasons for exclusions, improving transparency.
3.
MEDIUMreporting
Clarify in the Methods how missing or 'blank' physician notes were handled in the referral quality agreement analysis (e.g., whether they were excluded from the denominator).
Unclear handling of missing data in a key analysis could affect the validity of the agreement results.
4.
MEDIUMreporting
State explicitly whether the trial was conducted in accordance with CONSORT guidelines, and provide the full statistical analysis plan as a supplementary file.
Explicit adherence to reporting guidelines and a linked SAP would strengthen reproducibility and transparency.
5.
MEDIUMdata code
Provide more details on the data access request process, such as expected response time and criteria for approval, in the Data Availability statement.
A more concrete access process would facilitate independent validation and reuse of the data.
6.
MEDIUMdata code
Consider making the PreA chatbot available for research purposes through a controlled access mechanism, and document its architecture and versioning in the Methods.
Sharing the intervention would enable independent replication and validation, which is currently limited by commercial licensing.
7.
MEDIUMstatistics
Report effect sizes with confidence intervals for secondary outcomes, not just p-values.
Effect sizes with CIs provide a more informative measure of the magnitude and precision of effects than p-values alone.
8.
MEDIUMreporting
Add a note on the generalizability of the findings to other healthcare settings beyond the two hospitals in western China.
The study's context-specific nature warrants a caution to readers about external validity.
9.
LOWcopyedit
Change 'compared to' to 'compared with' in the Results section for formal academic style.
Minor grammatical preference that improves academic tone.
10.
LOWcopyedit
Fix the missing possessive apostrophe in the Discussion: change 'The data-tuned model replication' to 'The data-tuned model's replication'.
Corrects a grammatical error that could confuse readers.
11.
LOWcopyedit
Clarify in the Results that the 28.7% relative reduction refers specifically to consultation duration.
Improves clarity and prevents misinterpretation of the effect size.
12.
LOWcopyedit
Ensure consistency in the Abstract and Table 1 for sex reporting (e.g., use parentheses consistently for counts and percentages).
Minor style consistency improves readability.

Full text (68,971 chars)Analyzed: Aug 15, 2026