Kaimen Rigor reviewed

Rigor-Reviewed Manuscript

A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls: a cluster randomized trial.

Name: Kaimen Rigor review — A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls: a cluster randomized trial.
Item: A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls: a cluster randomized trial.
Rating: 100
Author: Alpha1

Hou Z, Wu Z, Qu Z, Gong L, Peng H, Jit M, Larson HJ, Wu JT, Lin L

Paper source

Read full text (PMC)

Embed this Rigor badge

Kaimen Rigor review — 3.5★ (Adequate), 0 critical, 7 high issues for “A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls: a cluster randomized trial.”

Links back to this Kaimen Rigor review.

Alpha1 Editorial

This rigor review was examined and confirmed by Alpha1 Editorial · August 15, 2026

Read alongside

A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls: a cluster randomized trial.

Hou Z, Wu Z, Qu Z, Gong L, Peng H, Jit M, Larson HJ, Wu JT, Lin L

Read full text (PMC)

DOI

10.1038/s41591-025-03618-6

Share this Kaimen Rigor review

Post on X Share on LinkedIn Share via Email

Embed this Rigor badge

<a href="https://alpha1science.com/verify/e62cbd71-b512-4634-b8f4-855123287a12" target="_blank" rel="noopener">
  <img src="https://alpha1science.com/verify/e62cbd71-b512-4634-b8f4-855123287a12/badge" alt="Kaimen Rigor review — 3.5★ (Adequate), 0 critical, 7 high issues for “A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls: a cluster randomized trial.”" width="440" height="104" style="max-width:100%;height:auto" />
</a>

Every embed links back to this Kaimen Rigor review.

Record fingerprint

e9e9f6a2c560d7f45683bf72f51a93e91f8de5cf8f091cf3330c3d1a2f1a5306

Verify via API

Issued Aug 15, 2026Engine 7.39.0

Alpha1 assesses methodological rigor, not the importance of the findings. See how we calibrate →

Alpha¹ by Kaimen

Fetching the latest research

Kaimen Rigor reviewed

Rigor-Reviewed Manuscript

A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls: a cluster randomized trial.

Hou Z, Wu Z, Qu Z, Gong L, Peng H, Jit M, Larson HJ, Wu JT, Lin L

Paper source

A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls: a cluster randomized trial.

Read full text (PMC)

Embed this Rigor badge

Links back to this Kaimen Rigor review.

Alpha1 Editorial

This rigor review was examined and confirmed by Alpha1 Editorial · August 15, 2026

Kaimen Rigor review

Kaimen Rigor reviewed

3.5 / 5

Adequate — real issues to address

Interventional Study

Critical

High-priority

How this rating was calculated▸

Started at 5★ — deductions

IntegrityIntegrity concern ×2−1★
CitationsUnresolved reference ×2−0.5★

Model-reported confidence: highReviewer agreement: highEngine: 7.39.0Source: full text

Not verified in this report

The numeric-impossibility checks (GRIM/GRIMMER/DEBIT/SPRITE) did not run: 18 reported means were read, and their group size is not stated where the values are printed (this source has no machine-readable table structure). These checks need the count the mean was averaged over, so none was performed.

Findings by category

101ArithmeticNumerical inconsistencies 02OverreachNone found 03AuthenticityNone found 04DisclosureNone found 105AvailabilityBroken references and links 06Copyedit10 minor suggestions

This Kaimen Rigor review uses Kaimen Rigor reviewers trained on a curated corpus of high-fidelity and retracted papers, with expert supervision and curation. It can still make mistakes; verify each finding against the source before relying on it.

This is a well-conducted and transparently reported cluster randomized trial. The design is rigorous, ethics approvals are documented, and the statistical analysis is appropriate with effect sizes and CIs reported. Minor reporting gaps (outlier handling, assumption verification) and several copyedit issues (typos, percentage inconsistencies) do not undermine the core findings.

Both reviewers classified the study as interventional (cluster RCT), and this was adopted. The evaluation covered the full text, with N/A for non-applicable sub-criteria (e.g., animal/housing, cell lines). The statistics verification covered only 4 tests with test statistics/CIs; other p-values were not machine-verified. The citation check found 2 references not found in registries (arXiv preprints), which are flagged for verification.

Numerical inconsistencies

1 finding · worst low

Values that contradict each other or are impossible for the stated sample: recomputed p-values and test statistics, GRIM/GRIMMER checks on summary numbers, percentages against their own counts, totals against their parts, and estimates against their own confidence intervals.

Internal contradictions in the reported numbersAssessed

Recomputed 4 tests: 4 consistent, 0 inconsistent; 4 via agent-written checks.

Overstated conclusions

None found

Conclusions that reach past what the paper's own results support — including a significance claim that no longer holds when the statistic is recomputed, and efficacy resting on an unvalidated surrogate endpoint.

Checked — nothing surfaced.

10 major claims checked against the paper's own evidence: all adequately supported.

Efficacy claim is anchored to an adequate endpoint and a meaningful effect.

Data authenticity concerns

None found

An adversarial read for patterns associated with data that may not be genuine: results that look too clean, implausibly large effects, duplicated data or images, and methods that do not match the results reported.

Checked — nothing surfaced.

Reporting gaps

None found

Required detail the manuscript never states — study design, biological variables, ethics approval and consent, key resources, statistical reporting, data and code availability, and overall transparency.

Checked — nothing surfaced.

Eight Dimensions8 pass

The introduction cites prior research on HPV burden, low vaccination coverage in China, parental hesitancy, and the potential of chatbots, while acknowledging limitations of prior work (e.g., lack of studies in non-routine immunization contexts with out-of-pocket costs). The rationale logically connects these gaps to the study's aim of evaluating a chatbot in China's context. Limitations of prior research are addressed by the study design (e.g., cluster RCT, diverse socioeconomic settings).

Sub-criteria

prior work citedADEQUATE

premise rationaleADEQUATE

limitations addressedADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The paper establishes a strong scientific premise with extensive prior work and a clear rationale linking the intervention to the study objectives.

Evidence

“Cervical cancer remains a major global health challenge, with 662,301 new cases and 348,874 deaths reported worldwide in 2022”

Introduction ¶1Find in source

“Our study addresses these gaps by examining how a chatbot intervention might influence HPV vaccination in China, where substantial cost barriers exist”

Introduction ¶5Find in source

paraphrase[Introduction, paragraph 5]

“The study addresses gaps in understanding mechanisms and effectiveness in out-of-pocket contexts.”

PASSReviewer 2· 95% conf

The paper establishes a clear scientific premise by citing prior work on HPV vaccination barriers, chatbot interventions, and knowledge gaps, with a logical rationale linking the premise to the study objectives.

Evidence

paraphrase[Introduction, paragraphs 1-4]

“China faces a heavey disease burden, accounting for 22.8% of the global incidence and 16.0% of the deaths... By 2022, only 10.15% of Chinese women aged 9–45 years had been vaccinated against HPV... Conversational artificial intelligence (AI), often implemented using chatbots, uses large language models to enable new, automated, human-like interactions.”

paraphrase[Introduction, paragraph 4]

“However, two critical knowledge gaps remain. First, the mechanisms through which chatbot interventions influence an individual vaccination decision-making process remain poorly understood. Second, although chatbot interventions have shown promise in emergency vaccination campaigns such as COVID-19... their effectiveness in contexts in which vaccines are not part of routine immunization programs and require substantial out-of-pocket costs requires investigation.”

“Our study addresses these gaps by examining how a chatbot intervention might influence HPV vaccination in China, where substantial cost barriers exist, while also exploring the pathways through which such digital intervention can impact vaccine-related decision-making.”

Introduction ¶5Find in source

Randomization was computer-generated, stratified by region, school, and grade, with classes as the unit. Blinding of participants and implementers was not possible due to the intervention nature, but randomization was blinded to schools, teachers, and participants. A priori power analysis is reported with assumptions (ICC, design effect, dropout). Inclusion/exclusion criteria are pre-specified. Outlier handling is not explicitly discussed but the analysis uses robust methods (GEE, mixed models). Controls are appropriate (usual care). Independent replication is not reported (n/a for a single trial).

Sub-criteria

controlsADEQUATE

power analysisADEQUATE

blinding levelsADEQUATE

outlier handlingNOT REPORTED

randomization unitADEQUATE

inclusion exclusionADEQUATE

randomization methodADEQUATE

replicate distinctionN/A

independent replicationN/A

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The cluster randomized trial design is rigorous, with clear randomization, stratification, and sample size justification.

Evidence

“Using class lists in each grade in a school, computer-generated randomization was used to determine whether class 1 and class 2 were assigned to intervention or control group”

MethodsFind in source

“The nature of the intervention did not allow for masking of the intervention to class teachers, participants or study implementers.”

MethodsFind in source

“Assuming a power of 80%, a two-sided significance level of 0.05 and incorporating this cluster design effect, we determined that a sample size of 648 participants per arm would be sufficient”

MethodsFind in source

PASSReviewer 2· 95% conf

The study design is rigorous for a cluster RCT: randomization method, unit, stratification, and sample size calculation are reported; blinding is described with rationale; inclusion/exclusion criteria are clear; and the analysis accounts for clustering.

Evidence

paraphrase[Methods, Sampling, randomization and masking]

“The classes as clusters were randomly assigned to either the intervention or control groups in a 1:1 ratio, using a stratified randomization approach... computer-generated randomization was used...”

“The nature of the intervention did not allow for masking of the intervention to class teachers, participants or study implementers.”

MethodsFind in source

paraphrase[Methods, Sample size]

“The sample size was calculated based on detecting a meaningful difference in the primary outcome... Assuming a power of 80%, a two-sided significance level of 0.05 and incorporating this cluster design effect, we determined that a sample size of 648 participants per arm would be sufficient...”

Sex of participants (mother/father) and daughters (all female) is reported. The study focuses on female students, so single-sex is justified by the research question. Age, grade, region, education, income, and other demographics are reported in Table 2. Species/strain/housing are not applicable (human study).

Sub-criteria

demographicsADEQUATE

sex reportedADEQUATE

sex justifiedADEQUATE

age weight healthADEQUATE

housing conditionsN/A

species strain sourceN/A

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

Demographic and health-related variables are comprehensively reported for both parents and daughters.

Evidence

“Mothers constituted 87.7% of participants”

Table 2Find in source

“Age, years (mean ± s.d.) | 13.1 ± 1.1”

Table 2Find in source

paraphrase[Results, Table 2]

“Education level, n (%) | High education | 1,167 (43.7)”

PASSReviewer 2· 95% conf

Human demographics (age, sex, region, education, income) are comprehensively reported for both parents and daughters; sex is reported (mothers/fathers) and justified by the study focus on female students.

Evidence

paraphrase[Table 2]

“Parenthood, n (%) | Mother 2,342 (87.7) | Father 329 (12.3)”

“Age, years (mean ± s.d.) | 13.1 ± 1.1”

Table 2Find in source

“A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls”

AbstractFind in source

The study received approval from the IRB of Fudan University School of Public Health and the Human Research Ethics Committee of the University of Hong Kong, with an approval number. Informed consent is described in detail, including a 14-day opt-out period. Regulatory compliance is implied through adherence to guidelines.

Sub-criteria

iacuc statementN/A

informed consentADEQUATE

irb ethics statementADEQUATE

regulatory complianceADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 95% conf

Ethical approvals from named IRBs and informed consent procedures are clearly documented.

Evidence

“This study was a cluster randomized trial approved by both the Institutional Review Board (IRB) of Fudan University School of Public Health and the Human Research Ethics Committee of the University of Hong Kong.”

MethodsFind in source

“All participants provided informed consent to participate this trial.”

MethodsFind in source

“This study was conducted in alignment with evidence-based clinical practice guidelines”

EthicsFind in source

PASSReviewer 2· 95% conf

Ethics approval from named IRBs (Fudan University and University of Hong Kong) with protocol number is reported; informed consent is described; regulatory compliance is stated.

Evidence

paraphrase[Methods, Trial design and setting; Inclusion and ethics statement]

paraphrase[Methods, Study procedures]

“All participants provided informed consent to participate this trial... The informed consent included nine key elements: (1) study purpose, (2) procedures, (3) risks, (4) benefits, (5) compensation details, (6) confidentiality measures, (7) right to withdraw, (8) signature for informed consent, and (9) investigator’s statement.”

“This study was conducted in alignment with evidence-based clinical practice guidelines and received approval from the IRB of Fudan University School of Public Health and Human Research Ethics Committee of the University of Hong Kong.”

EthicsFind in source

The chatbot is described in detail (knowledge base, roles, GPT-4, RAG, prompt engineering) with a URL. Statistical software (STATA v.15.1, R v.4.4.1) is identified. No antibodies, cell lines, or organisms are used (n/a). Reagents are not applicable (no wet-lab components).

Sub-criteria

mycoplasma testingN/A

reagents identifiedN/A

organisms identifiedN/A

antibodies identifiedN/A

cell line authenticationN/A

software tools identifiedADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The investigational product (chatbot) is described in detail, and software tools are identified.

Evidence

“In this study, we developed an AI-powered chatbot tailored for HPV vaccine consultation, designed specifically for the Chinese context.”

MethodsFind in source

“Statistical analyses were performed using STATA v.15.1 and R v.4.4.1 statistical software.”

Statistical analysisFind in source

PASSReviewer 2· 95% conf

The investigational product (chatbot) is described with its development, platform (WeChat/web), and URL; software tools (STATA, R, GPT-4) are identified with versions; no antibodies, cell lines, or organisms are used.

Evidence

“The AI-powered chatbot is accessible via WeChat and through web browsers at https://hpvchatbot.social-insight.ai .”

MethodsFind in source

“Statistical analyses were performed using STATA v.15.1 and R v.4.4.1 statistical software.”

Statistical analysisFind in source

“our chatbot uses advanced linguistic technologies through GPT-4 (ref. ) coupled with retrieval-augmented generation and prompt engineering”

MethodsFind in source

Tests are named (GEE for categorical, mixed-effects for continuous, log-binomial for engagement). Assumptions are not explicitly verified but the methods (GEE, mixed models) are robust to typical violations. Exact p-values are reported (e.g., P < 0.001). Effect sizes with 95% CIs are reported for all outcomes. Software is identified. Data presentation includes per-group n, percentages, means, SDs, and CIs. Mathematical plausibility checks: the primary outcome counts (92/1294 = 7.1%, 25/1377 = 1.8%) are consistent; subgroup counts in Table 2 sum to totals (e.g., 322+956+788+605 = 2671).

Sub-criteria

tests namedADEQUATE

exact p valuesADEQUATE

effect sizes ciADEQUATE

data presentationADEQUATE

software identifiedADEQUATE

assumptions verifiedINADEQUATE

mathematical plausibilityPASS

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 85% conf

Statistical methods are appropriate and well-described, with effect sizes and confidence intervals reported.

Evidence

“Mixed-effects models and generalized estimating equations (GEE) were used to evaluate the effectiveness of the chatbot intervention”

Statistical analysisFind in source

paraphrase[Results, Table 3]

“Adjusted RR (95% CI) | 3.85 (2.48 to 5.97)”

paraphrase[Results, Table 3]

“P value | <0.001”

PASSReviewer 2· 95% conf

Statistical tests are named (GEE, mixed-effects, log-binomial), effect sizes with CIs are reported, software is identified, and data presentation includes per-group n and CIs. Assumptions are handled by design (clustering, stratification). Exact p-values are reported for primary and key secondary outcomes.

Evidence

paraphrase[Methods, Statistical analysis]

“Mixed-effects models and generalized estimating equations (GEE) were used to evaluate the effectiveness of the chatbot intervention on primary and secondary outcomes... log-binomial generalized linear models to assess the associations between chatbot engagement characteristics...”

paraphrase[Table 3]

“Adjusted RR (95% CI) | 3.85 (2.48 to 5.97)”

paraphrase[Table 3]

“P value | <0.001”

The data availability statement explains that data are not publicly available due to privacy/consent but provides a mechanism for access (request to corresponding authors, one-month timeline, data use agreement). This is adequate for patient-level data. Code is shared on GitHub (https://github.com/wu-zhengdong/HPV-vaccine-chatbot.git). Repository deposit and accession numbers are not applicable (no sequencing or depositable non-identifiable data).

Sub-criteria

code sharingADEQUATE

accession numbersN/A

repository depositN/A

data availability statementADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

Data availability statement is concrete, and code is shared in a public repository.

Evidence

“Access to data will be provided upon application, with a timeline of one month determined in accordance with the request.”

Data availabilityFind in source

“All codes are freely available on GitHub at https://github.com/wu-zhengdong/HPV-vaccine-chatbot.git”

Data availabilityFind in source

PASSReviewer 2· 95% conf

Data availability statement is present with a concrete access route (request to authors with conditions and timeline); code is shared on GitHub with a URL.

Evidence

paraphrase[Data availability]

“The data are not publicly available... Researchers interested in accessing the data should submit requests to the corresponding author... Access to data will be provided upon application, with a timeline of one month determined in accordance with the request. All approved users must sign a data use agreement...”

“All codes are freely available on GitHub at https://github.com/wu-zhengdong/HPV-vaccine-chatbot.git .”

Data availabilityFind in source

Methods include trial design, setting, participants, intervention, outcomes, sample size, and statistical analysis in detail. Trial registration number is provided. CONSORT checklist is mentioned in supplementary materials. All pre-specified outcomes (primary and secondary) are reported with results. Limitations are discussed in detail (out-of-pocket costs, short duration, cross-contamination, etc.). Conclusions are proportional (e.g., 'further research is necessary to scale and sustain these gains'). Funding sources and COI are stated.

Sub-criteria

funding coiADEQUATE

trial registrationADEQUATE

reporting guidelineADEQUATE

methods completenessADEQUATE

all outcomes reportedADEQUATE

limitations discussedADEQUATE

conclusions proportionalADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The manuscript is transparent, with trial registration, CONSORT checklist, and thorough discussion of limitations.

Evidence

“Clinical trial registration: NCT06227689”

AbstractFind in source

“Supplementary Document 4. CONSORT checklist.”

Supplementary materialFind in source

“The study has several limitations.”

Discussion ¶6Find in source

PASSReviewer 2· 95% conf

Methods are detailed enough for replication; trial is registered (NCT06227689); CONSORT checklist is referenced; all outcomes are reported; limitations are discussed; conclusions are proportional; funding and COI are stated.

Evidence

“Clinical trial registration: NCT06227689 (https://clinicaltrials.gov/ct2/show/NCT06227689) .”

AbstractFind in source

“Supplementary Document 4. CONSORT checklist.”

Supplementary materialFind in source

paraphrase[Discussion, paragraph 5]

“The study has several limitations. First, HPV vaccination requires out-of-pocket payment and the preferred nonavalent vaccine is in short supply in our settings...”

Registered (1 ID: ClinicalTrials.gov). Reporting guideline cited: CONSORT.

Broken references and links

1 finding · worst low

References checked against Crossref, OpenAlex and Retraction Watch for retractions and resolvability, plus declared data and code links probed for whether they resolve to content matching the paper.

References not resolvable to a published paperRecomputed

Checked 55 references by DOI: 2 verified — 2 DOI unresolved, 51 no DOI (shown, not verified).

1 data/code link checked; 1 live.

Copyediting

10 minor

Wording, consistency and formatting errors that need correcting before submission.

No major wording or formatting errors. 10 minor suggestions below.

10 copyedit issues flagged: mostly consistency, typo, clarity.

Kaimen Rigor Action ItemsFinal pass before journal submission

The published work is robust and well-reported; an informed reader should weigh the minor reporting gaps (outlier handling, assumption verification) and the copyedit inconsistencies, but none threaten the validity of the conclusions. No erratum is warranted for the core findings, though the authors should correct the typographical and percentage errors in the tables.

1.
HIGHcopyedit
Correct the percentage for rural usual care in Table 1 from 38.9% to 28.9% (26/90).
The printed percentage is internally inconsistent with the count and could be misread as a data error.
2.
HIGHcopyedit
Fix the income range typo in Table 2: change '1000,000–200,000 CNY' to '100,000–200,000 CNY'.
The extra zero is a clear typographical error that could confuse readers.
3.
HIGHcopyedit
Remove the stray parenthesis in Table 2 age row: '40.3 ± 4.4)' should be '40.3 ± 4.4'.
Formatting error in a key demographic table.
4.
HIGHcopyedit
Fix the typo 'heavey' to 'heavy' in the Abstract and Introduction.
Typographical error in the opening lines.
5.
HIGHreporting
Add a statement in the Methods (Statistical analysis) on how outliers were handled (e.g., whether any data points were excluded and why).
One reviewer flagged outlier handling as not explicitly reported; this is a transparency gap for a cluster RCT.
6.
HIGHstatistics
Explicitly verify and report test assumptions (normality, equal variance) for continuous outcomes (e.g., HPV literacy scores) or justify the use of mixed-effects models without such verification.
Assumption verification was rated as reported_but_inadequate; adding this strengthens statistical transparency.
7.
HIGHreporting
Verify the two references not found in registries (GPT-4 technical report, arXiv:2303.08774; and 'A survey of large language models in medicine', arXiv:2311.05112) and correct or replace them if they are not accessible.
These references could not be located in Crossref/OpenAlex and may be fabricated or incorrectly cited.
8.
MEDIUMreporting
Report the intracluster correlation coefficient (ICC) for the primary outcome in the Results section.
The ICC was used in the sample size calculation but not reported in the results, which is a useful detail for readers.
9.
MEDIUMreporting
Clarify the stepwise covariate reduction procedure in the Methods (Statistical analysis), specifying the criteria used.
The current description is vague and not fully reproducible.
10.
MEDIUMreporting
Add a statement on how missing data were handled (e.g., complete-case analysis, imputation) for primary and secondary outcomes.
Missing data handling is not described, which is a common reviewer concern.
11.
MEDIUMreporting
Report exact p-values for all secondary outcomes, including non-significant ones, rather than only thresholds.
Exact p-values improve transparency and allow readers to assess borderline results.
12.
MEDIUMreporting
Provide a CONSORT-style flow diagram in the main text (currently only referenced as Fig. 1) to improve transparency.
A flow diagram is a key CONSORT element and aids reader comprehension of participant flow.
13.
MEDIUMdata code
Consider providing a more detailed data availability statement that includes a managed-access platform (e.g., Vivli, YODA) for patient-level data.
A managed-access platform would enhance data sharing credibility beyond author requests.
14.
MEDIUMreporting
Clarify the exact version of GPT-4 used (e.g., GPT-4-turbo) and whether any fine-tuning was performed on the chatbot's knowledge base.
This improves reproducibility of the intervention.
15.
MEDIUMreporting
Describe the randomization sequence generation in more detail (e.g., software used, seed) to enhance reproducibility.
Details on randomization sequence generation are important for reproducibility.
16.
LOWcopyedit
Fix grammar in Methods, Study procedures: 'participate this trial' → 'participate in this trial'.
Missing preposition is a minor grammar error.
17.
LOWcopyedit
Rephrase the subgroup analysis sentence in Results for clarity: 'vaccine receipt or scheduled appointment was 8.81 times higher (95% CI 2.74–28.35) in the chatbot group than in the usual care group'.
The current phrasing is slightly awkward and could be clearer.

Full text (71,040 chars)Analyzed: Aug 15, 2026

Start the discussion on JRNLClub

Community discussion on jrnlclub.com

Read alongside

A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls: a cluster randomized trial.

Hou Z, Wu Z, Qu Z, Gong L, Peng H, Jit M, Larson HJ, Wu JT, Lin L

Read full text (PMC)

DOI

10.1038/s41591-025-03618-6

Share this Kaimen Rigor review

Post on X Share on LinkedIn Share via Email

Embed this Rigor badge

<a href="https://alpha1science.com/verify/e62cbd71-b512-4634-b8f4-855123287a12" target="_blank" rel="noopener">
  <img src="https://alpha1science.com/verify/e62cbd71-b512-4634-b8f4-855123287a12/badge" alt="Kaimen Rigor review — 3.5★ (Adequate), 0 critical, 7 high issues for “A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls: a cluster randomized trial.”" width="440" height="104" style="max-width:100%;height:auto" />
</a>

Every embed links back to this Kaimen Rigor review.

Record fingerprint

e9e9f6a2c560d7f45683bf72f51a93e91f8de5cf8f091cf3330c3d1a2f1a5306

Verify via API

Issued Aug 15, 2026Engine 7.39.0

Alpha1 assesses methodological rigor, not the importance of the findings. See how we calibrate →

Kaimen Rigor review

Kaimen Rigor reviewed

3.5 / 5

Adequate — real issues to address

Interventional Study

Critical

High-priority

How this rating was calculated▸

Started at 5★ — deductions

IntegrityIntegrity concern ×2−1★
CitationsUnresolved reference ×2−0.5★

Model-reported confidence: highReviewer agreement: highEngine: 7.39.0Source: full text

Not verified in this report

The numeric-impossibility checks (GRIM/GRIMMER/DEBIT/SPRITE) did not run: 18 reported means were read, and their group size is not stated where the values are printed (this source has no machine-readable table structure). These checks need the count the mean was averaged over, so none was performed.

Findings by category

101ArithmeticNumerical inconsistencies 02OverreachNone found 03AuthenticityNone found 04DisclosureNone found 105AvailabilityBroken references and links 06Copyedit10 minor suggestions

Numerical inconsistencies

1 finding · worst low

Internal contradictions in the reported numbersAssessed

Recomputed 4 tests: 4 consistent, 0 inconsistent; 4 via agent-written checks.

Overstated conclusions

None found

Checked — nothing surfaced.

10 major claims checked against the paper's own evidence: all adequately supported.

Efficacy claim is anchored to an adequate endpoint and a meaningful effect.

Data authenticity concerns

None found

Checked — nothing surfaced.

Reporting gaps

None found

Checked — nothing surfaced.

Eight Dimensions8 pass

Sub-criteria

prior work citedADEQUATE

premise rationaleADEQUATE

limitations addressedADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The paper establishes a strong scientific premise with extensive prior work and a clear rationale linking the intervention to the study objectives.

Evidence

“Cervical cancer remains a major global health challenge, with 662,301 new cases and 348,874 deaths reported worldwide in 2022”

Introduction ¶1Find in source

“Our study addresses these gaps by examining how a chatbot intervention might influence HPV vaccination in China, where substantial cost barriers exist”

Introduction ¶5Find in source

paraphrase[Introduction, paragraph 5]

“The study addresses gaps in understanding mechanisms and effectiveness in out-of-pocket contexts.”

PASSReviewer 2· 95% conf

Evidence

paraphrase[Introduction, paragraphs 1-4]

paraphrase[Introduction, paragraph 4]

Introduction ¶5Find in source

Sub-criteria

controlsADEQUATE

power analysisADEQUATE

blinding levelsADEQUATE

outlier handlingNOT REPORTED

randomization unitADEQUATE

inclusion exclusionADEQUATE

randomization methodADEQUATE

replicate distinctionN/A

independent replicationN/A

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The cluster randomized trial design is rigorous, with clear randomization, stratification, and sample size justification.

Evidence

“Using class lists in each grade in a school, computer-generated randomization was used to determine whether class 1 and class 2 were assigned to intervention or control group”

MethodsFind in source

“The nature of the intervention did not allow for masking of the intervention to class teachers, participants or study implementers.”

MethodsFind in source

“Assuming a power of 80%, a two-sided significance level of 0.05 and incorporating this cluster design effect, we determined that a sample size of 648 participants per arm would be sufficient”

MethodsFind in source

PASSReviewer 2· 95% conf

Evidence

paraphrase[Methods, Sampling, randomization and masking]

“The nature of the intervention did not allow for masking of the intervention to class teachers, participants or study implementers.”

MethodsFind in source

paraphrase[Methods, Sample size]

Sub-criteria

demographicsADEQUATE

sex reportedADEQUATE

sex justifiedADEQUATE

age weight healthADEQUATE

housing conditionsN/A

species strain sourceN/A

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

Demographic and health-related variables are comprehensively reported for both parents and daughters.

Evidence

“Mothers constituted 87.7% of participants”

Table 2Find in source

“Age, years (mean ± s.d.) | 13.1 ± 1.1”

Table 2Find in source

paraphrase[Results, Table 2]

“Education level, n (%) | High education | 1,167 (43.7)”

PASSReviewer 2· 95% conf

Evidence

paraphrase[Table 2]

“Parenthood, n (%) | Mother 2,342 (87.7) | Father 329 (12.3)”

“Age, years (mean ± s.d.) | 13.1 ± 1.1”

Table 2Find in source

“A vaccine chatbot intervention for parents to improve HPV vaccination uptake among middle school girls”

AbstractFind in source

Sub-criteria

iacuc statementN/A

informed consentADEQUATE

irb ethics statementADEQUATE

regulatory complianceADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 95% conf

Ethical approvals from named IRBs and informed consent procedures are clearly documented.

Evidence

MethodsFind in source

“All participants provided informed consent to participate this trial.”

MethodsFind in source

“This study was conducted in alignment with evidence-based clinical practice guidelines”

EthicsFind in source

PASSReviewer 2· 95% conf

Ethics approval from named IRBs (Fudan University and University of Hong Kong) with protocol number is reported; informed consent is described; regulatory compliance is stated.

Evidence

paraphrase[Methods, Trial design and setting; Inclusion and ethics statement]

paraphrase[Methods, Study procedures]

EthicsFind in source

Sub-criteria

mycoplasma testingN/A

reagents identifiedN/A

organisms identifiedN/A

antibodies identifiedN/A

cell line authenticationN/A

software tools identifiedADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The investigational product (chatbot) is described in detail, and software tools are identified.

Evidence

“In this study, we developed an AI-powered chatbot tailored for HPV vaccine consultation, designed specifically for the Chinese context.”

MethodsFind in source

“Statistical analyses were performed using STATA v.15.1 and R v.4.4.1 statistical software.”

Statistical analysisFind in source

PASSReviewer 2· 95% conf

Evidence

“The AI-powered chatbot is accessible via WeChat and through web browsers at https://hpvchatbot.social-insight.ai .”

MethodsFind in source

“Statistical analyses were performed using STATA v.15.1 and R v.4.4.1 statistical software.”

Statistical analysisFind in source

“our chatbot uses advanced linguistic technologies through GPT-4 (ref. ) coupled with retrieval-augmented generation and prompt engineering”

MethodsFind in source

Sub-criteria

tests namedADEQUATE

exact p valuesADEQUATE

effect sizes ciADEQUATE

data presentationADEQUATE

software identifiedADEQUATE

assumptions verifiedINADEQUATE

mathematical plausibilityPASS

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 85% conf

Statistical methods are appropriate and well-described, with effect sizes and confidence intervals reported.

Evidence

“Mixed-effects models and generalized estimating equations (GEE) were used to evaluate the effectiveness of the chatbot intervention”

Statistical analysisFind in source

paraphrase[Results, Table 3]

“Adjusted RR (95% CI) | 3.85 (2.48 to 5.97)”

paraphrase[Results, Table 3]

“P value | <0.001”

PASSReviewer 2· 95% conf

Evidence

paraphrase[Methods, Statistical analysis]

paraphrase[Table 3]

“Adjusted RR (95% CI) | 3.85 (2.48 to 5.97)”

paraphrase[Table 3]

“P value | <0.001”

Sub-criteria

code sharingADEQUATE

accession numbersN/A

repository depositN/A

data availability statementADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

Data availability statement is concrete, and code is shared in a public repository.

Evidence

“Access to data will be provided upon application, with a timeline of one month determined in accordance with the request.”

Data availabilityFind in source

“All codes are freely available on GitHub at https://github.com/wu-zhengdong/HPV-vaccine-chatbot.git”

Data availabilityFind in source

PASSReviewer 2· 95% conf

Data availability statement is present with a concrete access route (request to authors with conditions and timeline); code is shared on GitHub with a URL.

Evidence

paraphrase[Data availability]

“All codes are freely available on GitHub at https://github.com/wu-zhengdong/HPV-vaccine-chatbot.git .”

Data availabilityFind in source

Sub-criteria

funding coiADEQUATE

trial registrationADEQUATE

reporting guidelineADEQUATE

methods completenessADEQUATE

all outcomes reportedADEQUATE

limitations discussedADEQUATE

conclusions proportionalADEQUATE

Per reviewer (2) — the votes behind the verdict

PASSReviewer 1· 90% conf

The manuscript is transparent, with trial registration, CONSORT checklist, and thorough discussion of limitations.

Evidence

“Clinical trial registration: NCT06227689”

AbstractFind in source

“Supplementary Document 4. CONSORT checklist.”

Supplementary materialFind in source

“The study has several limitations.”

Discussion ¶6Find in source

PASSReviewer 2· 95% conf

Evidence

“Clinical trial registration: NCT06227689 (https://clinicaltrials.gov/ct2/show/NCT06227689) .”

AbstractFind in source

“Supplementary Document 4. CONSORT checklist.”

Supplementary materialFind in source

paraphrase[Discussion, paragraph 5]

“The study has several limitations. First, HPV vaccination requires out-of-pocket payment and the preferred nonavalent vaccine is in short supply in our settings...”

Registered (1 ID: ClinicalTrials.gov). Reporting guideline cited: CONSORT.

Broken references and links

1 finding · worst low

References checked against Crossref, OpenAlex and Retraction Watch for retractions and resolvability, plus declared data and code links probed for whether they resolve to content matching the paper.

References not resolvable to a published paperRecomputed

Checked 55 references by DOI: 2 verified — 2 DOI unresolved, 51 no DOI (shown, not verified).

1 data/code link checked; 1 live.

Copyediting

10 minor

Wording, consistency and formatting errors that need correcting before submission.

No major wording or formatting errors. 10 minor suggestions below.

10 copyedit issues flagged: mostly consistency, typo, clarity.

Kaimen Rigor Action ItemsFinal pass before journal submission

1.
HIGHcopyedit
Correct the percentage for rural usual care in Table 1 from 38.9% to 28.9% (26/90).
The printed percentage is internally inconsistent with the count and could be misread as a data error.
2.
HIGHcopyedit
Fix the income range typo in Table 2: change '1000,000–200,000 CNY' to '100,000–200,000 CNY'.
The extra zero is a clear typographical error that could confuse readers.
3.
HIGHcopyedit
Remove the stray parenthesis in Table 2 age row: '40.3 ± 4.4)' should be '40.3 ± 4.4'.
Formatting error in a key demographic table.
4.
HIGHcopyedit
Fix the typo 'heavey' to 'heavy' in the Abstract and Introduction.
Typographical error in the opening lines.
5.
HIGHreporting
Add a statement in the Methods (Statistical analysis) on how outliers were handled (e.g., whether any data points were excluded and why).
One reviewer flagged outlier handling as not explicitly reported; this is a transparency gap for a cluster RCT.
6.
HIGHstatistics
Explicitly verify and report test assumptions (normality, equal variance) for continuous outcomes (e.g., HPV literacy scores) or justify the use of mixed-effects models without such verification.
Assumption verification was rated as reported_but_inadequate; adding this strengthens statistical transparency.
7.
HIGHreporting
Verify the two references not found in registries (GPT-4 technical report, arXiv:2303.08774; and 'A survey of large language models in medicine', arXiv:2311.05112) and correct or replace them if they are not accessible.
These references could not be located in Crossref/OpenAlex and may be fabricated or incorrectly cited.
8.
MEDIUMreporting
Report the intracluster correlation coefficient (ICC) for the primary outcome in the Results section.
The ICC was used in the sample size calculation but not reported in the results, which is a useful detail for readers.
9.
MEDIUMreporting
Clarify the stepwise covariate reduction procedure in the Methods (Statistical analysis), specifying the criteria used.
The current description is vague and not fully reproducible.
10.
MEDIUMreporting
Add a statement on how missing data were handled (e.g., complete-case analysis, imputation) for primary and secondary outcomes.
Missing data handling is not described, which is a common reviewer concern.
11.
MEDIUMreporting
Report exact p-values for all secondary outcomes, including non-significant ones, rather than only thresholds.
Exact p-values improve transparency and allow readers to assess borderline results.
12.
MEDIUMreporting
Provide a CONSORT-style flow diagram in the main text (currently only referenced as Fig. 1) to improve transparency.
A flow diagram is a key CONSORT element and aids reader comprehension of participant flow.
13.
MEDIUMdata code
Consider providing a more detailed data availability statement that includes a managed-access platform (e.g., Vivli, YODA) for patient-level data.
A managed-access platform would enhance data sharing credibility beyond author requests.
14.
MEDIUMreporting
Clarify the exact version of GPT-4 used (e.g., GPT-4-turbo) and whether any fine-tuning was performed on the chatbot's knowledge base.
This improves reproducibility of the intervention.
15.
MEDIUMreporting
Describe the randomization sequence generation in more detail (e.g., software used, seed) to enhance reproducibility.
Details on randomization sequence generation are important for reproducibility.
16.
LOWcopyedit
Fix grammar in Methods, Study procedures: 'participate this trial' → 'participate in this trial'.
Missing preposition is a minor grammar error.
17.
LOWcopyedit
Rephrase the subgroup analysis sentence in Results for clarity: 'vaccine receipt or scheduled appointment was 8.81 times higher (95% CI 2.74–28.35) in the chatbot group than in the usual care group'.
The current phrasing is slightly awkward and could be clearer.

Full text (71,040 chars)Analyzed: Aug 15, 2026