Stewardship Prompts to Improve Antibiotic Selection for Urinary Tract Infection: The INSPIRE Randomized Clinical Trial.
Gohil SK, Septimus E, Kleinman K, Varma N, Avery TR, Heim L, Rahm R, Cooper WS, Cooper M, McLean LE, Nickolay NG, Weinstein RA, Burgess LH, Coady MH, Rosen E, Sljivo S, Sands KE, Moody J, Vigeant J, Rashid S, Gilbert RF, Smith KN, Carver B, Poland RE, Hickok J, Sturdevant SG, Calderwood MS, Weiland A, Kubiak DW, Reddy S, Neuhauser MM, Srinivasan A, Jernigan JA, Hayden MK, Gowda A, Eibensteiner K, Wolf R, Perlin JB, Platt R, Huang SS
- DOI
- 10.1001/jama.2024.6259
- Record issued
- 2026-08-16
- Engine
- 7.39.0
- Exported
- 2026-09-22
Prepared by Alpha1. This document is confidential: it is intended for the recipient it was shared with and must not be redistributed. The live record at alpha1science.com/verify/69f3aa7a-8430-446c-8a40-3d4d124857a2 is authoritative.
How this rating was calculated
- IntegrityIntegrity concern ×3−1.5★
- ClaimsOverstated claim ×2−1★
- ClaimsEfficacy rests on an unvalidated surrogate endpoint−0.5★
- ClaimsTreatment effect not shown to be clinically meaningful−0.5★
- ReportingData & code availability partially met−0.25★
- No data or code availability links were detected to verify.
- 01Efficacy rests on an unvalidated surrogate endpoint
The primary outcome is empiric extended-spectrum antibiotic days of therapy, a process measure (antibiotic use) rather than a hard clinical outcome. The paper does not establish a validated link between reduced antibiotic use and improved clinical outcomes; safety outcomes (ICU transfer, length of stay) are secondary and show no significant difference. The efficacy claim rests on reducing antibiotic exposure, which is a surrogate for potential patient benefit (e.g., fewer adverse events, less resistance), but no direct evidence links this reduction to improved clinical outcomes.
“The primary outcome was empiric (first 3 days of hospitalization) extended-spectrum antibiotic days of therapy.”
- 02Treatment effect not shown to be clinically meaningful
The primary effect is a 17.4% relative reduction in extended-spectrum antibiotic days of therapy, but the absolute reduction is modest (from 392.2 to 326.0 per 1000 empiric days in the intervention group). The paper does not anchor this reduction to a minimal clinically important difference or demonstrate that it translates into meaningful patient benefit. The safety outcomes show no significant difference, so the clinical meaningfulness of the reduction is not established.
“a 17.4% (95% CI, 11.2% - 23.2%) reduction in empiric extended-spectrum days of therapy (rate ratio, 0.83 [95% CI, 0.77 - 0.89]; P < .001)”
- 03Conclusion reaches beyond the evidence
The intervention could reduce unnecessary antibiotics in up to 200,000 US adults annually.
suggesting that this intervention could be a viable strategy to reduce extended-spectrum antibiotics in up to 200 000 adults hospitalized in the US who receive unnecessarily broad antibiotics for UTI annually.
Discussion ¶1reviewer’s wording - 04Conclusion reaches beyond the evidence
The intervention is a viable strategy to reduce extended-spectrum antibiotics in up to 200,000 adults annually.
suggesting that this intervention could be a viable strategy to reduce extended-spectrum antibiotics in up to 200 000 adults hospitalized in the US who receive unnecessarily broad antibiotics for UTI annually
Discussion ¶1reviewer’s wording - 05Other integrity concern
Trial NCT03697096 was first submitted to ClinicalTrials.gov on 2018-10-03, after the registered study start date of 2018-10-01. Retrospective registration means the protocol and outcomes were not on the public record before the study ran, which is what prospective registration exists to establish.
NCT03697096
reviewer’s wording
This Kaimen Rigor review uses Kaimen Rigor reviewers trained on a curated corpus of high-fidelity and retracted papers, with expert supervision and curation. It can still make mistakes; verify each finding against the source before relying on it.
This is a well-designed cluster-randomized trial with rigorous methodology, clear reporting, and appropriate statistical analysis. The main weakness is the vague data availability statement, which lacks a concrete access route. Minor reporting gaps include lack of explicit blinding description and retrospective trial registration.
Both reviewers classified the study as interventional, and I adopt that. The evaluation covered the full text, including methods, results, and supplements referenced. Non-applicable criteria (e.g., blinding, species/strain, reagents) were excluded from scoring. The statistics verification covered only a subset of tests; the rest are unverified but not flagged.
Numerical inconsistencies
1 finding · worst lowValues that contradict each other or are impossible for the stated sample: recomputed p-values and test statistics, GRIM/GRIMMER checks on summary numbers, percentages against their own counts, totals against their parts, and estimates against their own confidence intervals.
- Internal contradictions in the reported numbersAssessed
Recomputed 3 tests: 3 consistent, 0 inconsistent; 3 via agent-written checks.
- CONSISTENTreported p < .001 · recomputed p = <.001Reviewers 1, 2Primary outcome rate ratio p-value from CI
“rate ratio, 0.83 [95% CI, 0.77-0.89]; P < .001”
Taken as given: The rate ratio is 0.83 with 95% CI 0.77-0.89.; The CI is two-sided at 95%.; The p-value is two-tailed.Method: Compute p from estimate and CI using normal approximation for log-ratio.How we recomputed it: pCI(0.83, 0.77, 0.89, 1) - CONSISTENTreported p = .002 · recomputed p = .004Reviewer 2Secondary outcome vancomycin rate ratio p-value from CI
“Vancomycin days-of-therapy ... 0.89 (0.82-0.96) 0.0021”
Taken as given: The reported RR is 0.89 with 95% CI 0.82-0.96.; The CI is two-sided at 95%.; The RR is on a log scale for the p-value calculation.Method: Used pCI function to compute two-tailed p-value from the reported RR and CI.How we recomputed it: pCI(0.89, 0.82, 0.96, 1) - CONSISTENTreported p < .001 · recomputed p = <.001Reviewer 2Secondary outcome antipseudomonal rate ratio p-value from CI
“Antipseudomonal days-of-therapy ... 0.79 (0.72-0.87) <.001”
Taken as given: The reported RR is 0.79 with 95% CI 0.72-0.87.; The CI is two-sided at 95%.; The RR is on a log scale for the p-value calculation.Method: Used pCI function to compute two-tailed p-value from the reported RR and CI.How we recomputed it: pCI(0.79, 0.72, 0.87, 1)
- lowinternal contradictionThe abstract reports 55,412 intervention period patients, but the results section reports 27,505 and 27,907 for the two groups, which sum to 55,412. This is consistent.
55,412 intervention period
Abstractreviewer’s wording
Overstated conclusions
3 findings · worst highConclusions that reach past what the paper's own results support — including a significance claim that no longer holds when the statistic is recomputed, and efficacy resting on an unvalidated surrogate endpoint.
- Efficacy rests on an unvalidated surrogate endpointAssessed
- Treatment effect not shown to be clinically meaningfulAssessed
- Conclusions overstated beyond the evidenceAssessed
7 major claims checked against the paper's own evidence: 2 not fully backed by the presented evidence (unsupported or overstated).
- overstatedReviewer 1The intervention could reduce unnecessary antibiotics in up to 200,000 US adults annually.This extrapolation is based on national estimates and assumes generalizability, which is not directly demonstrated by the trial.Evidence: Discussion states 'could be a viable strategy to reduce extended-spectrum antibiotics in up to 200 000 adults hospitalized in the US'.
suggesting that this intervention could be a viable strategy to reduce extended-spectrum antibiotics in up to 200 000 adults hospitalized in the US who receive unnecessarily broad antibiotics for UTI annually.
Discussion ¶1reviewer’s wording - overstatedReviewer 2The intervention is a viable strategy to reduce extended-spectrum antibiotics in up to 200,000 adults annually.The claim extrapolates from the trial results to a national estimate, which is speculative and not directly measured in the study.Evidence: Discussion section extrapolates from the observed reduction to national numbers.
suggesting that this intervention could be a viable strategy to reduce extended-spectrum antibiotics in up to 200 000 adults hospitalized in the US who receive unnecessarily broad antibiotics for UTI annually
Discussion ¶1reviewer’s wording - supportedReviewer 1CPOE prompts significantly reduced empiric extended-spectrum antibiotic use.The primary outcome shows a significant reduction with a rate ratio of 0.83 and p<0.001.Evidence: Primary outcome: rate ratio 0.83 (95% CI 0.77-0.89), P<.001.
the group using CPOE prompts had a 17.4% (95% CI, 11.2%-23.2%) reduction in empiric extended-spectrum days of therapy (rate ratio, 0.83 [95% CI, 0.77-0.89]; P < .001).
Abstractreviewer’s wording - supportedReviewer 1The reduction occurred without increasing ICU transfers or length of stay.Safety outcomes showed no significant differences between groups.Evidence: Safety outcomes: length of stay HR 0.96 (95% CI 0.91-1.01), ICU transfer HR 0.98 (95% CI 0.85-1.12), both non-significant.
Length-of-Stay 6.3 (6.0) 6.5 (6.6) 0.99 (0.95-1.02) ... 0.96 (0.91-1.01) 0.21
Table 2reviewer’s wording - supportedReviewer 2CPOE prompts significantly reduced empiric extended-spectrum antibiotic use in hospitalized patients with UTI.The primary outcome showed a significant reduction with a rate ratio of 0.83 (95% CI, 0.77-0.89; P < .001).Evidence: Primary outcome analysis in Table 2 and Figure 3A.
computerized provider order entry prompts promoting standard-spectrum antibiotics for patients at low risk of infection with MDROs significantly reduced empiric extended-spectrum antibiotic use in hospitalized patients with UTI by 17.4%
Abstractreviewer’s wording - supportedReviewer 2The reduction in antibiotic use did not increase ICU transfers or length of stay.Safety outcomes showed no significant differences between groups.Evidence: Safety outcomes in Table 2 and Figure 3B.
“without increasing intensive care unit transfers or length of stay”
Abstract - supportedReviewer 2The reductions occurred without a change in safety outcomes and persisted despite COVID-19 disruptions.Safety outcomes were not significantly different, and the temporal trends show sustained reductions during the pandemic.Evidence: Safety outcomes in Table 2 and Figure 2A.
the reductions occurred without a change in the safety outcomes of ICU transfer and hospital length of stay and persisted despite COVID-19 pandemic disruptions
Discussion ¶1reviewer’s wording
Premise concern: surrogate not validated for clinical benefit; effect size not shown to be clinically meaningful.
- INADEQUATESurrogate endpointThe primary outcome is empiric extended-spectrum antibiotic days of therapy, a process measure (antibiotic use) rather than a hard clinical outcome. The paper does not establish a validated link between reduced antibiotic use and improved clinical outcomes; safety outcomes (ICU transfer, length of stay) are secondary and show no significant difference. The efficacy claim rests on reducing antibiotic exposure, which is a surrogate for potential patient benefit (e.g., fewer adverse events, less resistance), but no direct evidence links this reduction to improved clinical outcomes.
“The primary outcome was empiric (first 3 days of hospitalization) extended-spectrum antibiotic days of therapy.”
- INADEQUATEEffect sizeThe primary effect is a 17.4% relative reduction in extended-spectrum antibiotic days of therapy, but the absolute reduction is modest (from 392.2 to 326.0 per 1000 empiric days in the intervention group). The paper does not anchor this reduction to a minimal clinically important difference or demonstrate that it translates into meaningful patient benefit. The safety outcomes show no significant difference, so the clinical meaningfulness of the reduction is not established.
“a 17.4% (95% CI, 11.2% - 23.2%) reduction in empiric extended-spectrum days of therapy (rate ratio, 0.83 [95% CI, 0.77 - 0.89]; P < .001)”
Data authenticity concerns
1 finding · worst mediumAn adversarial read for patterns associated with data that may not be genuine: results that look too clean, implausibly large effects, duplicated data or images, and methods that do not match the results reported.
- Other integrity concernAssessed
3 integrity concerns flagged (0 high).
- mediumotherTrial NCT03697096 was first submitted to ClinicalTrials.gov on 2018-10-03, after the registered study start date of 2018-10-01. Retrospective registration means the protocol and outcomes were not on the public record before the study ran, which is what prospective registration exists to establish.
NCT03697096
reviewer’s wording - lowotherThe paper reports a 17.4% reduction in the primary outcome, but the rate ratio is 0.83, which corresponds to a 17% reduction. The difference is due to rounding.
17.4% (95% CI, 11.2%-23.2%) reduction
Abstractreviewer’s wording
Reporting gaps
1 finding · worst mediumRequired detail the manuscript never states — study design, biological variables, ethics approval and consent, key resources, statistical reporting, data and code availability, and overall transparency.
- Data/code availability incompleteAssessed
The introduction cites prior studies on UTI burden, antibiotic overuse, and stewardship strategies, and acknowledges gaps (e.g., few studies address empiric selection before culture results). The rationale for using patient-specific MDRO risk estimates to guide empiric therapy is clearly articulated. Limitations of prior work are implicitly addressed by the novel risk-based prompt approach.
Randomization method is described (1:1 ratio within matched pairs using Mahalanobis distance). Unit of randomization is the hospital (cluster). Blinding is not applicable in this pragmatic trial; the paper does not mention blinding, but it is not required for this design. Power analysis is reported (97% power to detect 12.5% difference). Inclusion/exclusion criteria are pre-specified (noncritically ill adults, UTI discharge codes, exclusions for prison and early ICU transfer). Outlier handling is addressed through sensitivity analyses and pre-specified analysis populations. Controls are the routine stewardship group. Independent replication is not applicable for a single pivotal trial.
Sex is reported for all groups (e.g., 30.5% male overall). Age is reported as mean (SD) and in age bands. Demographics include race/ethnicity, insurance type, and Elixhauser comorbidity counts. Since both sexes are enrolled, sex justification is not applicable. Species/strain and housing conditions are not applicable for a human trial.
“the mean (SD) age was 69.4 (17.9) years, 30.5% were male”
“the mean (SD) age was 69.4 (17.9) years, 30.5% were male”
“Race and ethnicity were included as collected in the HCA electronic health record to address population diversity and generalizability.”
The paper states that the Harvard Pilgrim Health Care Institute IRB provided centralized oversight with reliance agreements, and informed consent was waived. This satisfies both IRB approval and consent handling. Regulatory compliance is implied by adherence to CONSORT and mention of national guidance, though not explicitly named as a framework; however, the IRB statement is adequate.
“Individual informed consent was waived.”
“ClinicalTrials.gov Identifier: NCT03697096”
The intervention is a CPOE prompt system, not a drug or device; the paper describes the algorithm and its components. Statistical software is identified (SAS 9.4, R 4.0.0). No antibodies, cell lines, or organisms are used, so those criteria are not applicable. The CPOE algorithm is described in detail, and the risk model is referenced.
“All analyses were performed using SAS (version 9.4; SAS Institute) or R (version 4.0.0; R Project for Statistical Computing).”
“All analyses were performed using SAS (version 9.4; SAS Institute) or R (version 4.0.0; R Project for Statistical Computing).”
Tests are named (generalized linear mixed-effects models, proportional hazards models). Assumptions are handled by design (clustering, pre-specified models). Exact p-values are reported (e.g., P < .001). Effect sizes with 95% CIs are provided for primary and secondary outcomes. Software is identified. Data presentation includes per-group n and rates; individual data points are not expected for a large trial. Mathematical plausibility checks: the reported percentages and counts appear consistent; no arithmetic errors detected.
The paper states 'Data Sharing Statement: See Supplement 3.' This is a placeholder, not a concrete access route. No repository or accession numbers are provided. Since this is a clinical trial with patient data, managed access is acceptable, but the statement lacks specifics. Code sharing is not mentioned.
Trial registration is provided (NCT03697096). CONSORT guidelines are referenced. All pre-specified outcomes are reported, including safety outcomes. Limitations are explicitly discussed. Conclusions are proportional to the evidence. Funding and conflicts of interest are disclosed.
“Trial Registration ClinicalTrials.gov Identifier: NCT03697096”
“ClinicalTrials.gov Identifier: NCT03697096”
“There are several limitations to this study.”
Registered (1 ID: ClinicalTrials.gov). No reporting guideline cited.
Broken references and links
None found · partly checkedReferences checked against Crossref, OpenAlex and Retraction Watch for retractions and resolvability, plus declared data and code links probed for whether they resolve to content matching the paper.
Nothing surfaced — but not everything feeding this category ran (missing: data/code link verification), so read this as a partial clean bill.
Checked 35 references by DOI: 32 verified — 3 no DOI (shown, not verified).
- NO DOIAntibiotic Resistance Threats in the United States, 2019No DOI in the reference — shown for manual review; not independently verifiable (not a fabrication signal).
- NO DOICore Elements of Hospital Antibiotic Stewardship ProgramsNo DOI in the reference — shown for manual review; not independently verifiable (not a fabrication signal).
- NO DOIOn the generalised distance in statisticsNo DOI in the reference — shown for manual review; not independently verifiable (not a fabrication signal).
Copyediting
4 minorWording, consistency and formatting errors that need correcting before submission.
No major wording or formatting errors. 4 minor suggestions below.
4 copyedit issues flagged: mostly consistency, typo, clarity.
- MINORtypoAuthor list“Naoise G. Nickolay RPh”→ Add comma after 'Nickolay' for consistency with other authors.Minor formatting inconsistency.
- MINORconsistencyResults, Patient Characteristics“27 907”→ Use consistent spacing: '27,907'.Inconsistent number formatting.
- MINORconsistencyResults, Patient Characteristics“27 907”→ Use comma as thousands separator: '27,907'.Inconsistent number formatting.
- MINORclarityMethods, Statistical Analysis“The a priori statistical analytic plan is provided in Supplement 1.”→ Consider moving this sentence to the beginning of the Statistical Analysis section for clarity.Minor organizational suggestion.
The published work is robust and credible, with strong methodology and transparent reporting. An informed reader should weigh the vague data sharing statement and the retrospective registration as minor limitations, but neither undermines the validity of the findings. No erratum is warranted based on the identified issues.
- 1.HIGHdata codeReplace the placeholder 'Data Sharing Statement: See Supplement 3' with a concrete statement specifying a managed-access platform (e.g., YODA, Vivli) or a data-access committee with conditions and timeframe.A vague data sharing statement is a reporting gap that limits reproducibility and is a common reviewer concern.
- 2.HIGHdata codeAdd a statement about code availability, even if it is 'available on request', to clarify whether analysis scripts are shared.Code sharing is not mentioned, and providing it would enhance reproducibility.
- 3.HIGHreportingClarify the blinding status of outcome assessors in the Methods section, even if the intervention is unblinded to clinicians.Reviewer 2 flagged blinding as inadequately reported; a brief statement would resolve this.
- 4.HIGHreportingTemper the claim that the intervention could reduce unnecessary antibiotics in up to 200,000 US adults annually, as this extrapolation is not directly demonstrated by the trial.The claim audit flagged this as overstated; the conclusion should be limited to what the trial directly shows.
- 5.HIGHreportingTemper the claim that the intervention is a viable strategy to reduce extended-spectrum antibiotics in up to 200,000 adults annually, as it extrapolates beyond the study population.The claim audit flagged this as overstated; the conclusion should be limited to what the trial directly shows.
- 6.MEDIUMreportingAdd a note in the Methods or Discussion acknowledging that trial registration was retrospective (submitted after the study start date).The integrity check flagged this; transparency about registration timing is important for readers.
- 7.MEDIUMreportingConsider providing a CONSORT flow diagram in the main text to improve transparency of participant flow.Reviewer 2 suggested this; it would enhance reporting completeness.
- 8.MEDIUMstatisticsInclude a note on the handling of missing data for the primary outcome in the statistical analysis section.Reviewer 2 suggested this; it would clarify the analysis approach.
- 9.MEDIUMreportingSpecify the version of the MEDITECH EHR system used, as it may affect generalizability.Reviewer 2 suggested this; it would improve reproducibility.
- 10.MEDIUMreportingProvide more detail on the validation of the MDRO risk prediction model, including discrimination and calibration metrics.Reviewer 2 suggested this; it would strengthen the methods.
- 11.MEDIUMreportingConsider reporting the number of patients who were excluded due to ICU transfer within 2 days, to clarify the cohort definition.Reviewer 2 suggested this; it would improve transparency.
- 12.LOWcopyeditAdd a comma after 'Nickolay' in the author list for consistency with other authors.Minor formatting inconsistency flagged by copyedit.
- 13.LOWcopyeditUse consistent number formatting: change '27 907' to '27,907' in the Results section.Inconsistent number formatting flagged by copyedit.
- 14.LOWcopyeditConsider moving the sentence 'The a priori statistical analytic plan is provided in Supplement 1' to the beginning of the Statistical Analysis section for clarity.Minor organizational suggestion from copyedit.
The star rating is the report’s one-glance summary. Every paper starts at 5★ and loses stars for the concrete problems the review finds — so a rating is never a vague average, it’s a running total you can read line by line under “How this rating was calculated.”
- Reporting — 8 dimensionseach dimension that fully fails−½★
- each dimension partially met−¼★
- Statistics · Integrity · Claimseach serious problem−1★
- each medium problem−½★
- Citationseach retracted or unverifiable reference−¼★
- Copyeditonly when the manuscript needs a full edit−½★
The rating never drops below 1★, and a demonstrable critical failure (an impossible statistic, a proven ethics violation) caps it at 1★ on its own — so the stars can never look healthy when the verdict is CRITICAL.
The rating draws on a panel of agents. Three independent Kaimen Rigor reviewers grade the eight dimensions below across several independent passes (the shown verdict is their majority vote — steadier than any single run), isolate the paper’s major claims and check its own evidence backs them, and flag integrity concerns. Alongside them, a citation agent resolves every reference against Crossref, OpenAlex, and Retraction Watch; a statistics agent recomputes reported tests; and rule-based checks verify that declared data/code links actually resolve. Full text is required — an abstract-only submission is not analyzed.
Graded against NIH, MDAR, ARRIVE 2.0, CONSORT, EQUATOR, and RRID guidelines. A dimension that doesn’t apply to the study type is skipped, never penalized.
This Kaimen Rigor review is model-assisted and is not a substitute for formal expert review. It complements human evaluation by surfacing potential methodological concerns — verify each finding against the source.