De novo design of RNA pseudoknots with deep learning

Name: Verified Rigor Record — De novo design of RNA pseudoknots with deep learning
Item: De novo design of RNA pseudoknots with deep learning
Rating: 83
Author: Αlpha¹

Townley, J.; Kladwang, W.; Baker, D.; Blair, H. M.; Choe, C.; El Nesr, G.; Favor, A.; Fisker, E.; Haack, D. B.; He, S.; Hingey, J.; Huang, R.; Huang, P.-S.; Joshi, C. K.; Karagianes, T. G.; Kubaney, A.; Lio, P.; Mancino, A.; Romano, J.; Rudolfs, B.; Spellmon, N.; Toor, N.; Wu, V.; Yu, Z.; Eterna players, ; Das, R.

Paper source

Read PDF

Αlpha¹ Rigor Review

Rigor Review Verified

Critical

High-priority

Action items to resolve · Critical / High

Confidence: moderateEngine: 7.6.0Source: full text

Not verified in this report

References were not verified against Crossref/OpenAlex.
No reported statistical tests were found to recompute.

This Alpha1 Rigor Review uses AI Rigor Reviewers trained on a curated corpus of high-fidelity and retracted papers, with expert supervision and curation. It can still make mistakes; verify each finding against the source before relying on it.

The paper presents a well-motivated AI benchmark for RNA pseudoknot design with solid experimental validation. The scientific premise and data/code availability are strong, but the study design and statistical reporting have notable gaps (missing power analysis, unverified assumptions, p-values as thresholds, unreported outlier handling) that a reader should weigh when interpreting the robustness of comparisons between AI methods.

Evaluated based on full text with three independent rigor reviewers and a copyedit pass. Dimensions N/A: biological variables and ethical approvals (no animal/human subjects). The three reviewers broadly agreed on most dimensions, with divergences on study design (pass vs warn) and statistical analysis (pass vs warn) that were resolved by weighing specific verifiable evidence.

9 major claims checked against the paper's own evidence: all adequately supported.

supportedAbstractReviewer 1
AI methods matched experienced human designers in solving most blind challenges.
The paper provides extensive evidence from SHAPE, M2R-seq, and cryo-EM experiments showing that AI methods achieved comparable performance to human designers across multiple rounds.
Evidence: Figures 1-3 show OpenKnot scores and M2R-seq recovery rates for AI and human designs, with statistical tests showing no significant difference.
“In an Eterna competition involving 57 pseudoknots, generative AI methods matched experienced human designers in solving most blind challenges, evaluated by single-nucleotide-resolution chemical mapping, compensatory mutagenesis, and cryogenic electron microscopy.”
supportedAbstractReviewers 1, 2, 3
AI-generated molecules with accurate secondary structures formed well-ordered 3D folds stabilized by noncanonical tertiary interactions not modeled during design.
Cryo-EM structures of three AI designs confirmed the formation of all target stems and revealed noncanonical interactions, as shown in Figure 4.
Evidence: Figure 4 shows cryo-EM maps and models for three AI designs, with insets highlighting noncanonical interactions.
“Unexpectedly, AI-generated molecules with accurate secondary structures formed well-ordered 3D folds stabilized by noncanonical tertiary interactions not modeled during design.”
supportedAbstractReviewers 1, 2
Success was guided by a RNet foundation model trained on prior chemical mapping data.
The paper describes how RNet was used to filter designs and improve performance from Round 1 to Round 2, and was integrated into AI methods.
Evidence: Results show that RNet-based filtering improved OpenKnot scores, and AI methods that incorporated RNet performed better in Round 2.
“Success was guided by a RNet foundation model trained on prior chemical mapping data, suggesting that some difficult RNA design tasks may be tractable without first solving RNA 3D structure prediction.”
supportedDiscussion, paragraph 1Reviewer 1
De novo design of novel RNA pseudoknots is now achievable through deep learning.
The paper demonstrates successful design of 57 pseudoknots, including novel targets, with high accuracy validated by multiple experimental methods.
Evidence: Results from Rounds 3 and 4 show that AI methods achieved OpenKnot scores >90 on 19/20 targets, and M2R-seq confirmed stem formation.
“Reliable de novo design of complex RNA pseudoknots is now achievable through deep learning.”
supportedAbstract, ResultsReviewers 2, 3
Intricate RNA structures can be generated with current deep learning tools through accurate de novo design of pseudoknot secondary structures.
The paper presents extensive experimental evidence (SHAPE, M2R-seq, cryo-EM) showing that AI-designed pseudoknots match target secondary structures with high accuracy.
Evidence: OpenKnot scores >90 for 19/20 targets in Rounds 3 and 4, M2R-seq stem-wise recovery >80% for 19/20 targets, cryo-EM structures confirming all 7 stems.
“show that intricate RNA structures can be generated with current deep learning tools through accurate de novo design of pseudoknot secondary structures.”
supportedAbstract, ResultsReviewer 2
Generative AI methods matched experienced human designers in solving most blind challenges.
The paper shows that AI methods achieved similar success rates as Eterna participants across multiple rounds, with no statistically significant difference in Rounds 3 and 4.
Evidence: OpenKnot scores >90 on at least 80% of targets for both AI and human designers; no significant difference (p > 0.05) in Rounds 3 and 4.
“generative AI methods matched experienced human designers in solving most blind challenges”
supportedAbstract, DiscussionReviewer 2
Some difficult RNA design tasks may be tractable without first solving RNA 3D structure prediction.
The paper successfully designed complex pseudoknots without relying on 3D structure prediction, using only secondary structure targets and RNet-based evaluation.
Evidence: AI methods achieved high success rates without input 3D structures; cryo-EM shows well-ordered folds that were not predicted by AlphaFold 3.
“some difficult RNA design tasks may be tractable without first solving RNA 3D structure prediction.”
supportedAbstractReviewer 3
In an Eterna competition involving 57 pseudoknots, generative AI methods matched experienced human designers in solving most blind challenges.
Results show that AI methods (MPNN-fixbb, gRNAde, Struct2SeQ) achieve similar success rates to Eterna participants, with no statistically significant difference in Round 2 and beyond.
Evidence: Figure 2C shows both AI and Eterna methods achieving >90% success on most targets; statistical tests show p>0.05 for differences between top methods.
“generative AI methods matched experienced human designers in solving most blind challenges”
supportedAbstractReviewer 3
Success was guided by a RNet foundation model trained on prior chemical mapping data, suggesting that some difficult RNA design tasks may be tractable without first solving RNA 3D structure prediction.
The paper shows that RNet predictions correlate with experimental outcomes, and AI methods using RNet (e.g., Struct2SeQ) perform well, supporting the claim that 3D structure prediction is not strictly necessary for secondary structure design.
Evidence: Text states RNet gave simulated scores that reproduced experimental scores (Results, paragraph 3); Struct2SeQ, guided purely by RNet, achieved high performance.
“Success was guided by a RNet foundation model trained on prior chemical mapping data, suggesting that some difficult RNA design tasks may be tractable without first solving RNA 3D structure prediction.”

Rigor

Eight Dimensions3 pass · 3 warn

9 major claims9 supported

9 copyedit issues flagged: mostly typo, consistency, punctuation.

Reproducibility

6 data/code links checked; 6 live.

datahttps://rmdb.stanford.eduLIVEHTTP 200
Resolved page looks like data.
datahttps://rmdb.stanford.edu/detail/OK45LIB_2A3_0000LIVEHTTP 200
Resolved page looks like data.
datahttps://rmdb.stanford.edu/detail/OK6LIB_2A3_0000LIVEHTTP 200
Resolved page looks like data.
datahttps://rmdb.stanford.edu/detail/OK7ALIB_2A3_0000LIVEHTTP 200
Resolved page looks like data.
datahttps://rmdb.stanford.edu/detail/OK7BLIB_2A3_0000LIVEHTTP 200
Resolved page looks like data.
datahttps://rmdb.stanford.edu/detail/OK7BLIB_2A3_0001LIVEHTTP 200
Resolved page looks like data.

Alpha¹ Rigor Action ItemsFinal pass before journal submission

The published work is robust in its central claims — de novo RNA pseudoknot design is demonstrated with multiple independent validation methods. However, informed readers should weigh the unverified statistical test assumptions, the use of threshold p-values rather than exact values, and the lack of pre-specified target selection criteria when evaluating the strength of the AI-to-human comparisons. The paper would benefit from a post-publication correction adding exact p-values, effect sizes with confidence intervals, and a statement on outlier handling in the SHAPE analysis.

1.
HIGHstatistics
Replace all p-value thresholds (e.g., 'p > 0.05', 'p < 0.05') with exact p-values (e.g., 'p = 0.12', 'p = 0.003') in the Results section.
Threshold p-values obscure the strength of evidence; exact p-values are required for rigorous interpretation and are a standard reporting expectation.
2.
HIGHstatistics
Provide effect sizes (e.g., difference in mean OpenKnot scores) with 95% confidence intervals for all key method comparisons in Results.
Effect sizes quantify practical significance and confidence intervals convey precision, both of which are missing and are critical for readers to assess the reported differences.
3.
HIGHstatistics
Add a statement in the Methods specifying the statistical software and packages used (e.g., R, SciPy, or custom scripts).
Reproducibility of statistical analyses requires naming the software and version so readers can verify or replicate the calculations.
4.
HIGHrigor
Add a section in Methods describing how outliers in SHAPE data were handled (e.g., normalization method, clipping thresholds, exclusion criteria).
Outlier handling directly affects the OpenKnot scores and method rankings; omitting it leaves the reproducibility of the central metric uncertain.
5.
HIGHrigor
Add a power analysis or sample-size rationale in Methods justifying the number of targets and submissions per round.
Without a power analysis, readers cannot assess whether the study was adequately sized to detect meaningful differences among methods.
6.
HIGHrigor
Pre-specify and state the inclusion/exclusion criteria for target selection and design submissions in Methods (e.g., minimum sequence identity, OpenKnot score cutoffs).
Post hoc selection risks bias; pre-specified criteria improve the objectivity and reproducibility of the benchmark.
7.
HIGHreporting
Specify the exact SHAPE chemical reagent used (e.g., 1M7, NMIA) and provide supplier/catalog numbers in Methods.
The SHAPE reagent is a key resource without which the assay cannot be independently replicated; current text only mentions 'SHAPE' generically.
8.
HIGHreporting
Add a code availability statement in Data and materials availability, including a link to a public repository (e.g., GitHub) for design pipelines and analysis scripts.
Code sharing is essential for reproducibility of computational methods; its absence is a notable gap in an otherwise well-deposited data paper.
9.
HIGHreporting
Identify software tools (RNet, gRNAde, MPNN-fixbb, Struct2Seq) with version numbers in Methods or a table.
Version-specific software behavior can affect results; without version numbers, exact replication of the computational pipeline is not possible.
10.
HIGHstatistics
Verify and report the assumptions for each statistical test used (e.g., for a paired binomial test, report that the independence and fixed-probability assumptions hold).
Unverified assumptions can invalidate statistical conclusions; explicit verification documentation is a standard rigor criterion.
11.
HIGHrigor
Distinguish between biological and technical replicates in Methods, e.g., state whether each designed sequence is a biological replicate or if multiple SHAPE measurements of the same sequence form technical replicates.
Replicate type affects error calculation and statistical power; the current text mentions 'up to 20 submissions' without clarifying replicate structure.
12.
MEDIUMcopyedit
Standardize the method name 'Struct2SeQ' to 'Struct2Seq' throughout the manuscript for consistency.
Inconsistent capitalization of a key AI method name can confuse readers and suggests incomplete proofreading.
13.
MEDIUMcopyedit
Fix the missing closing parenthesis after 'ProteinMPNN' in the Results section ('Benchmarking AI design against experienced humans').
The missing parenthesis is a typographical error that interrupts readability.
14.
MEDIUMcopyedit
Replace the semicolon before 'and' with a comma in the cryo-EM accession numbers sentence in Data and materials availability.
The semicolon violates standard punctuation rules for lists and may be flagged as a grammatical error.
15.
MEDIUMcopyedit
Rephrase the awkward sentence in Results paragraph 5: 'The exception was Struct2SeQ-SHAPE which significantly outperformed...' to 'The exception was Struct2SeQ-SHAPE, which significantly outperformed...; however, this method's designs were submitted after release of initial results, allowing additional development time.'
The misplaced 'however' creates an ambiguous causal relationship that can mislead readers about the timing of submissions.
16.
MEDIUMcopyedit
Change 're-design' to 'redesign' (no hyphen) in Introduction paragraph 1 for consistency with standard usage.
Minor hyphenation inconsistency that would be caught by careful copyediting.
17.
LOWcopyedit
In Acknowledgements, replace the placeholder text 'supporting , and ; all AI-assisted output' with the intended figure numbers (e.g., 'supporting Fig. S1, S2, and S3').
The placeholder suggests an incomplete section that would confuse readers if published as is.
18.
LOWcopyedit
Separate the preprint metadata from the author list at the top of the manuscript to improve formatting clarity.
The concatenated author/ORCID/role text is likely a formatting artifact but could be confusing in the final preprint or journal version.

Full text (45,520 chars)Analyzed: Jul 6, 2026

Αlpha¹ assesses methodological rigor, not the importance of the findings. See how we calibrate →