Accurate protein stability prediction for small domains using mega-scale experiments

Name: Verified Rigor Record — Accurate protein stability prediction for small domains using mega-scale experiments
Item: Accurate protein stability prediction for small domains using mega-scale experiments
Rating: 89
Author: Αlpha¹

Cho, Y.; Tsuboyama, K.; Litberg, T. J.; Jung, M. D.; Obisesan, A.; Wang, Q.; Phoumyvong, C. M.; Thibeault, J.; Ovchinnikov, S.; Rocklin, G. J.

Paper source

Read PDF

Αlpha¹ Rigor Review

Rigor Review Verified

Critical

High-priority

Action items to resolve · Critical / High

Confidence: moderateEngine: 7.6.0Source: full text

Not verified in this report

No reported statistical tests were found to recompute.

This Alpha1 Rigor Review uses AI Rigor Reviewers trained on a curated corpus of high-fidelity and retracted papers, with expert supervision and curation. It can still make mistakes; verify each finding against the source before relying on it.

The paper combines large-scale experimental stability measurements with machine learning to develop predictors, with strong scientific premise and study design. However, data availability is compromised by an invalid repository link, and key resource reporting (software versions, reagent details) is incomplete; a conflict of interest statement is also missing.

Three independent reviewers were synthesized; all agreed on most dimensions but diverged on study design, key resources, and data code availability, with the final judgments reflecting a conservative weighing of the evidence. Biological variables and ethical approvals were N/A due to the in vitro/computational nature.

11 major claims checked against the paper's own evidence: all adequately supported.

partialResults, paragraph 7Reviewer 1
The models show promising performance on larger proteins, despite training on domains <80 residues.
Performance on the S1724 benchmark (larger proteins) is variable, with lower accuracy for very stable domains (>10 kcal/mol). The authors acknowledge this limitation.
Evidence: Figure 4a-f and Discussion note reduced accuracy for large/stable proteins.
“Augmented ESM3ΔG performance varied from protein to protein, with ΔG and ΔΔG accuracy lower for the groups of mutants from the largest and most stable wild-type domains”
supportedAbstractReviewer 1
SaProtΔG and ESM3ΔG accurately predict absolute folding stability for small domains with RMSE of 0.8 kcal/mol and Spearman correlation of 0.88.
The reported test set performance (n=3283, 30% identity filtered) directly supports this claim.
Evidence: Figure 1f,g show scatter plots with RMSE 0.80 and Spearman 0.88/0.87 for the two models.
“SaProtΔG and ESM3ΔG, which accurately predict absolute folding stability for small domains with root mean squared error of 0.8 kcal/mol over a 6 kcal/mol range (Spearman rank correlation of 0.88)”
supportedAbstractReviewers 1, 2, 3
These predictors show high accuracy at predicting effects of substitutions, insertions, and deletions.
Benchmarks against the Megascale dataset (28,172 mutants) and MGnify indels (106,210 variants) show Spearman correlations >0.7 and classification accuracy of 83.7% for stabilizing indels.
Evidence: Figure 3a-f and associated text report performance on point mutations and indels.
“These predictors show high accuracy at predicting effects of substitutions, insertions, and deletions”
supportedAbstractReviewer 1
The predictors successfully identify global trends toward higher stability in thermophilic organisms.
Analysis of 583 CATH families across organisms with varying OGT shows a positive slope of predicted ΔG vs. OGT, with an organism-level R² of 0.65.
Evidence: Figure 5d-h and accompanying text report the correlation between predicted ΔG and OGT.
“successfully identify global trends toward higher stability in thermophilic organisms”
supportedAbstractReviewer 1
The predictors improve discrimination of stable and unstable computationally designed proteins.
ESM3ΔG achieves AUC 0.76 on Rosetta designs (better than pLDDT and ProteinMPNN), and correlates strongly (Spearman 0.88) with CD measurements on de novo designs.
Evidence: Figure 6a-f and text show improved discrimination on three design sets.
“improve discrimination of stable and unstable computationally designed proteins”
supportedAbstract and ResultsReviewer 2
SaProtΔG and ESM3ΔG accurately predict absolute folding stability for small domains with RMSE 0.8 kcal/mol and Spearman correlation of 0.88.
The test set of 3,283 sequences with low sequence identity to training shows the stated performance. Figures and metrics support the claim.
Evidence: Figure 1f,g and results text: 'SaProtΔG and ESM3ΔG achieved Spearman correlations of 0.88 and 0.87, respectively, with RMSE values of 0.80 kcal mol−1'.
“SaProtΔG and ESM3ΔG, which accurately predict absolute folding stability for small domains with root mean squared error of 0.8 kcal/mol over a 6 kcal/mol range (Spearman rank correlation of 0.88).”
supportedResults, section 'Predicted folding stability correlates with organismal optimal growth temperature'Reviewer 2
The models identify global trends toward higher stability in thermophilic organisms.
The OGT analysis shows a positive slope between predicted ΔG and optimal growth temperature across 583 CATH families; R²=0.65 at organism level. Evidence is strong.
Evidence: Figure 5 and text: 'positive slopes were observed in both small and large domains ... organism-level R²=0.65'.
“Augmented ESM3ΔG predicted a linear slope of 2.43 kcal/mol per 100 °C.”
supportedResults, section 'Stability filtering improves protein design'Reviewers 2, 3
The models improve discrimination of stable and unstable computationally designed proteins.
ESM3ΔG outperforms AF2 pLDDT and ProteinMPNN at discriminating stable vs unstable Rosetta designs (AUC 0.76) and correlates well with experimental ΔG on de novo designs (Spearman 0.88).
Evidence: Figure 6a-d and text: 'ESM3ΔG showed a strong ability to discriminate experimentally stable from unstable designs (AUC 0.76 ± 0.02)'.
“ESM3ΔG showed a strong ability to discriminate experimentally stable from unstable designs (AUC 0.76 ± 0.02).”
supportedAbstractReviewer 3
SaProtΔG and ESM3ΔG accurately predict absolute folding stability for small domains with RMSE of 0.8 kcal/mol over a 6 kcal/mol range (Spearman rank correlation of 0.88).
The paper presents evidence from the MGnify test set (Figure 2) showing these metrics, and the methods are clearly described.
Evidence: Figure 2 shows scatter plots with RMSE=0.80 and Spearman=0.88 for SaProtΔG.
“SaProtΔG and ESM3ΔG, which accurately predict absolute folding stability for small domains with root mean squared error of 0.8 kcal/mol over a 6 kcal/mol range (Spearman rank correlation of 0.88).”
supportedAbstract and Results section 'Predicted folding stability correlates with organismal optimal growth temperature'Reviewer 3
The models successfully identify global trends toward higher stability in thermophilic organisms.
The correlation with optimal growth temperature (OGT) using the TED dataset (Figure 5) shows positive slopes for most CATH families, supporting this claim.
Evidence: Figure 5 shows box plots and slopes of predicted ΔG vs OGT for multiple domain families, with a global R²=0.65 at organism level.
“successfully identify global trends toward higher stability in thermophilic organisms”
supportedDiscussion, first sentenceReviewer 3
Combining large-scale experimental data and modern deep learning models enables accurate folding stability predictions across a large and diverse space of protein sequences and folds.
The study demonstrates the value of the MGnify dataset and fine-tuning approach, with generalization to diverse sequences and structural classes as shown in multiple benchmarks.
Evidence: Training on MGnify data outperforms previous datasets; models generalize to new folds and longer proteins (ThermoMut, nanobodies).
“Our results illustrate that combining large-scale experimental data and modern deep learning models can enable accurate folding stability predictions across a large and diverse space of protein sequences and folds.”

4 integrity concerns flagged (0 high).

Rigor

Eight Dimensions4 pass · 2 warn

11 major claims1 partial10 supported

9 copyedit issues flagged (1 major): mostly consistency, grammar, other.

Checked 109 references: 97 verified — 12 not checked.

Reproducibility

7 data/code links checked; 7 live.

datahttps://forms.gle/4ZnXZSnTBvaykkAi9LIVEHTTP 200
Resolved page looks like data.
codeGitHubLIVEHTTP 200
https://github.com/yehlincho/absolute-stability-predictor
Resolves to GitHub (code repository).
codeGitHubLIVEHTTP 200
https://github.com/Rocklin-Lab/mhdx_analysis/tree/main/scripts/feature_extraction
Resolves to GitHub (code repository).
codeGitHubLIVEHTTP 200
https://github.com/evo-design/protein-dpo
Resolves to GitHub (code repository).
codeGitHubLIVEHTTP 200
https://github.com/HParklab/IFUM
Resolves to GitHub (code repository).
codeGitHubLIVEHTTP 200
https://github.com/microsoft/bioemu
Resolves to GitHub (code repository).
dataHugging FaceLIVEHTTP 200
https://huggingface.co/Yehlin/absolute-stability
Resolves to Hugging Face (data repository).

Alpha¹ Rigor Action ItemsFinal pass before journal submission

In a post-publication audit, the published work is methodologically sound but has two actionable gaps: the Zenodo data link is a broken Google Forms URL, and the funding/COI statement is absent. An informed reader should weigh these as they affect reproducibility and transparency; an erratum or repository correction is warranted.

1.
HIGHdata code
Replace the invalid Zenodo link (https://forms.gle/...) with a valid DOI from a recognized repository (e.g., Zenodo, Dryad) for the MGnify Stability dataset (Methods, Library design and clustering).
The current link is a Google Forms URL, not a persistent repository; without a valid DOI, the data are effectively inaccessible.
2.
HIGHdata code
Deposit raw NGS sequencing reads in a public archive (e.g., SRA, GEO) and include the accession numbers in the Data Availability section.
Raw sequencing data are essential for independent verification of the processing pipeline; their absence limits reproducibility.
3.
HIGHreporting
Add a funding statement and conflict-of-interest disclosure to the manuscript (e.g., at the end of the main text, before references).
Most journals require these statements; their absence is a reporting gap that could be flagged by readers or editors.
4.
HIGHstatistics
Report exact p-values for the main correlation coefficients (Spearman, Pearson) alongside the bootstrap confidence intervals (Results, Figures).
While bootstrap CIs are informative, exact p-values are standard and allow readers to assess significance directly.
5.
HIGHother
Provide version numbers for all major software tools (e.g., SaProt, ESM3, MMseqs2, JackHMMER, PyTorch) and the Python environment (Methods, Fine-tuning structure-informed models).
Version information is critical for reproducibility; without it, exact computational conditions cannot be recreated.
6.
HIGHrigor
Correct the dataset total: the sum of training (960,216), test (3,283), and validation (2,357) is 965,856, not 966,000 as stated (Methods, Library design and clustering).
This minor arithmetic inconsistency could undermine confidence in the reporting; a correction clarifies the numbers.
7.
HIGHrigor
Explicitly classify replicates as biological or technical in the experimental design (Results, paragraph 2).
The paper mentions replicate experiments but does not distinguish between independent biological preparations and technical repeats, which is important for variance interpretation.
8.
HIGHother
Provide vendor names and catalog numbers for all key reagents (trypsin, chymotrypsin, GdnHCl, Jasco CD instrument, etc.) in the Methods.
Full reagent identification enables others to replicate the experimental conditions precisely.
9.
MEDIUMreporting
Add confidence intervals for RMSE in all benchmarking results, not only for selected analyses (Results, Figures 2-4).
Consistent reporting of uncertainty across all metrics strengthens the statistical presentation.
10.
MEDIUMcopyedit
Spell out the abbreviation 'D.I.T.' in the Methods (Circular dichroism) or provide a definition (e.g., 'digital integration time').
Undefined abbreviations can confuse readers; clarity is essential for a method described in detail.
11.
MEDIUMcopyedit
Correct the redundant phrase in Results: change 'SaProtΔG and ESM3ΔG both models achieved' to 'SaProtΔG and ESM3ΔG both achieved' or 'both models achieved'.
The current wording is grammatically awkward and could be misinterpreted.
12.
MEDIUMcopyedit
Complete the incomplete figure reference in Methods (cDNA display proteolysis) by adding the specific figure number (e.g., 'shown in Supplementary Figure X').
An incomplete reference disrupts the reader's ability to cross-reference data.
13.
LOWcopyedit
Add a space in '298K' to read '298 K' in the Abstract for consistency with the rest of the text.
Minor formatting consistency improves professional appearance.

Full text (105,132 chars)Analyzed: Jul 6, 2026

Αlpha¹ assesses methodological rigor, not the importance of the findings. See how we calibrate →