{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T04:57:52Z","timestamp":1780635472902,"version":"3.54.1"},"reference-count":37,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2025,9,1]],"date-time":"2025-09-01T00:00:00Z","timestamp":1756684800000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,8,31]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>In silico perturbation models, computational methods that can predict cellular responses to perturbations, present an opportunity to reduce the need for costly and time-intensive in vitro experiments. Many recently proposed models predict high-dimensional cellular responses, such as gene or protein expression to perturbations such as gene knockout or drugs. However, evaluating in silico performance has largely relied on metrics such as $R^{2}$, which assess overall prediction accuracy but fail to capture biologically significant outcomes like the identification of differentially expressed (DE) genes. In this study, we present a novel evaluation framework that introduces the AUPRC metric to assess the precision and recall of DE gene predictions. By applying this framework to both single-cell and pseudo-bulked datasets, we systematically benchmark simple and advanced computational models. Our results highlight a significant discrepancy between $R^{2}$ and AUPRC, with models achieving high $R^{2}$ values but struggling to identify DE genes, as reflected in their low AUPRC values. This finding underscores the limitations of traditional evaluation metrics and the importance of biologically relevant assessments. Our framework provides a more comprehensive understanding of model capabilities, advancing the application of computational approaches in cellular perturbation research.<\/jats:p>","DOI":"10.1093\/bib\/bbaf426","type":"journal-article","created":{"date-parts":[[2025,9,1]],"date-time":"2025-09-01T14:24:12Z","timestamp":1756736652000},"source":"Crossref","is-referenced-by-count":2,"title":["AUPRC: a metric for evaluating the performance of <i>in-silico<\/i> perturbation methods in identifying differentially expressed genes"],"prefix":"10.1093","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3774-4495","authenticated-orcid":false,"given":"Hongxu","family":"Zhu","sequence":"first","affiliation":[{"name":"Department of Biostatistics and Data Science, The University of Texas Health Science Center at Houston School of Public Health , 1200 Pressler St., 77030, TX ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amir","family":"Asiaee","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University Medical Center , 2525 West End Avenue, 37203, TN ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leila","family":"Azinfar","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, Vanderbilt University Medical Center , 2525 West End Avenue, 37203, TN ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jun","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Bioinformatics and Computational Biology, The University of Texas MD Anderson Cancer Center , 7007 Bertner Ave., 77030, TX ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Han","family":"Liang","sequence":"additional","affiliation":[{"name":"Department of Bioinformatics and Computational Biology, The University of Texas MD Anderson Cancer Center , 7007 Bertner Ave., 77030, TX ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ehsan","family":"Irajizad","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, The University of Texas MD Anderson Cancer Center , 7007 Bertner Ave., 77030, TX ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kim-Anh","family":"Do","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, The University of Texas MD Anderson Cancer Center , 7007 Bertner Ave., 77030, TX ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James P","family":"Long","sequence":"additional","affiliation":[{"name":"Department of Biostatistics, The University of Texas MD Anderson Cancer Center , 7007 Bertner Ave., 77030, TX ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,9,1]]},"reference":[{"key":"2025090110240872400_ref1","doi-asserted-by":"crossref","first-page":"677","DOI":"10.1016\/j.cell.2006.06.044","article-title":"Matrix elasticity directs stem cell lineage specification","volume":"126","author":"Engler","year":"2006","journal-title":"Cell"},{"key":"2025090110240872400_ref2","doi-asserted-by":"publisher","first-page":"721","DOI":"10.1038\/nrc1187","article-title":"Targeting HIF-1 for cancer therapy","volume":"3","author":"Semenza","year":"2003","journal-title":"Nat Rev Cancer"},{"key":"2025090110240872400_ref3","doi-asserted-by":"publisher","first-page":"1032","DOI":"10.1002\/bit.21003","article-title":"Characterization of an experimental miniature bioreactor for cellular perturbation studies","volume":"95","author":"Aboka","year":"2006","journal-title":"Biotechnol Bioeng"},{"key":"2025090110240872400_ref4","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1038\/msb.2008.53","article-title":"Models from experiments: combinatorial drug perturbations of cancer cells","volume":"4","author":"Nelander","year":"2008","journal-title":"Mol Syst Biol"},{"key":"2025090110240872400_ref5","doi-asserted-by":"publisher","first-page":"19306","DOI":"10.1073\/pnas.0807038105","article-title":"Characterizing heterogeneous cellular responses to perturbations","volume":"105","author":"Slack","year":"2008","journal-title":"Proc Natl Acad Sci"},{"key":"2025090110240872400_ref6","doi-asserted-by":"publisher","first-page":"6069","DOI":"10.1016\/j.biomaterials.2014.03.087","article-title":"Simulation and analysis of cellular internalization pathways and membrane perturbation for graphene nanosheets","volume":"35","author":"Mao","year":"2014","journal-title":"Biomaterials"},{"key":"2025090110240872400_ref7","doi-asserted-by":"publisher","first-page":"e1003290","DOI":"10.1371\/journal.pcbi.1003290","article-title":"Perturbation biology: inferring signaling networks in cellular systems","volume":"9","author":"Molinelli","year":"2013","journal-title":"PLoS Comput Biol"},{"key":"2025090110240872400_ref8","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1016\/j.cell.2018.11.029","article-title":"A genome-wide framework for mapping gene regulation via cellular genetic screens","volume":"176","author":"Gasperini","year":"2019","journal-title":"Cell"},{"key":"2025090110240872400_ref9","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1016\/j.cels.2017.11.001","article-title":"The library of integrated network-based cellular signatures NIH program: system-level cataloging of human cells response to perturbations","volume":"6","author":"Keenan","year":"2018","journal-title":"Cell Syst"},{"key":"2025090110240872400_ref10","doi-asserted-by":"publisher","first-page":"e10798","DOI":"10.15252\/msb.202110798","article-title":"Ultra-high sensitivity mass spectrometry quantifies single-cell proteome changes upon perturbation","volume":"18","author":"Brunner","year":"2022","journal-title":"Mol Syst Biol"},{"key":"2025090110240872400_ref11","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1038\/s41592-019-0392-0","article-title":"Multiplexed detection of proteins, transcriptomes, clonotypes and crispr perturbations in single cells","volume":"16","author":"Mimitou","year":"2019","journal-title":"Nat Methods"},{"key":"2025090110240872400_ref12","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1111\/imr.12618","article-title":"The aim 2 inflammasome: sensor of pathogens and cellular perturbations","volume":"281","author":"Lugrin","year":"2018","journal-title":"Immunol Rev"},{"key":"2025090110240872400_ref13","article-title":"Scperturb: information resource for harmonized single-cell perturbation data","author":"Peidli","year":"2022"},{"key":"2025090110240872400_ref14","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1016\/j.cels.2020.11.013","article-title":"Cellbox: interpretable machine learning for perturbation biology with application to the design of cancer combination therapy","volume":"12","author":"Yuan","year":"2021","journal-title":"Cell Syst"},{"key":"2025090110240872400_ref15","doi-asserted-by":"publisher","first-page":"i610","DOI":"10.1093\/bioinformatics\/btaa800","article-title":"Conditional out-of-distribution generation for unpaired data using transfer VAE","volume":"36","author":"Lotfollahi","year":"2020","journal-title":"Bioinformatics"},{"key":"2025090110240872400_ref16","article-title":"Compositional perturbation autoencoder for single-cell response modeling","author":"Lotfollahi","year":"2021","journal-title":"BioRxiv"},{"key":"2025090110240872400_ref17","doi-asserted-by":"publisher","first-page":"715","DOI":"10.1038\/s41592-019-0494-8","article-title":"Scgen predicts single-cell perturbation responses","volume":"16","author":"Mohammad Lotfollahi","year":"2019","journal-title":"Nat Methods"},{"key":"2025090110240872400_ref18","first-page":"2022","article-title":"Gears: predicting transcriptional outcomes of novel multi-gene perturbations","author":"Roohani","year":"2022","journal-title":"BioRxiv"},{"key":"2025090110240872400_ref19","first-page":"688","article-title":"Causal imputation via synthetic interventions","volume-title":"Conference on Causal Learning and Reasoning","author":"Squires","year":"2022"},{"key":"2025090110240872400_ref20","doi-asserted-by":"publisher","first-page":"7361","DOI":"10.1073\/pnas.1510493113","article-title":"Methods for causal inference from gene perturbation experiments and validation","volume":"113","author":"Meinshausen","year":"2016","journal-title":"Proc Natl Acad Sci"},{"key":"2025090110240872400_ref21","first-page":"209","article-title":"Bicycle: intervention-based causal discovery with cycles","volume-title":"Causal Learning and Reasoning","author":"Rohbeck","year":"2024"},{"key":"2025090110240872400_ref22","first-page":"1","article-title":"Scgpt: toward building a foundation model for single-cell multi-omics using generative AI","author":"Cui","year":"2024","journal-title":"Nat Methods"},{"key":"2025090110240872400_ref23","first-page":"2023","article-title":"Optimal distance metrics for single-cell RNA-Seq populations","author":"Ji","year":"2023","journal-title":"bioRxiv"},{"key":"2025090110240872400_ref24","doi-asserted-by":"publisher","first-page":"323","DOI":"10.1186\/s12859-022-04787-8","article-title":"Improved prediction of gene expression through integrating cell signalling models with machine learning","volume":"23","author":"Al Taweraqi","year":"2022","journal-title":"BMC Bioinform"},{"key":"2025090110240872400_ref25","doi-asserted-by":"publisher","first-page":"1196","DOI":"10.1038\/s41592-021-01252-x","article-title":"Effective gene expression prediction from sequence by integrating long-range interactions","volume":"18","author":"Avsec","year":"2021","journal-title":"Nat Methods"},{"key":"2025090110240872400_ref26","doi-asserted-by":"publisher","first-page":"1077","DOI":"10.3389\/fgene.2019.01077","article-title":"Gene expression value prediction based on xgboost algorithm","volume":"10","author":"Li","year":"2019","journal-title":"Front Genet"},{"key":"2025090110240872400_ref27","doi-asserted-by":"publisher","first-page":"1759","DOI":"10.1038\/s41592-023-01969-x","article-title":"Learning single-cell perturbation responses using neural optimal transport","volume":"20","author":"Bunne","year":"2023","journal-title":"Nat Methods"},{"key":"2025090110240872400_ref28","doi-asserted-by":"publisher","first-page":"btae265","DOI":"10.1093\/bioinformatics\/btae265","article-title":"Scpram accurately predicts single-cell gene expression perturbation response based on attention mechanism","volume":"40","author":"Jiang","year":"2024","journal-title":"Bioinformatics"},{"key":"2025090110240872400_ref29","first-page":"1","article-title":"Large-scale foundation model on single-cell transcriptomics","author":"Hao","year":"2024","journal-title":"Nat Methods"},{"key":"2025090110240872400_ref30","doi-asserted-by":"publisher","first-page":"995","DOI":"10.3390\/e24070995","article-title":"Differential expression analysis of single-cell rna-seq data: current statistical approaches and outstanding challenges","volume":"24","author":"Das","year":"2022","journal-title":"Entropy"},{"key":"2025090110240872400_ref31","author":"Aguirre","year":"2024","journal-title":"Gene Regulatory Network Structure Informs the Distribution of Perturbation Effects"},{"key":"2025090110240872400_ref32","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1145\/1143844.1143874","article-title":"The relationship between precision-recall and roc curves","volume-title":"Proceedings of the 23rd International Conference on Machine Learning","author":"Davis","year":"2006"},{"key":"2025090110240872400_ref33","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1038\/nbt.4042","article-title":"Multiplexed droplet single-cell RNA-Sequencing using natural genetic variation","volume":"36","author":"Kang","year":"2018","journal-title":"Nat Biotechnol"},{"key":"2025090110240872400_ref34","volume-title":"Open Problems\u2014Single-Cell Perturbations","author":"Burkhardt","year":"2023"},{"key":"2025090110240872400_ref35","doi-asserted-by":"publisher","first-page":"e47","DOI":"10.1093\/nar\/gkv007","article-title":"Limma powers differential expression analyses for rna-sequencing and microarray studies","volume":"43","author":"Ritchie","year":"2015","journal-title":"Nucleic Acids Res"},{"key":"2025090110240872400_ref36","first-page":"2024","article-title":"Deep learning-based predictions of gene perturbation effects do not yet outperform simple linear methods","author":"Ahlmann-Eltze","year":"2024","journal-title":"BioRxiv"},{"key":"2025090110240872400_ref37","first-page":"1","article-title":"Interpretable identification of cancer genes across biological networks via transformer-powered graph representation learning","author":"Xiaorui","year":"2025","journal-title":"Nat Biomed Eng"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/26\/5\/bbaf426\/64171629\/bbaf426.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/26\/5\/bbaf426\/64171629\/bbaf426.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,1]],"date-time":"2025-09-01T14:24:14Z","timestamp":1756736654000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbaf426\/8245187"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,31]]},"references-count":37,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,8,31]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbaf426","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,9]]},"published":{"date-parts":[[2025,8,31]]},"article-number":"bbaf426"}}