{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T03:26:40Z","timestamp":1783135600015,"version":"3.54.6"},"reference-count":24,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2025,10,4]],"date-time":"2025-10-04T00:00:00Z","timestamp":1759536000000},"content-version":"vor","delay-in-days":34,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000925","name":"National Health and Medical Research Council","doi-asserted-by":"publisher","award":["1173469"],"award-info":[{"award-number":["1173469"]}],"id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Chan Zuckerberg Initiative Single Cell Biology Data Insights"},{"name":"Chan Zuckerberg Initiative Single Cell Biology Data Insights","award":["DI2\u20130000000197"],"award-info":[{"award-number":["DI2\u20130000000197"]}]},{"name":"Chan Zuckerberg Initiative Single Cell Biology Data Insights","award":["DI-0000000027"],"award-info":[{"award-number":["DI-0000000027"]}]},{"name":"Australian Research Council Discovery Early Career Researcher Awards","award":["DE220100964"],"award-info":[{"award-number":["DE220100964"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,8,31]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>With the rapid development of computational methods for single-cell sequencing data, benchmarking serves as a valuable resource. As the number of benchmarking studies surges, it is timely to assess the current state of the field. We conducted a systematic literature search and assessed 282 papers, including all 130 benchmark-only papers from the search and an additional 152 method development papers containing benchmarking. This collective effort provides the most comprehensive quantitative summary of the current landscape of single-cell benchmarking studies. We examine performances across nine broad categories, including often ignored aspects such as role of datasets, robustness of methods and downstream evaluation. Our analysis highlights challenges such as how to effectively combine knowledge across multiple benchmarking studies and in what ways can the community recognize the risk and prevent benchmarking fatigue. This paper highlights the importance of adopting a community-led research paradigm to tackle these challenges and establish best practice standards.<\/jats:p>","DOI":"10.1093\/bib\/bbaf380","type":"journal-article","created":{"date-parts":[[2025,10,4]],"date-time":"2025-10-04T17:25:44Z","timestamp":1759598744000},"source":"Crossref","is-referenced-by-count":4,"title":["The current landscape and emerging challenges of benchmarking single-cell methods"],"prefix":"10.1093","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2356-4031","authenticated-orcid":false,"given":"Yue","family":"Cao","sequence":"first","affiliation":[{"name":"School of Mathematics and Statistics, University of Sydney , Sydney ,","place":["Australia"]},{"name":"Sydney Precision Data Science Centre, University of Sydney , NSW, 2006 ,","place":["Australia"]},{"name":"Charles Perkins Centre, University of Sydney , Sydney ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6735-9569","authenticated-orcid":false,"given":"Lijia","family":"Yu","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics, University of Sydney , Sydney ,","place":["Australia"]},{"name":"Sydney Precision Data Science Centre, University of Sydney , NSW, 2006 ,","place":["Australia"]},{"name":"Charles Perkins Centre, University of Sydney , Sydney ,","place":["Australia"]},{"name":"Computational Systems Biology Group, Children\u2019s Medical Research Institute, University of Sydney , Westmead ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marni","family":"Torkel","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics, University of Sydney , Sydney ,","place":["Australia"]},{"name":"Sydney Precision Data Science Centre, University of Sydney , NSW, 2006 ,","place":["Australia"]},{"name":"Charles Perkins Centre, University of Sydney , Sydney ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sanghyun","family":"Kim","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics, University of Sydney , Sydney ,","place":["Australia"]},{"name":"Sydney Precision Data Science Centre, University of Sydney , NSW, 2006 ,","place":["Australia"]},{"name":"Charles Perkins Centre, University of Sydney , Sydney ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yingxin","family":"Lin","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics, University of Sydney , Sydney ,","place":["Australia"]},{"name":"Sydney Precision Data Science Centre, University of Sydney , NSW, 2006 ,","place":["Australia"]},{"name":"Charles Perkins Centre, University of Sydney , Sydney ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1098-3138","authenticated-orcid":false,"given":"Pengyi","family":"Yang","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics, University of Sydney , Sydney ,","place":["Australia"]},{"name":"Sydney Precision Data Science Centre, University of Sydney , NSW, 2006 ,","place":["Australia"]},{"name":"Charles Perkins Centre, University of Sydney , Sydney ,","place":["Australia"]},{"name":"Computational Systems Biology Group, Children\u2019s Medical Research Institute, University of Sydney , Westmead ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5403-7998","authenticated-orcid":false,"given":"Terence P","family":"Speed","sequence":"additional","affiliation":[{"name":"Walter and Eliza Hall Institute for Medical Research , Parkville, VIC 3052 ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shila","family":"Ghazanfar","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics, University of Sydney , Sydney ,","place":["Australia"]},{"name":"Sydney Precision Data Science Centre, University of Sydney , NSW, 2006 ,","place":["Australia"]},{"name":"Charles Perkins Centre, University of Sydney , Sydney ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5271-2603","authenticated-orcid":false,"given":"Jean Yee Hwa","family":"Yang","sequence":"additional","affiliation":[{"name":"School of Mathematics and Statistics, University of Sydney , Sydney ,","place":["Australia"]},{"name":"Sydney Precision Data Science Centre, University of Sydney , NSW, 2006 ,","place":["Australia"]},{"name":"Charles Perkins Centre, University of Sydney , Sydney ,","place":["Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,10,4]]},"reference":[{"key":"2025100413253703800_ref1","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1186\/s13059-021-02519-4","article-title":"Over 1000 tools reveal trends in the single-cell RNA-seq analysis landscape","volume":"22","author":"Zappia","year":"2021","journal-title":"Genome Biol"},{"key":"2025100413253703800_ref2","doi-asserted-by":"publisher","first-page":"577","DOI":"10.1038\/d41586-022-04426-5","article-title":"Which single-cell analysis tool is best?","volume":"612","author":"Dance","year":"2022","journal-title":"Scientists offer advice Nature"},{"key":"2025100413253703800_ref3","doi-asserted-by":"publisher","first-page":"339","DOI":"10.1186\/s13059-021-02552-3","article-title":"Benchmarking UMI-based single-cell RNA-seq preprocessing workflows","volume":"22","author":"You","year":"2021","journal-title":"Genome Biol"},{"key":"2025100413253703800_ref4","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1038\/s41592-021-01336-8","article-title":"Benchmarking atlas-level data integration in single-cell genomics","volume":"19","author":"Luecken","year":"2021","journal-title":"Nat Methods"},{"key":"2025100413253703800_ref5","doi-asserted-by":"publisher","first-page":"747","DOI":"10.1038\/s41587-020-0469-4","article-title":"Benchmarking single-cell RNA-sequencing protocols for cell atlas projects","volume":"38","author":"Mereu","year":"2020","journal-title":"Nat Biotechnol"},{"key":"2025100413253703800_ref6","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1038\/s41592-019-0425-8","article-title":"Benchmarking single cell RNA-sequencing analysis pipelines using mixture control experiments","volume":"16","author":"Tian","year":"2019","journal-title":"Nat Methods"},{"key":"2025100413253703800_ref7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41467-023-37168-7","article-title":"A comprehensive benchmarking with practical guidelines for cellular deconvolution of spatial transcriptomics","volume":"14","author":"Li","year":"2023","journal-title":"Nat Commun"},{"key":"2025100413253703800_ref8","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1186\/s13059-019-1738-8","article-title":"Essential guidelines for computational method benchmarking","volume":"20","author":"Weber","year":"2019","journal-title":"Genome Biol"},{"key":"2025100413253703800_ref9","doi-asserted-by":"publisher","first-page":"1393","DOI":"10.1038\/s41467-019-09406-4","article-title":"Systematic benchmarking of omics computational tools","volume":"10","author":"Mangul","year":"2019","journal-title":"Nat Commun"},{"key":"2025100413253703800_ref10","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1186\/s13059-023-02962-5","article-title":"Meta-analysis of (single-cell method) benchmarks reveals the need for extensibility and interoperability","volume":"24","author":"Sonrel","year":"2023","journal-title":"Genome Biol"},{"key":"2025100413253703800_ref11","doi-asserted-by":"publisher","first-page":"105906","DOI":"10.1016\/j.ijsu.2021.105906","article-title":"The PRISMA 2020 statement: An updated guideline for reporting systematic reviews","volume":"88","author":"Page","year":"2021","journal-title":"Int J Surg"},{"key":"2025100413253703800_ref12","doi-asserted-by":"publisher","first-page":"3920","DOI":"10.1073\/pnas.1901326117","article-title":"Veridical data science","volume":"117","author":"Yu","year":"2020","journal-title":"Proc Natl Acad Sci U S A"},{"key":"2025100413253703800_ref13","doi-asserted-by":"publisher","first-page":"D596","DOI":"10.1093\/nar\/gkab1020","article-title":"DISCO: A database of deeply integrated human single-cell omics data","volume":"50","author":"Li","year":"2021","journal-title":"Nucleic Acids Res"},{"key":"2025100413253703800_ref14","doi-asserted-by":"publisher","DOI":"10.1038\/550451a","volume-title":"The Human Cell Atlas: From Vision to Reality","author":"Rozenblatt-Rosen","year":"2017"},{"key":"2025100413253703800_ref15","doi-asserted-by":"publisher","first-page":"160018","DOI":"10.1038\/sdata.2016.18","article-title":"The FAIR guiding principles for scientific data management and stewardship","volume":"3","author":"Wilkinson","year":"2016","journal-title":"Sci Data"},{"key":"2025100413253703800_ref16","doi-asserted-by":"publisher","author":"Fang","DOI":"10.1109\/ICORR66766.2025.11062946"},{"key":"2025100413253703800_ref17","doi-asserted-by":"publisher","first-page":"d5928","DOI":"10.1136\/bmj.d5928","article-title":"The Cochrane Collaboration\u2019s tool for assessing risk of bias in randomised trials","volume":"343","author":"Higgins","year":"2011","journal-title":"BMJ"},{"key":"2025100413253703800_ref18","doi-asserted-by":"publisher","first-page":"789","DOI":"10.1093\/bioinformatics\/btk046","article-title":"Comparison of Affymetrix GeneChip expression measures","volume":"22","author":"Irizarry","year":"2006","journal-title":"Bioinformatics"},{"key":"2025100413253703800_ref19","doi-asserted-by":"publisher","author":"Mallona","DOI":"10.1093\/nar\/gkaa1168"},{"key":"2025100413253703800_ref20","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2025100413253703800_ref21","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1038\/s41562-024-02046-9","article-title":"Large language models surpass human experts in predicting neuroscience results","volume":"9","author":"Luo","year":"2025","journal-title":"Nat Hum Behav"},{"key":"2025100413253703800_ref22","doi-asserted-by":"publisher","first-page":"e2411962122","DOI":"10.1073\/pnas.2411962122","article-title":"Transforming literature screening: The emerging role of large language models in systematic reviews","volume":"122","author":"Delgado-Chaves","year":"2025","journal-title":"Proc Natl Acad Sci U S A"},{"key":"2025100413253703800_ref23","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1186\/s13643-024-02575-4","article-title":"Title and abstract screening for literature reviews using large language models: An exploratory study in the biomedical domain","volume":"13","author":"Dennst\u00e4dt","year":"2024","journal-title":"Syst Rev"},{"key":"2025100413253703800_ref24","doi-asserted-by":"publisher","first-page":"e2401238121","DOI":"10.1073\/pnas.2401238121","article-title":"Automating the practice of science: Opportunities, challenges, and implications","volume":"122","author":"Musslick","year":"2025","journal-title":"Proc Natl Acad Sci U S A"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/26\/5\/bbaf380\/64495567\/bbaf380.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/26\/5\/bbaf380\/64495567\/bbaf380.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,4]],"date-time":"2025-10-04T17:25:45Z","timestamp":1759598745000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbaf380\/8273832"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,31]]},"references-count":24,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,8,31]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbaf380","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,9]]},"published":{"date-parts":[[2025,8,31]]},"article-number":"bbaf380"}}