{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T07:00:43Z","timestamp":1773126043069,"version":"3.50.1"},"reference-count":21,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2019,11,28]],"date-time":"2019-11-28T00:00:00Z","timestamp":1574899200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000925","name":"National Health and Medical Research Council","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000925","name":"NHMRC","doi-asserted-by":"publisher","award":["GNT1143163"],"award-info":[{"award-number":["GNT1143163"]}],"id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000925","name":"NHMRC","doi-asserted-by":"publisher","award":["GNT1124812"],"award-info":[{"award-number":["GNT1124812"]}],"id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Career Development Fellowship","award":["GNT1104924"],"award-info":[{"award-number":["GNT1104924"]}]},{"name":"Chan Zuckerberg Initiative DAF"},{"DOI":"10.13039\/100000923","name":"Silicon Valley Community Foundation","doi-asserted-by":"publisher","award":["2018-182819"],"award-info":[{"award-number":["2018-182819"]}],"id":[{"id":"10.13039\/100000923","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000923","name":"Silicon Valley Community Foundation","doi-asserted-by":"publisher","award":["2019-002443"],"award-info":[{"award-number":["2019-002443"]}],"id":[{"id":"10.13039\/100000923","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Melbourne Research Scholarship"},{"name":"Victorian State Government Operational Infrastructure Support and Australian Government"},{"DOI":"10.13039\/501100000925","name":"NHMRC","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,4,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>Bioinformatic analysis of single-cell gene expression data is a rapidly evolving field. Hundreds of bespoke methods have been developed in the past few years to deal with various aspects of single-cell analysis and consensus on the most appropriate methods to use under different settings is still emerging. Benchmarking the many methods is therefore of critical importance and since analysis of single-cell data usually involves multi-step pipelines, effective evaluation of pipelines involving different combinations of methods is required. Current benchmarks of single-cell methods are mostly implemented with ad-hoc code that is often difficult to reproduce or extend, and exhaustive manual coding of many combinations is infeasible in most instances. Therefore, new software is needed to manage pipeline benchmarking.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The CellBench R software facilitates method comparisons in either a task-centric or combinatorial way to allow pipelines of methods to be evaluated in an effective manner. CellBench automatically runs combinations of methods, provides facilities for measuring running time and delivers output in tabular form which is highly compatible with tidyverse R packages for summary and visualization. Our software has enabled comprehensive benchmarking of single-cell RNA-seq normalization, imputation, clustering, trajectory analysis and data integration methods using various performance metrics obtained from data with available ground truth. CellBench is also amenable to benchmarking other bioinformatics analysis tasks.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Availability and implementation<\/jats:title>\n                  <jats:p>Available from https:\/\/bioconductor.org\/packages\/CellBench.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btz889","type":"journal-article","created":{"date-parts":[[2019,11,26]],"date-time":"2019-11-26T12:11:54Z","timestamp":1574770314000},"page":"2288-2290","source":"Crossref","is-referenced-by-count":21,"title":["<i>CellBench<\/i>: <i>R\/Bioconductor<\/i> software for comparing single-cell RNA-seq analysis methods"],"prefix":"10.1093","volume":"36","author":[{"given":"Shian","family":"Su","sequence":"first","affiliation":[{"name":"Epigenetics and Development Division, The Walter and Eliza Hall Institute of Medical Research , Parkville, VIC 3052, Australia"},{"name":"Department of Medical Biology, The University of Melbourne , Parkville, VIC 3010, Australia"}]},{"given":"Luyi","family":"Tian","sequence":"additional","affiliation":[{"name":"Epigenetics and Development Division, The Walter and Eliza Hall Institute of Medical Research , Parkville, VIC 3052, Australia"},{"name":"Department of Medical Biology, The University of Melbourne , Parkville, VIC 3010, Australia"}]},{"given":"Xueyi","family":"Dong","sequence":"additional","affiliation":[{"name":"Epigenetics and Development Division, The Walter and Eliza Hall Institute of Medical Research , Parkville, VIC 3052, Australia"},{"name":"Department of Medical Biology, The University of Melbourne , Parkville, VIC 3010, Australia"}]},{"given":"Peter F","family":"Hickey","sequence":"additional","affiliation":[{"name":"Epigenetics and Development Division, The Walter and Eliza Hall Institute of Medical Research , Parkville, VIC 3052, Australia"},{"name":"Department of Medical Biology, The University of Melbourne , Parkville, VIC 3010, Australia"}]},{"given":"Saskia","family":"Freytag","sequence":"additional","affiliation":[{"name":"Epigenetics and Genomics, Harry Perkins Institute of Medical Research , Nedlands, WA 6009, Australia"}]},{"given":"Matthew E","family":"Ritchie","sequence":"additional","affiliation":[{"name":"Epigenetics and Development Division, The Walter and Eliza Hall Institute of Medical Research , Parkville, VIC 3052, Australia"},{"name":"Department of Medical Biology, The University of Melbourne , Parkville, VIC 3010, Australia"},{"name":"School of Mathematics and Statistics, The University of Melbourne , Parkville, VIC 3010, Australia"}]}],"member":"286","published-online":{"date-parts":[[2019,11,28]]},"reference":[{"key":"2023062300073740900_btz889-B1","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-019-0654-x","article-title":"Orchestrating single-cell analysis with Bioconductor","author":"Amezquita","year":"2019","journal-title":"Nat Methods"},{"key":"2023062300073740900_btz889-B2","doi-asserted-by":"crossref","first-page":"411","DOI":"10.1038\/nbt.4096","article-title":"Integrating single-cell transcriptomic data across different conditions, technologies, and species","volume":"36","author":"Butler","year":"2018","journal-title":"Nat. Biotechnol"},{"key":"2023062300073740900_btz889-B3","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1186\/s12859-018-2226-y","article-title":"DrImpute: imputing dropout events in single cell RNA sequencing data","volume":"19","author":"Gong","year":"2018","journal-title":"BMC Bioinformatics"},{"key":"2023062300073740900_btz889-B4","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1038\/nature14966","article-title":"Single-cell messenger RNA sequencing reveals rare intestinal cell types","volume":"525","author":"Gr\u00fcn","year":"2015","journal-title":"Nature"},{"key":"2023062300073740900_btz889-B5","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1038\/nmeth.3252","article-title":"Orchestrating high-throughput genomic analysis with Bioconductor","volume":"12","author":"Huber","year":"2015","journal-title":"Nat. Methods"},{"key":"2023062300073740900_btz889-B6","doi-asserted-by":"crossref","first-page":"e117","DOI":"10.1093\/nar\/gkw430","article-title":"TSCAN: pseudo-time reconstruction and evaluation in single-cell RNA-seq analysis","volume":"44","author":"Ji","year":"2016","journal-title":"Nucleic Acids Res"},{"key":"2023062300073740900_btz889-B7","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1093\/bioinformatics\/bty627","article-title":"Reproducible and replicable comparisons using SummarizedBenchmark","volume":"35","author":"Kimes","year":"2019","journal-title":"Bioinformatics"},{"key":"2023062300073740900_btz889-B8","author":"Lun","year":"2019"},{"key":"2023062300073740900_btz889-B9","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1186\/s13059-016-0947-7","article-title":"Pooling across cells to normalize single-cell RNA sequencing data with many zero counts","volume":"17","author":"Lun","year":"2016","journal-title":"Genome Biol"},{"key":"2023062300073740900_btz889-B10","author":"M\u00fcller","year":"2019"},{"key":"2023062300073740900_btz889-B11","year":"2019"},{"key":"2023062300073740900_btz889-B12","doi-asserted-by":"crossref","first-page":"R25","DOI":"10.1186\/gb-2010-11-3-r25","article-title":"A scaling normalization method for differential expression analysis of RNA-seq data","volume":"11","author":"Robinson","year":"2010","journal-title":"Genome Biol"},{"key":"2023062300073740900_btz889-B13","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1038\/s41587-019-0071-9","article-title":"A comparison of single-cell trajectory inference methods","volume":"37","author":"Saelens","year":"2019","journal-title":"Nat. Biotechnol"},{"key":"2023062300073740900_btz889-B14","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1038\/nmeth.3805","article-title":"iCOBRA: open, reproducible, standardized and live method benchmarking","volume":"13","author":"Soneson","year":"2016","journal-title":"Nat. Methods"},{"key":"2023062300073740900_btz889-B15","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1038\/nmeth.4612","article-title":"Bias, robustness and scalability in single-cell differential expression analysis","volume":"15","author":"Soneson","year":"2018","journal-title":"Nat. Methods"},{"key":"2023062300073740900_btz889-B16","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1186\/s13059-016-0940-1","article-title":"A benchmark for RNA-seq quantification pipelines","volume":"17","author":"Teng","year":"2016","journal-title":"Genome Biol"},{"key":"2023062300073740900_btz889-B17","doi-asserted-by":"crossref","first-page":"479","DOI":"10.1038\/s41592-019-0425-8","article-title":"Benchmarking single cell RNA-sequencing analysis pipelines using mixture control experiments","volume":"16","author":"Tian","year":"2019","journal-title":"Nat. Methods"},{"key":"2023062300073740900_btz889-B18","author":"Wagner","year":"2017"},{"key":"2023062300073740900_btz889-B19","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-24277-4","volume-title":"ggplot2: Elegant Graphics for Data Analysis","author":"Wickham","year":"2016"},{"key":"2023062300073740900_btz889-B20","doi-asserted-by":"crossref","first-page":"e179","DOI":"10.1093\/nar\/gkx828","article-title":"Linnorm: improved statistical analysis for single cell RNA-seq expression data","volume":"45","author":"Yip","year":"2017","journal-title":"Nucleic Acids Res"},{"key":"2023062300073740900_btz889-B21","doi-asserted-by":"crossref","first-page":"e1006245","DOI":"10.1371\/journal.pcbi.1006245","article-title":"Exploring the single-cell RNA-seq analysis landscape with the scRNA-tools database","volume":"14","author":"Zappia","year":"2018","journal-title":"PLoS Comput. Biol"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btz889\/31570599\/btz889.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/36\/7\/2288\/50670494\/bioinformatics_36_7_2288.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/36\/7\/2288\/50670494\/bioinformatics_36_7_2288.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,24]],"date-time":"2023-06-24T18:08:59Z","timestamp":1687630139000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/36\/7\/2288\/5645177"}},"subtitle":[],"editor":[{"given":"Jan","family":"Gorodkin","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2019,11,28]]},"references-count":21,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2020,4,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btz889","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,4,1]]},"published":{"date-parts":[[2019,11,28]]}}}