{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T19:47:16Z","timestamp":1790884036130,"version":"4.1.0"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1010467","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,9,20]],"date-time":"2022-09-20T00:00:00Z","timestamp":1663632000000}}],"reference-count":56,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2022,9,8]],"date-time":"2022-09-08T00:00:00Z","timestamp":1662595200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>The development of increasingly efficient and cost-effective high throughput DNA sequencing techniques has enhanced the possibility of studying complex microbial systems. Recently, researchers have shown great interest in studying the microorganisms that characterise different ecological niches. Differential abundance analysis aims to find the differences in the abundance of each taxa between two classes of subjects or samples, assigning a significance value to each comparison. Several bioinformatic methods have been specifically developed, taking into account the challenges of microbiome data, such as sparsity, the different sequencing depth constraint between samples and compositionality. Differential abundance analysis has led to important conclusions in different fields, from health to the environment. However, the lack of a known biological truth makes it difficult to validate the results obtained. In this work we exploit metaSPARSim, a microbial sequencing count data simulator, to simulate data with differential abundance features between experimental groups. We perform a complete comparison of recently developed and established methods on a common benchmark with great effort to the reliability of both the simulated scenarios and the evaluation metrics. The performance overview includes the investigation of numerous scenarios, studying the effect on methods\u2019 results on the main covariates such as sample size, percentage of differentially abundant features, sequencing depth, feature variability, normalisation approach and ecological niches. Mainly, we find that methods show a good control of the type I error and, generally, also of the false discovery rate at high sample size, while recall seem to depend on the dataset and sample size.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1010467","type":"journal-article","created":{"date-parts":[[2022,9,8]],"date-time":"2022-09-08T17:32:52Z","timestamp":1662658372000},"page":"e1010467","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":68,"title":["Investigating differential abundance methods in microbiome data: A benchmark study"],"prefix":"10.1371","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1693-7792","authenticated-orcid":true,"given":"Marco","family":"Cappellato","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6129-5007","authenticated-orcid":true,"given":"Giacomo","family":"Baruzzo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8415-4688","authenticated-orcid":true,"given":"Barbara","family":"Di Camillo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2022,9,8]]},"reference":[{"issue":"4","key":"pcbi.1010467.ref001","doi-asserted-by":"crossref","first-page":"795","DOI":"10.1016\/j.cell.2019.07.008","article-title":"Tumor Microbiome Diversity and Composition Influence Pancreatic Cancer Outcomes","volume":"178","author":"E Riquelme","year":"2019","journal-title":"Cell"},{"issue":"1","key":"pcbi.1010467.ref002","doi-asserted-by":"crossref","first-page":"4822","DOI":"10.1038\/s41467-020-18649-5","article-title":"Abiraterone acetate preferentially enriches for the gut commensal Akkermansia muciniphila in castrate-resistant prostate cancer patients","volume":"11","author":"BA Daisley","year":"2020","journal-title":"Nat Commun"},{"key":"pcbi.1010467.ref003","doi-asserted-by":"crossref","first-page":"1245","DOI":"10.3389\/fimmu.2020.01245","article-title":"Low IgA Associated With Oropharyngeal Microbiota Changes and Lung Disease in Primary Antibody Deficiency.","volume":"11","author":"R-M Berbers","year":"2020","journal-title":"Front Immunol"},{"issue":"2","key":"pcbi.1010467.ref004","doi-asserted-by":"crossref","first-page":"432","DOI":"10.3390\/microorganisms9020432","article-title":"Staphylococcal Communities on Skin Are Associated with Atopic Dermatitis and Disease Severity","volume":"9","author":"SM Edslev","year":"2021","journal-title":"Microorganisms"},{"issue":"1","key":"pcbi.1010467.ref005","doi-asserted-by":"crossref","first-page":"e6","DOI":"10.5808\/GI.2019.17.1.e6","article-title":"Statistical Analysis of Metagenomics Data","volume":"17","author":"ML Calle","year":"2019","journal-title":"Genomics Inform"},{"issue":"1","key":"pcbi.1010467.ref006","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12859-021-04193-6","article-title":"Comparison study of differential abundance testing methods using two large Parkinson disease gut microbiome datasets derived from 16S amplicon sequencing","volume":"22","author":"ZD Wallen","year":"2021","journal-title":"BMC Bioinformatics"},{"key":"pcbi.1010467.ref007","doi-asserted-by":"crossref","first-page":"2224","DOI":"10.3389\/fmicb.2017.02224","article-title":"Microbiome Datasets Are Compositional: And This Is Not Optional","volume":"8","author":"GB Gloor","year":"2017","journal-title":"Front Microbiol"},{"issue":"8","key":"pcbi.1010467.ref008","doi-asserted-by":"crossref","first-page":"692","DOI":"10.1139\/cjm-2015-0821","article-title":"Compositional analysis: a valid approach to analyze microbiome high-throughput sequencing data","volume":"62","author":"GB Gloor","year":"2016","journal-title":"Can J Microbiol"},{"issue":"5","key":"pcbi.1010467.ref009","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1016\/j.annepidem.2016.03.003","article-title":"It\u2019s all relative: analyzing microbiome data as compositions.","volume":"26","author":"GB Gloor","year":"2016","journal-title":"Ann Epidemiol.May"},{"issue":"16","key":"pcbi.1010467.ref010","doi-asserted-by":"crossref","first-page":"2870","DOI":"10.1093\/bioinformatics\/bty175","article-title":"Understanding sequencing data as compositions: an outlook and review","volume":"34","author":"TP Quinn","year":"2018","journal-title":"Bioinformatics"},{"issue":"1","key":"pcbi.1010467.ref011","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1186\/s40168-017-0237-y","article-title":"Normalization and microbial differential abundance strategies depend upon data characteristics.","volume":"5","author":"S Weiss","year":"2017","journal-title":"Microbiome"},{"issue":"2","key":"pcbi.1010467.ref012","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1111\/j.2517-6161.1982.tb01195.x","article-title":"The Statistical Analysis of Compositional Data","volume":"44","author":"J. Aitchison","year":"1982","journal-title":"J R Stat Soc Ser B"},{"issue":"1","key":"pcbi.1010467.ref013","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1038\/s41522-020-00160-w","article-title":"Analysis of microbial compositions: a review of normalization and differential abundance analysis","volume":"6","author":"H Lin","year":"2020","journal-title":"NPJ Biofilms Microbiomes"},{"issue":"1","key":"pcbi.1010467.ref014","doi-asserted-by":"crossref","first-page":"2719","DOI":"10.1038\/s41467-019-10656-5","article-title":"Establishing microbial composition measurement standards with reference frames.","volume":"10","author":"JT Morton","year":"2019","journal-title":"Nat Commun"},{"issue":"4","key":"pcbi.1010467.ref015","doi-asserted-by":"crossref","first-page":"e1003531","DOI":"10.1371\/journal.pcbi.1003531","article-title":"Waste Not, Want Not: Why Rarefying Microbiome Data Is Inadmissible.","volume":"10","author":"PJ McMurdie","year":"2014","journal-title":"PLOS Comput Biol."},{"issue":"1","key":"pcbi.1010467.ref016","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13059-017-1359-z","article-title":"Experimental design and quantitative analysis of microbial community multiomics","volume":"18","author":"H Mallick","year":"2017","journal-title":"Genome Biol"},{"issue":"1","key":"pcbi.1010467.ref017","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1093\/bib\/bbx104","article-title":"A broken promise: microbiome differential abundance methods do not control the false discovery rate","volume":"20","author":"S Hawinkel","year":"2019","journal-title":"Brief Bioinform"},{"key":"pcbi.1010467.ref018","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1186\/s12864-016-2386-y","article-title":"Statistical evaluation of methods for identification of differentially abundant genes in comparative metagenomics","volume":"17","author":"V Jonsson","year":"2016","journal-title":"BMC Genomics"},{"issue":"1","key":"pcbi.1010467.ref019","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1186\/s13059-020-02104-1","article-title":"Assessment of statistical methods from single cell, bulk RNA-seq, and metagenomics applied to microbiome data","volume":"21","author":"M Calgaro","year":"2020","journal-title":"Genome Biol"},{"issue":"11","key":"pcbi.1010467.ref020","doi-asserted-by":"crossref","first-page":"e0259973","DOI":"10.1371\/journal.pone.0259973","article-title":"Analysing microbiome intervention design studies: Comparison of alternative multivariate statistical methods","volume":"16","author":"M Khomich","year":"2021","journal-title":"PLoS One"},{"issue":"4","key":"pcbi.1010467.ref021","doi-asserted-by":"crossref","first-page":"e0224909","DOI":"10.1371\/journal.pone.0224909","article-title":"Sequence count data are poorly fit by the negative binomial distribution","volume":"15","author":"S Hawinkel","year":"2020","journal-title":"PLoS One."},{"key":"pcbi.1010467.ref022","first-page":"1844","article-title":"A Zero-Inflated Latent Dirichlet Allocation Model for Microbiome Studies.","volume":"11","author":"RA Deek","year":"2021","journal-title":"Front Genet"},{"issue":"Suppl 9","key":"pcbi.1010467.ref023","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1186\/s12859-019-2882-6","article-title":"metaSPARSim: a 16S rRNA gene sequencing count data simulator","volume":"20","author":"I Patuzzi","year":"2019","journal-title":"BMC Bioinformatics"},{"issue":"1","key":"pcbi.1010467.ref024","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1186\/2049-2618-2-15","article-title":"Unifying the analysis of high-throughput sequencing datasets: characterizing RNA-seq, 16S rRNA gene sequencing and selective growth experiments by compositional data analysis","volume":"2","author":"AD Fernandes","year":"2014","journal-title":"Microbiome"},{"issue":"1","key":"pcbi.1010467.ref025","first-page":"255","article-title":"An empirical Bayes approach to normalization and differential abundance testing for microbiome data","volume":"21","author":"T Liu","year":"2020","journal-title":"BMC Bioinformatics"},{"key":"pcbi.1010467.ref026","first-page":"27663","article-title":"Analysis of composition of microbiomes: a novel method for studying microbial composition.","volume":"26","author":"S Mandal","year":"2015","journal-title":"Microb Ecol Heal Dis."},{"issue":"1","key":"pcbi.1010467.ref027","doi-asserted-by":"crossref","first-page":"3514","DOI":"10.1038\/s41467-020-17041-7","article-title":"Analysis of compositions of microbiomes with bias correction","volume":"11","author":"H Lin","year":"2020","journal-title":"Nat Commun"},{"issue":"1","key":"pcbi.1010467.ref028","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1214\/19-AOAS1283","article-title":"MODELING MICROBIAL ABUNDANCES AND DYSBIOSIS WITH BETA-BINOMIAL REGRESSION.","volume":"14","author":"BD Martin","year":"2020","journal-title":"Ann Appl Stat."},{"issue":"11","key":"pcbi.1010467.ref029","doi-asserted-by":"crossref","first-page":"e1009442","DOI":"10.1371\/journal.pcbi.1009442","article-title":"Multivariable association discovery in population-scale meta-omics studies","volume":"17","author":"H Mallick","year":"2021","journal-title":"PLOS Comput Biol"},{"issue":"12","key":"pcbi.1010467.ref030","doi-asserted-by":"crossref","first-page":"1200","DOI":"10.1038\/nmeth.2658","article-title":"Differential abundance analysis for microbial marker-gene surveys.","volume":"10","author":"JN Paulson","year":"2013","journal-title":"Nat Methods"},{"issue":"1","key":"pcbi.1010467.ref031","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1093\/bioinformatics\/btp616","article-title":"edgeR: a Bioconductor package for differential expression analysis of digital gene expression data","volume":"26","author":"MD Robinson","year":"2010","journal-title":"Bioinformatics"},{"issue":"12","key":"pcbi.1010467.ref032","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1186\/s13059-014-0550-8","article-title":"Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2","volume":"15","author":"MI Love","year":"2014","journal-title":"Genome Biol"},{"issue":"4","key":"pcbi.1010467.ref033","doi-asserted-by":"crossref","first-page":"e61217","DOI":"10.1371\/journal.pone.0061217","article-title":"phyloseq: An R Package for Reproducible Interactive Analysis and Graphics of Microbiome Census Data","volume":"8","author":"PJ McMurdie","year":"2013","journal-title":"PLoS One"},{"issue":"2","key":"pcbi.1010467.ref034","first-page":"65","article-title":"A Simple Sequentially Rejective Multiple Test Procedure","volume":"6","author":"S. Holm","year":"1979","journal-title":"Scand J Stat"},{"key":"pcbi.1010467.ref035","author":"DS Clausen","year":"2022","journal-title":"Modeling complex measurement error in microbiome experiments"},{"key":"pcbi.1010467.ref036","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.46923","article-title":"Consistent and correctable bias in metagenomic sequencing experiments","volume":"8","author":"MR McLaren","year":"2019","journal-title":"Elife"},{"issue":"7402","key":"pcbi.1010467.ref037","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1038\/nature11234","article-title":"Structure, function and diversity of the healthy human microbiome","volume":"486","author":"C Huttenhower","year":"2012","journal-title":"Nature"},{"issue":"7402","key":"pcbi.1010467.ref038","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1038\/nature11209","article-title":"A framework for human microbiome research","volume":"486","author":"BA Meth\u00e9","year":"2012","journal-title":"Nature"},{"issue":"1","key":"pcbi.1010467.ref039","doi-asserted-by":"crossref","first-page":"11589","DOI":"10.1038\/s41598-019-47953-4","article-title":"Fecal microbiome and metabolome of infants fed bovine MFGM supplemented formula or standard formula with breast-fed infants as reference: a randomized controlled trial","volume":"9","author":"X He","year":"2019","journal-title":"Sci Rep"},{"issue":"7758","key":"pcbi.1010467.ref040","doi-asserted-by":"crossref","first-page":"655","DOI":"10.1038\/s41586-019-1237-9","article-title":"Multi-omics of the gut microbial ecosystem in inflammatory bowel diseases","volume":"569","author":"J Lloyd-Price","year":"2019","journal-title":"Nature"},{"issue":"4","key":"pcbi.1010467.ref041","doi-asserted-by":"crossref","first-page":"e4600","DOI":"10.7717\/peerj.4600","article-title":"GMPR: A robust normalization method for zero-inflated count data with application to microbiome sequencing data","volume":"2018","author":"L Chen","year":"2018","journal-title":"PeerJ2"},{"issue":"3","key":"pcbi.1010467.ref042","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1093\/bioinformatics\/btt688","article-title":"RNA-seq differential expression studies: more sequence or more replication?","volume":"30","author":"Y Liu","year":"2014","journal-title":"Bioinformatics"},{"issue":"1","key":"pcbi.1010467.ref043","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1186\/s12859-018-2445-2","article-title":"Empirical assessment of the impact of sample number and read depth on RNA-Seq analysis workflow performance","volume":"19","author":"A Baccarella","year":"2018","journal-title":"BMC Bioinformatics"},{"issue":"2","key":"pcbi.1010467.ref044","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/gigascience\/giab005","article-title":"MB-GAN: Microbiome Simulation via Generative Adversarial Network","volume":"10","author":"R Rong","year":"2021","journal-title":"Gigascience"},{"issue":"9","key":"pcbi.1010467.ref045","doi-asserted-by":"crossref","first-page":"e1008913","DOI":"10.1371\/journal.pcbi.1008913","article-title":"A statistical model for describing and simulating microbial community profiles","volume":"17","author":"S Ma","year":"2021","journal-title":"PLOS Comput Biol"},{"key":"pcbi.1010467.ref046","article-title":"Generative Adversarial Nets","volume":"27","author":"IJ Goodfellow","year":"2014","journal-title":"Adv Neural Inf Process Syst"},{"issue":"1","key":"pcbi.1010467.ref047","first-page":"1","article-title":"Comparison between 16S rRNA and shotgun sequencing data for the taxonomic characterization of the gut microbiota","volume":"11","author":"F Durazzi","year":"2021","journal-title":"Sci Reports"},{"issue":"2","key":"pcbi.1010467.ref048","doi-asserted-by":"crossref","first-page":"e0228899","DOI":"10.1371\/journal.pone.0228899","article-title":"Microbial resolution of whole genome shotgun and 16S amplicon metagenomic sequencing using publicly available NEON data","volume":"15","author":"KD Brumfield","year":"2020","journal-title":"PLoS One"},{"issue":"1","key":"pcbi.1010467.ref049","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13040-021-00270-x","article-title":"Comparison of 16S and whole genome dog microbiomes using machine learning","volume":"14","author":"S Lewis","year":"2021","journal-title":"BioData Min"},{"key":"pcbi.1010467.ref050","unstructured":"Cario MC, Nelson BL. Modeling and Generating Random Vectors with Arbitrary Marginal Distributions and Correlation Matrix (pp. 1\u201319). Technical Report, Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, Illinois. 1997."},{"issue":"5","key":"pcbi.1010467.ref051","doi-asserted-by":"crossref","first-page":"e1004226","DOI":"10.1371\/journal.pcbi.1004226","article-title":"Sparse and Compositionally Robust Inference of Microbial Ecological Networks.","volume":"11","author":"ZD Kurtz","year":"2015","journal-title":"PLOS Comput Biol."},{"issue":"6","key":"pcbi.1010467.ref052","doi-asserted-by":"crossref","first-page":"e1009089","DOI":"10.1371\/journal.pcbi.1009089","article-title":"A zero inflated log-normal model for inference of sparse microbial association networks","volume":"17","author":"V Prost","year":"2021","journal-title":"PLOS Comput Biol"},{"issue":"15","key":"pcbi.1010467.ref053","first-page":"1","article-title":"Beware to ignore the rare: how imputing zero-values can improve the quality of 16S rRNA gene studies results","volume":"22","author":"G Baruzzo","year":"2021","journal-title":"BMC Bioinformatics"},{"issue":"1","key":"pcbi.1010467.ref054","article-title":"Microbiome differential abundance methods produce different results across 38 datasets","volume":"13","author":"JT Nearing","year":"2022","journal-title":"Nat Commun"},{"issue":"1","key":"pcbi.1010467.ref055","doi-asserted-by":"crossref","first-page":"e00162","DOI":"10.1128\/mSystems.00162-16","article-title":"Balance Trees Reveal Microbial Niche Differentiation","volume":"2","author":"JT Morton","year":"2017","journal-title":"mSystems"},{"issue":"4","key":"pcbi.1010467.ref056","first-page":"e00053","article-title":"Balances: a New Perspective for Microbiome Analysis","volume":"3","author":"J Rivera-Pinto","year":"2018","journal-title":"mSystemsJul"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1010467","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,9,20]],"date-time":"2022-09-20T00:00:00Z","timestamp":1663632000000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010467","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,3]],"date-time":"2024-10-03T13:54:38Z","timestamp":1727963678000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010467"}},"subtitle":[],"editor":[{"given":"Luis Pedro","family":"Coelho","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2022,9,8]]},"references-count":56,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2022,9,8]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1010467","relation":{"new_version":[{"id-type":"doi","id":"10.1371\/journal.pcbi.1010467","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,8]]}}}