{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,26]],"date-time":"2026-04-26T05:06:56Z","timestamp":1777180016728,"version":"3.51.4"},"reference-count":58,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2017,8,22]],"date-time":"2017-08-22T00:00:00Z","timestamp":1503360000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/100008897","name":"Janssen Pharmaceuticals","doi-asserted-by":"publisher","award":["NCT3.0_2015.13"],"award-info":[{"award-number":["NCT3.0_2015.13"]}],"id":[{"id":"10.13039\/100008897","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100008897","name":"Janssen Pharmaceuticals","doi-asserted-by":"publisher","award":["NCT3.0_2015.2 SPL\/RP"],"award-info":[{"award-number":["NCT3.0_2015.2 SPL\/RP"]}],"id":[{"id":"10.13039\/100008897","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,1,18]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>High-throughput sequencing technologies allow easy characterization of the human microbiome, but the statistical methods to analyze microbiome data are still in their infancy. Differential abundance methods aim at detecting associations between the abundances of bacterial species and subject grouping factors. The results of such methods are important to identify the microbiome as a prognostic or diagnostic biomarker or to demonstrate efficacy of prodrug or antibiotic drugs. Because of a lack of benchmarking studies in the microbiome field, no consensus exists on the performance of the statistical methods. We have compared a large number of popular methods through extensive parametric and nonparametric simulation as well as real data shuffling algorithms. The results are consistent over the different approaches and all point to an alarming excess of false discoveries. This raises great doubts about the reliability of discoveries in past studies and imperils reproducibility of microbiome experiments. To further improve method benchmarking, we introduce a new simulation tool that allows to generate correlated count data following any univariate count distribution; the correlation structure may be inferred from real data. Most simulation studies discard the correlation between species, but our results indicate that this correlation can negatively affect the performance of statistical methods.<\/jats:p>","DOI":"10.1093\/bib\/bbx104","type":"journal-article","created":{"date-parts":[[2017,8,4]],"date-time":"2017-08-04T19:11:45Z","timestamp":1501873905000},"page":"210-221","source":"Crossref","is-referenced-by-count":164,"title":["A broken promise: microbiome differential abundance methods do not control the false discovery rate"],"prefix":"10.1093","volume":"20","author":[{"given":"Stijn","family":"Hawinkel","sequence":"first","affiliation":[{"name":"Department of Mathematical Modelling, Statistics and Bioinformatics at Ghent University, Belgium"}]},{"given":"Federico","family":"Mattiello","sequence":"first","affiliation":[{"name":"Department of Mathematical Modelling, Statistics and Bioinformatics at Ghent University, Belgium"}]},{"given":"Luc","family":"Bijnens","sequence":"first","affiliation":[{"name":"Center for Statistics at Hasselt University, Belgium"}]},{"given":"Olivier","family":"Thas","sequence":"first","affiliation":[{"name":"Department of Mathematical Modelling, Statistics and Bioinformatics at Ghent University, Belgium"}]}],"member":"286","published-online":{"date-parts":[[2017,8,22]]},"reference":[{"key":"2020040809051990500_bbx104-B1","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1038\/nature11234","article-title":"Structure, function and diversity of the healthy human microbiome","volume":"486","author":"The Human Microbiome Project Consortium","year":"2012","journal-title":"Nature"},{"key":"2020040809051990500_bbx104-B2","doi-asserted-by":"crossref","first-page":"4159","DOI":"10.1113\/jphysiol.2009.172742","article-title":"The role of the intestinal microbiota in enteric infection","volume":"587","author":"Sekirov","year":"2009","journal-title":"J Physiol"},{"key":"2020040809051990500_bbx104-B3","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1016\/j.cell.2009.09.033","article-title":"Induction of intestinal Th17 cells by segmented filamentous bacteria","volume":"139","author":"Ivanov","year":"2009","journal-title":"Cell"},{"key":"2020040809051990500_bbx104-B4","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1038\/mi.2010.3","article-title":"Segmented filamentous bacteria take the stage","volume":"3","author":"Ivanov","year":"2010","journal-title":"Mucosal Immunol"},{"key":"2020040809051990500_bbx104-B5","doi-asserted-by":"crossref","first-page":"4680","DOI":"10.1073\/pnas.1002611107","article-title":"Vaginal microbiome of reproductive-age women","volume":"108","author":"Ravel","year":"2011","journal-title":"Proc Natl Acad Sci USA"},{"key":"2020040809051990500_bbx104-B6","doi-asserted-by":"crossref","first-page":"733","DOI":"10.1038\/nrmicro2903","article-title":"Microbiome: Gut microbiome as a marker for diabetes","volume":"10","author":"Kahrstrom","year":"2012","journal-title":"Nat Rev Micro"},{"key":"2020040809051990500_bbx104-B7","doi-asserted-by":"crossref","first-page":"260","DOI":"10.1016\/j.chom.2015.01.001","article-title":"The dynamics of the human infant gut microbiome in development and in progression towards type 1 diabetes","volume":"17","author":"Kostic","year":"2015","journal-title":"Cell Host Microbe"},{"key":"2020040809051990500_bbx104-B8","doi-asserted-by":"crossref","first-page":"3083","DOI":"10.1002\/art.34539","article-title":"Periodontal disease and the oral microbiota in new-onset rheumatoid arthritis","volume":"64","author":"Scher","year":"2012","journal-title":"Arthritis Rheum"},{"key":"2020040809051990500_bbx104-B9","doi-asserted-by":"crossref","first-page":"2761","DOI":"10.1128\/JCM.01228-07","article-title":"16S rRNA gene sequencing for bacterial identification in the diagnostic laboratory: pluses, perils, and pitfalls","volume":"45","author":"Janda","year":"2007","journal-title":"J Clin Microbiol"},{"key":"2020040809051990500_bbx104-B10","doi-asserted-by":"crossref","first-page":"e1002808.","DOI":"10.1371\/journal.pcbi.1002808","article-title":"Chapter 12: human microbiome analysis","volume":"8","author":"Morgan","year":"2012","journal-title":"PLoS Comput Biol"},{"key":"2020040809051990500_bbx104-B11","doi-asserted-by":"crossref","first-page":"1200","DOI":"10.1038\/nmeth.2658","article-title":"Robust methods for differential abundance analysis in marker gene surveys","volume":"10","author":"Paulson","year":"2013","journal-title":"Nat Methods"},{"key":"2020040809051990500_bbx104-B12","doi-asserted-by":"crossref","first-page":"e1003531","DOI":"10.1371\/journal.pcbi.1003531","article-title":"Waste not, want not: why rarefying microbiome data is inadmissible","volume":"10","author":"McMurdie","year":"2014","journal-title":"PLoS Comput Biol"},{"key":"2020040809051990500_bbx104-B13","doi-asserted-by":"crossref","first-page":"R106","DOI":"10.1186\/gb-2010-11-10-r106","article-title":"Differential expression analysis for sequence count data","volume":"11","author":"Anders","year":"2010","journal-title":"Genome Biol"},{"key":"2020040809051990500_bbx104-B14","doi-asserted-by":"crossref","first-page":"R25","DOI":"10.1186\/gb-2010-11-3-r25","article-title":"A scaling normalization method for differential expression analysis of RNA-seq data","volume":"11","author":"Robinson","year":"2010","journal-title":"Genome Biol"},{"key":"2020040809051990500_bbx104-B15","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1186\/1471-2105-11-94","article-title":"Evaluation of statistical methods for normalization and differential expression in mRNA-Seq experiments","volume":"11","author":"Bullard","year":"2010","journal-title":"BMC Bioinformatics"},{"key":"2020040809051990500_bbx104-B16","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1093\/biostatistics\/kxr031","article-title":"Normalization, testing, and false discovery rate estimation for RNA-sequencing data","volume":"13","author":"Li","year":"2012","journal-title":"Biostatistics"},{"key":"2020040809051990500_bbx104-B17","first-page":"27663","article-title":"Analysis of composition of microbiomes: a novel method for studying microbial composition","volume":"26","author":"Mandal","year":"2015","journal-title":"Microb Ecol Health Dis"},{"key":"2020040809051990500_bbx104-B18","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1186\/2049-2618-2-15","article-title":"Unifying the analysis of highthroughput sequencing datasets: characterizing RNA-seq, 16S rRNA gene sequencing and selective growth experiments by compositional data analysis","volume":"2","author":"Fernandes","year":"2014","journal-title":"Microbiome"},{"key":"2020040809051990500_bbx104-B19","doi-asserted-by":"crossref","first-page":"P17","DOI":"10.1186\/1465-6906-12-S1-P17","article-title":"Metastats: an improved statistical method for analysis of metagenomic data","volume":"12","author":"Paulson","year":"2011","journal-title":"Genome Biol"},{"key":"2020040809051990500_bbx104-B20","doi-asserted-by":"crossref","first-page":"766","DOI":"10.15252\/msb.20145645","article-title":"Potential of fecal microbiota for early-stage detection of colorectal cancer","volume":"10","author":"Zeller","year":"2014","journal-title":"Mol Syst Biol"},{"key":"2020040809051990500_bbx104-B21","doi-asserted-by":"crossref","first-page":"R60","DOI":"10.1186\/gb-2011-12-6-r60","article-title":"Metagenomic biomarker discovery and explanation","volume":"12","author":"Segata","year":"2011","journal-title":"Genome Biol"},{"key":"2020040809051990500_bbx104-B22","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","article-title":"Controlling the false discovery rate: a practical and powerful approach to multiple testing","volume":"57","author":"Benjamini","year":"1995","journal-title":"J R Stat Soc Series B Methodol"},{"key":"2020040809051990500_bbx104-B23","doi-asserted-by":"crossref","first-page":"10084","DOI":"10.1093\/nar\/gks804","article-title":"A comprehensive comparison of RNA-Seq-based transcriptome analysis from reads to differential gene expression and cross-comparison with microarrays: a case study in Saccharomyces cerevisiae","volume":"40","author":"Nookaew","year":"2012","journal-title":"Nucleic Acids Res"},{"key":"2020040809051990500_bbx104-B24","first-page":"1","article-title":"Synthetic data sets for the identification of key ingredients for RNA-seq differential analysis","volume":"17","author":"Rigaill","year":"2016","journal-title":"Brief Bioinform"},{"key":"2020040809051990500_bbx104-B25","doi-asserted-by":"crossref","first-page":"2881","DOI":"10.1093\/bioinformatics\/btm453","article-title":"Moderated statistical tests for assessing differences in tag abundance","volume":"23","author":"Robinson","year":"2007","journal-title":"Bioinformatics"},{"key":"2020040809051990500_bbx104-B26","doi-asserted-by":"crossref","first-page":"R29","DOI":"10.1186\/gb-2014-15-2-r29","article-title":"voom: precision weights unlock linear model analysis tools for RNA-seq read counts","volume":"15","author":"Law","year":"2014","journal-title":"Genome Biol"},{"key":"2020040809051990500_bbx104-B27","doi-asserted-by":"crossref","first-page":"1165","DOI":"10.1214\/aos\/1013699998","article-title":"The control of the false discovery rate in multiple testing under dependency","volume":"29","author":"Benjamini","year":"2001","journal-title":"Ann Statist"},{"key":"2020040809051990500_bbx104-B28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1214\/07-STS236","article-title":"Microarrays, empirical bayes and the two-groups model","volume":"23","author":"Efron","year":"2008","journal-title":"Statist Sci"},{"key":"2020040809051990500_bbx104-B29","doi-asserted-by":"crossref","first-page":"2131","DOI":"10.1093\/bioinformatics\/btv124","article-title":"SimSeq: a nonparametric approach to simulation of RNA-sequence datasets","volume":"31","author":"Benidt","year":"2015","journal-title":"Bioinformatics"},{"key":"2020040809051990500_bbx104-B30","doi-asserted-by":"crossref","first-page":"178","DOI":"10.3389\/fgene.2013.00178","article-title":"Evaluating statistical analysis models for RNA sequencing experiments","volume":"4","author":"Reeb","year":"2013","journal-title":"Front Genet"},{"key":"2020040809051990500_bbx104-B31","doi-asserted-by":"crossref","first-page":"248","DOI":"10.3732\/ajb.1100340","article-title":"A comparison of statistical methods for detecting differentially expressed genes from RNAseq data","volume":"99","author":"Kvam","year":"2012","journal-title":"Am J Bot"},{"key":"2020040809051990500_bbx104-B32","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1186\/s13059-014-0550-8","article-title":"Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2","volume":"15","author":"Love","year":"2014","journal-title":"Genome Biol"},{"key":"2020040809051990500_bbx104-B33","doi-asserted-by":"crossref","first-page":"2317","DOI":"10.1101\/gr.096651.109","article-title":"The NIH human microbiome project","volume":"19","author":"The NIH HMP Working Group","year":"2009","journal-title":"Genome Res"},{"key":"2020040809051990500_bbx104-B34","year":"2015"},{"key":"2020040809051990500_bbx104-B35","doi-asserted-by":"crossref","first-page":"91\u201391","DOI":"10.1186\/1471-2105-14-91","article-title":"A comparison of methods for differential expression analysis of RNA-seq data","volume":"14","author":"Soneson","year":"2013","journal-title":"BMC Bioinformatics"},{"key":"2020040809051990500_bbx104-B36","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1186\/s12864-016-2386-y","article-title":"Statistical evaluation of methods for identification of differentially abundant genes in comparative metagenomics","volume":"17","author":"Jonsson","year":"2016","journal-title":"BMC Genomics"},{"key":"2020040809051990500_bbx104-B37","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1093\/bib\/bbt086","article-title":"Comparison of software packages for detecting differential expression in RNA-seq studies","volume":"16","author":"Seyednasrollah","year":"2013","journal-title":"Brief Bioinform"},{"key":"2020040809051990500_bbx104-B38","doi-asserted-by":"crossref","first-page":"e576","DOI":"10.7717\/peerj.576","article-title":"Error estimates for the analysis of differential expression from RNA-seq count data","volume":"2","author":"Burden","year":"2014","journal-title":"PeerJ"},{"key":"2020040809051990500_bbx104-B39","doi-asserted-by":"crossref","first-page":"e1004226","DOI":"10.1371\/journal.pcbi.1004226","article-title":"Sparse and compositionally robust inference of microbial ecological networks","volume":"11","author":"Kurtz","year":"2015","journal-title":"PLoS Comput Biol"},{"key":"2020040809051990500_bbx104-B40","doi-asserted-by":"crossref","first-page":"1777","DOI":"10.1080\/03610928808829713","article-title":"Parameter estimation for the Dirichlet-multinomial distribution using supplementary beta-binomial data","volume":"17","author":"Danaher","year":"1988","journal-title":"Commun Stat Theory Methods"},{"key":"2020040809051990500_bbx104-B41","doi-asserted-by":"crossref","first-page":"1489","DOI":"10.1053\/j.gastro.2014.02.009","article-title":"The microbiome in inammatory bowel diseases: current status and the future ahead","volume":"146","author":"Kostic","year":"2014","journal-title":"Gastroenterology"},{"key":"2020040809051990500_bbx104-B42","doi-asserted-by":"crossref","first-page":"1691","DOI":"10.1073\/pnas.1120238109","article-title":"Infeed antibiotic effects on the swine intestinal microbiome","volume":"109","author":"Looft","year":"2012","journal-title":"Proc Natl Acad Sci USA"},{"key":"2020040809051990500_bbx104-B43","doi-asserted-by":"crossref","first-page":"1084","DOI":"10.1126\/science.1233521","article-title":"Sex differences in the gut microbiome drive hormone-dependent regulation of autoimmunity","volume":"339","author":"Markle","year":"2013","journal-title":"Science"},{"key":"2020040809051990500_bbx104-B44","doi-asserted-by":"crossref","first-page":"e1000352","DOI":"10.1371\/journal.pcbi.1000352","article-title":"Statistical methods for detecting differentially abundant features in clinical metagenomic samples","volume":"5","author":"White","year":"2009","journal-title":"PLoS Comput Biol"},{"key":"2020040809051990500_bbx104-B45","doi-asserted-by":"crossref","first-page":"R95","DOI":"10.1186\/gb-2013-14-9-r95","article-title":"Comprehensive evaluation of differential gene expression analysis methods for RNA-seq data","volume":"14","author":"Rapaport","year":"2013","journal-title":"Genome Biol"},{"key":"2020040809051990500_bbx104-B46","doi-asserted-by":"crossref","first-page":"e78687","DOI":"10.1371\/journal.pone.0078687","article-title":"Low incidence of spontaneous type 1 diabetes in non-obese diabetic mice raised on gluten-free diets is associated with changes in the intestinal microbiome","volume":"8","author":"Marietta","year":"2013","journal-title":"PLoS One"},{"key":"2020040809051990500_bbx104-B47","doi-asserted-by":"crossref","first-page":"380","DOI":"10.1016\/j.annepidem.2016.03.007","article-title":"Impact of age and sex on the composition and abundance of the intestinal microbiota in individuals with and without enteric infections","volume":"26","author":"Singh","year":"2016","journal-title":"Ann Epidemiol"},{"key":"2020040809051990500_bbx104-B48","doi-asserted-by":"crossref","first-page":"470","DOI":"10.1016\/j.cell.2012.07.008","article-title":"Host remodeling of the gut microbiome and metabolic changes during pregnancy","volume":"150","author":"Koren","year":"2012","journal-title":"Cell"},{"key":"2020040809051990500_bbx104-B49","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1016\/j.tim.2013.01.001","article-title":"The uses of race and ethnicity in human microbiome research","volume":"21","author":"Fortenberry","year":"2013","journal-title":"Trends Microbiol"},{"key":"2020040809051990500_bbx104-B50","doi-asserted-by":"crossref","first-page":"e9085","DOI":"10.1371\/journal.pone.0009085","article-title":"Gut microbiota in human adults with type 2 diabetes differs from non-diabetic adults","volume":"5","author":"Larsen","year":"2010","journal-title":"PLoS One"},{"key":"2020040809051990500_bbx104-B51","doi-asserted-by":"crossref","first-page":"e61217","DOI":"10.1371\/journal.pone.0061217","article-title":"phyloseq: an R package for reproducible interactive analysis and graphics of microbiome census data","volume":"8","author":"McMurdie","year":"2013","journal-title":"PLoS One"},{"key":"2020040809051990500_bbx104-B52","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1186\/1471-2105-9-303","article-title":"A unified approach to false discovery rate estimation","volume":"9","author":"Strimmer","year":"2008","journal-title":"BMC Bioinformatics"},{"key":"2020040809051990500_bbx104-B53","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1177\/0962280211428386","article-title":"Finding consistent patterns: a nonparametric approach for identifying differential expression in RNA-Seq data","volume":"22","author":"Li","year":"2013","journal-title":"Stat Methods Med Res"},{"key":"2020040809051990500_bbx104-B54","doi-asserted-by":"crossref","first-page":"e52078","DOI":"10.1371\/journal.pone.0052078","article-title":"Hypothesis testing and power calculations for taxonomic-based human microbiome data","volume":"7","author":"La Rosa","year":"2012","journal-title":"PLoS One"},{"key":"2020040809051990500_bbx104-B55","doi-asserted-by":"crossref","first-page":"e91","DOI":"10.1093\/nar\/gku310","article-title":"Robustly detecting differential expression in RNA sequencing data using observation weights","volume":"42","author":"Zhou","year":"2014","journal-title":"Nucleic Acids Res"},{"key":"2020040809051990500_bbx104-B56","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1093\/bib\/bbs046","article-title":"A comprehensive evaluation of normalization methods for Illumina high-throughput RNA sequencing data analysis","volume":"14","author":"Dillies","year":"2013","journal-title":"Brief Bioinform"},{"key":"2020040809051990500_bbx104-B57","first-page":"10","article-title":"How many biological replicates are needed in an RNA-seq experiment and which differential expression tool should you use?","volume":"43","author":"Schurch","year":"2015","journal-title":"arXiv"},{"key":"2020040809051990500_bbx104-B58","doi-asserted-by":"crossref","first-page":"1684","DOI":"10.1261\/rna.046011.114","article-title":"Power analysis and sample size estimation for RNA-Seq differential expression","volume":"20","author":"Ching","year":"2014","journal-title":"RNA"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/20\/1\/210\/33025785\/bbx104.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/20\/1\/210\/33025785\/bbx104.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,26]],"date-time":"2024-06-26T00:25:41Z","timestamp":1719361541000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/20\/1\/210\/4091293"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,8,22]]},"references-count":58,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2017,8,22]]},"published-print":{"date-parts":[[2019,1,18]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbx104","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,1]]},"published":{"date-parts":[[2017,8,22]]}}}