{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,13]],"date-time":"2026-01-13T06:15:29Z","timestamp":1768284929957,"version":"3.49.0"},"reference-count":45,"publisher":"Oxford University Press (OUP)","issue":"14","license":[{"start":{"date-parts":[[2019,7,8]],"date-time":"2019-07-08T00:00:00Z","timestamp":1562544000000},"content-version":"vor","delay-in-days":7,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"National Institute of Health","award":["1R15AI128714-01"],"award-info":[{"award-number":["1R15AI128714-01"]}]},{"DOI":"10.13039\/100000005","name":"Department of Defense","doi-asserted-by":"publisher","award":["W911NF-16-1-0494"],"award-info":[{"award-number":["W911NF-16-1-0494"]}],"id":[{"id":"10.13039\/100000005","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100005289","name":"National Institute of Justice","doi-asserted-by":"publisher","award":["582 2017-NE-BX-0001"],"award-info":[{"award-number":["582 2017-NE-BX-0001"]}],"id":[{"id":"10.13039\/100005289","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,7,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>Bacterial metagenomics profiling for metagenomic whole sequencing (mWGS) usually starts by aligning sequencing reads to a collection of reference genomes. Current profiling tools are designed to work against a small representative collection of genomes, and do not scale very well to larger reference genome collections. However, large reference genome collections are capable of providing a more complete and accurate profile of the bacterial population in a metagenomics dataset. In this paper, we discuss a scalable, efficient and affordable approach to this problem, bringing big data solutions within the reach of laboratories with modest resources.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>We developed Flint, a metagenomics profiling pipeline that is built on top of the Apache Spark framework, and is designed for fast real-time profiling of metagenomic samples against a large collection of reference genomes. Flint takes advantage of Spark\u2019s built-in parallelism and streaming engine architecture to quickly map reads against a large (170 GB) reference collection of 43 552 bacterial genomes from Ensembl. Flint runs on Amazon\u2019s Elastic MapReduce service, and is able to profile 1 million Illumina paired-end reads against over 40\u2009K genomes on 64 machines in 67\u2009s\u2014an order of magnitude faster than the state of the art, while using a much larger reference collection. Streaming the sequencing reads allows this approach to sustain mapping rates of 55 million reads per hour, at an hourly cluster cost of $8.00 USD, while avoiding the necessity of storing large quantities of intermediate alignments.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Availability and implementation<\/jats:title>\n                  <jats:p>Flint is open source software, available under the MIT License (MIT). Source code is available at https:\/\/github.com\/camilo-v\/flint.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Supplementary information<\/jats:title>\n                  <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btz356","type":"journal-article","created":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T19:14:13Z","timestamp":1557774853000},"page":"i13-i22","source":"Crossref","is-referenced-by-count":10,"title":["Large scale microbiome profiling in the cloud"],"prefix":"10.1093","volume":"35","author":[{"given":"Camilo","family":"Valdes","sequence":"first","affiliation":[{"name":"Bioinformatics Research Group (BioRG), School of Computing and Information Sciences, Florida International University, Miami, FL, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vitalii","family":"Stebliankin","sequence":"additional","affiliation":[{"name":"Bioinformatics Research Group (BioRG), School of Computing and Information Sciences, Florida International University, Miami, FL, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Giri","family":"Narasimhan","sequence":"additional","affiliation":[{"name":"Bioinformatics Research Group (BioRG), School of Computing and Information Sciences, Florida International University, Miami, FL, USA"},{"name":"Biomolecular Sciences Institute, Florida International University, Miami, FL, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2019,7,5]]},"reference":[{"key":"2023062712330686800_btz356-B1","year":"2018"},{"key":"2023062712330686800_btz356-B2","year":"2018"},{"key":"2023062712330686800_btz356-B3","year":"2018"},{"key":"2023062712330686800_btz356-B4","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1016\/j.nbt.2008.12.009","article-title":"Next-generation DNA sequencing techniques","volume":"25","author":"Ansorge","year":"2009","journal-title":"New Biotechnol"},{"key":"2023062712330686800_btz356-B5","year":"2018"},{"key":"2023062712330686800_btz356-B6","year":"2018"},{"key":"2023062712330686800_btz356-B7","doi-asserted-by":"crossref","first-page":"1621","DOI":"10.1038\/ismej.2012.8","article-title":"Ultra-high-throughput microbial community analysis on the Illumina HiSeq and MiSeq platforms","volume":"6","author":"Caporaso","year":"2012","journal-title":"ISME J"},{"key":"2023062712330686800_btz356-B8","first-page":"53","volume-title":"Advances in Artificial Life, Evolutionary Computation, and Systems Chemistry","author":"Cattaneo","year":"2016"},{"key":"2023062712330686800_btz356-B9","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1145\/1327452.1327492","article-title":"MapReduce: simplified data processing on large clusters","volume":"51","author":"Dean","year":"2008","journal-title":"Commun. ACM"},{"key":"2023062712330686800_btz356-B10","year":"2018"},{"key":"2023062712330686800_btz356-B11","year":"2018"},{"key":"2023062712330686800_btz356-B12","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1093\/bioinformatics\/bty630","article-title":"Simulating Illumina metagenomic data with InSilicoSeq","volume":"35","author":"Gourl\u00e9","year":"2018","journal-title":"Bioinformatics"},{"key":"2023062712330686800_btz356-B13","first-page":"giy098","article-title":"Bioinformatics applications on Apache Spark","volume":"7","author":"Guo","year":"2018","journal-title":"GigaScience"},{"key":"2023062712330686800_btz356-B14","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1126\/science.1235872","article-title":"Predicting and manipulating cardiac drug inactivation by the human gut Bacterium Eggerthella lenta","volume":"341","author":"Haiser","year":"2013","journal-title":"Science (New York, NY)"},{"key":"2023062712330686800_btz356-B15","first-page":"1457","article-title":"Analyzing large scale genomic data on the cloud with Sparkhit","volume":"34","author":"Huang","year":"2018","journal-title":"Bioinformatics (Oxford, England)"},{"key":"2023062712330686800_btz356-B16","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1038\/nature11209","article-title":"A framework for human microbiome research","volume":"486","year":"2012","journal-title":"Nature"},{"key":"2023062712330686800_btz356-B17","year":"2019"},{"key":"2023062712330686800_btz356-B18","doi-asserted-by":"crossref","first-page":"576","DOI":"10.1038\/nm.3145","article-title":"Intestinal microbiota metabolism of L-carnitine, a nutrient in red meat, promotes atherosclerosis","volume":"19","author":"Koeth","year":"2013","journal-title":"Nat. Med"},{"key":"2023062712330686800_btz356-B19","doi-asserted-by":"crossref","first-page":"R134","DOI":"10.1186\/gb-2009-10-11-r134","article-title":"Searching for SNPs with cloud computing","volume":"10","author":"Langmead","year":"2009","journal-title":"Genome Biol"},{"key":"2023062712330686800_btz356-B20","doi-asserted-by":"crossref","first-page":"R25.","DOI":"10.1186\/gb-2009-10-3-r25","article-title":"Ultrafast and memory-efficient alignment of short DNA sequences to the human genome","volume":"10","author":"Langmead","year":"2009","journal-title":"Genome Biol"},{"key":"2023062712330686800_btz356-B21","doi-asserted-by":"crossref","first-page":"gkw290","DOI":"10.1093\/nar\/gkw290","article-title":"Interactive tree of life (itol) v3: an online tool for the display and annotation of phylogenetic and other trees","volume":"44","author":"Letunic","year":"2016","journal-title":"Nucleic Acids Res"},{"key":"2023062712330686800_btz356-B22","first-page":"2078","article-title":"The Sequence Alignment\/Map format and SAMtools","volume":"25","author":"Li","year":"2009","journal-title":"Bioinformatics (Oxford, England)"},{"key":"2023062712330686800_btz356-B23","article-title":"HiLive: real-time mapping of Illumina reads while sequencing","volume":"33","author":"Lindner","year":"2017","journal-title":"Bioinformatics (Oxford, England"},{"key":"2023062712330686800_btz356-B24","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1186\/s13059-017-1299-7","article-title":"Comprehensive benchmarking and ensemble approaches for metagenomic classifiers","volume":"18","author":"McIntyre","year":"2017","journal-title":"Genome Biol"},{"key":"2023062712330686800_btz356-B25","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1038\/nrg2626","article-title":"Sequencing technologies \u2013 the next generation","volume":"11","author":"Metzker","year":"2010","journal-title":"Nat. Rev. Genet"},{"key":"2023062712330686800_btz356-B26","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1186\/s13059-018-1554-6","article-title":"RefSeq database growth influences the accuracy of k-mer-based lowest common ancestor species identification","volume":"19","author":"Nasko","year":"2018","journal-title":"Genome Biol"},{"key":"2023062712330686800_btz356-B27","doi-asserted-by":"crossref","first-page":"967","DOI":"10.1016\/j.bbrc.2015.12.083","article-title":"Analysis of the microbiome: advantages of whole genome shotgun versus 16S amplicon sequencing","volume":"469","author":"Ranjan","year":"2016","journal-title":"Biochem. Biophys. Res. Commun"},{"key":"2023062712330686800_btz356-B28","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1186\/1471-2105-14-358","article-title":"Fragment assignment in the cloud with eXpress-D","volume":"14","author":"Roberts","year":"2013","journal-title":"BMC Bioinformatics"},{"key":"2023062712330686800_btz356-B29","first-page":"549","article-title":"A map-reduce framework for clustering metagenomes","volume-title":"2013 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum, May 20\u201324, 2013","author":"Rasheed","year":"2013"},{"key":"2023062712330686800_btz356-B30","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1186\/s40537-018-0114-y","article-title":"StreamAligner: a streaming based sequence aligner on Apache Spark","volume":"5","author":"Rathee","year":"2018","journal-title":"J. Big Data"},{"key":"2023062712330686800_btz356-B31","author":"Schaeffer","year":"2015"},{"key":"2023062712330686800_btz356-B32","article-title":"LiveKraken \u2013 Real-time metagenomic classification of Illumina data","author":"Tausch","year":"2018","journal-title":"Bioinformatics (Oxford, England)"},{"key":"2023062712330686800_btz356-B33","year":"2018"},{"key":"2023062712330686800_btz356-B34","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1016\/j.chom.2014.08.014","article-title":"The integrative human microbiome project: dynamic analysis of microbiome-host omics profiles during periods of human health and disease","volume":"16","year":"2014","journal-title":"Cell Host Microbe"},{"key":"2023062712330686800_btz356-B35","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1038\/nbt.1621","article-title":"Transcript assembly and quantification by RNA-Seq reveals unannotated transcripts and isoform switching during cell differentiation","volume":"28","author":"Trapnell","year":"2010","journal-title":"Nat. Biotechnol"},{"key":"2023062712330686800_btz356-B36","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1038\/nbt0509-455","article-title":"How to map billions of short reads onto genomes","volume":"27","author":"Trapnell","year":"2009","journal-title":"Nat. Biotechnol"},{"key":"2023062712330686800_btz356-B37","doi-asserted-by":"crossref","first-page":"477","DOI":"10.4310\/SII.2015.v8.n4.a7","article-title":"Detecting bacterial genomes in a metagenomic sample using NGS reads","volume":"8","author":"Valdes","year":"2015","journal-title":"Stat. Interface"},{"key":"2023062712330686800_btz356-B38","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1016\/j.mib.2014.11.016","article-title":"Ten years of pan-genome analyses","volume":"23","author":"Vernikos","year":"2015","journal-title":"Curr. Opin. Microbiol"},{"key":"2023062712330686800_btz356-B39","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1038\/nrg2484","article-title":"RNA-Seq: a revolutionary tool for transcriptomics","volume":"10","author":"Wang","year":"2009","journal-title":"Nat. Rev. Genet"},{"key":"2023062712330686800_btz356-B40","doi-asserted-by":"crossref","first-page":"R46","DOI":"10.1186\/gb-2014-15-3-r46","article-title":"Kraken: ultrafast metagenomic sequence classification using exact alignments","volume":"15","author":"Wood","year":"2014","journal-title":"Genome Biol"},{"key":"2023062712330686800_btz356-B41","doi-asserted-by":"crossref","first-page":"774","DOI":"10.1016\/j.cgh.2013.03.038","article-title":"Analysis of the human gut microbiome and association with disease","volume":"11","author":"Wu","year":"2013","journal-title":"Clin. Gastroenterol. Hepatol"},{"key":"2023062712330686800_btz356-B42","first-page":"15","author":"Zaharia","year":"2012"},{"key":"2023062712330686800_btz356-B43","doi-asserted-by":"crossref","first-page":"26263","DOI":"10.3390\/ijms161125957","article-title":"Metagenomics: a new way to illustrate the crosstalk between infectious diseases and host microbiome","volume":"16","author":"Zhang","year":"2015","journal-title":"Int. J. Mol. Sci"},{"key":"2023062712330686800_btz356-B44","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1186\/s40168-018-0399-2","article-title":"ReprDB and panDB: minimalist databases with maximal microbial representation","volume":"6","author":"Zhou","year":"2018","journal-title":"Microbiome"},{"key":"2023062712330686800_btz356-B45","first-page":"1090","article-title":"MetaSpark: a spark-based distributed processing tool to recruit metagenomic reads to reference genomes","volume":"33","author":"Zhou","year":"2017","journal-title":"Bioinformatics (Oxford, England)"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/35\/14\/i13\/50720306\/bioinformatics_35_14_i13.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/35\/14\/i13\/50720306\/bioinformatics_35_14_i13.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,27]],"date-time":"2023-06-27T12:34:18Z","timestamp":1687869258000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/35\/14\/i13\/5529230"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7]]},"references-count":45,"journal-issue":{"issue":"14","published-print":{"date-parts":[[2019,7,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btz356","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,7]]},"published":{"date-parts":[[2019,7]]}}}