{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,12]],"date-time":"2026-04-12T06:36:53Z","timestamp":1775975813725,"version":"3.50.1"},"reference-count":22,"publisher":"Springer Science and Business Media LLC","issue":"1","content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"published-print":{"date-parts":[[2010,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:sec>\n            <jats:title>Background<\/jats:title>\n            <jats:p>Artificial duplicates from pyrosequencing reads may lead to incorrect interpretation of the abundance of species and genes in metagenomic studies. Duplicated reads were filtered out in many metagenomic projects. However, since the duplicated reads observed in a pyrosequencing run also include natural (non-artificial) duplicates, simply removing all duplicates may also cause underestimation of abundance associated with natural duplicates.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Results<\/jats:title>\n            <jats:p>We implemented a method for identification of exact and nearly identical duplicates from pyrosequencing reads. This method performs an all-against-all sequence comparison and clusters the duplicates into groups using an algorithm modified from our previous sequence clustering method cd-hit. This method can process a typical dataset in ~10 minutes; it also provides a consensus sequence for each group of duplicates. We applied this method to the underlying raw reads of 39 genomic projects and 10 metagenomic projects that utilized pyrosequencing technique. We compared the occurrences of the duplicates identified by our method and the natural duplicates made by independent simulations. We observed that the duplicates, including both artificial and natural duplicates, make up 4-44% of reads. The number of natural duplicates highly correlates with the samples' read density (number of reads divided by genome size). For high-complexity metagenomic samples lacking dominant species, natural duplicates only make up &lt;1% of all duplicates. But for some other samples like transcriptomic samples, majority of the observed duplicates might be natural duplicates.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Conclusions<\/jats:title>\n            <jats:p>Our method is available from <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"http:\/\/cd-hit.org\" ext-link-type=\"uri\">http:\/\/cd-hit.org<\/jats:ext-link> as a downloadable program and a web server. It is important not only to identify the duplicates from metagenomic datasets but also to distinguish whether they are artificial or natural duplicates. We provide a tool to estimate the number of natural duplicates according to user-defined sample types, so users can decide whether to retain or remove duplicates in their projects.<\/jats:p>\n          <\/jats:sec>","DOI":"10.1186\/1471-2105-11-187","type":"journal-article","created":{"date-parts":[[2010,4,13]],"date-time":"2010-04-13T18:14:55Z","timestamp":1271182495000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":228,"title":["Artificial and natural duplicates in pyrosequencing reads of metagenomic data"],"prefix":"10.1186","volume":"11","author":[{"given":"Beifang","family":"Niu","sequence":"first","affiliation":[]},{"given":"Limin","family":"Fu","sequence":"additional","affiliation":[]},{"given":"Shulei","family":"Sun","sequence":"additional","affiliation":[]},{"given":"Weizhong","family":"Li","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2010,4,13]]},"reference":[{"issue":"3","key":"3644_CR1","doi-asserted-by":"publisher","first-page":"e77","DOI":"10.1371\/journal.pbio.0050077","volume":"5","author":"DB Rusch","year":"2007","unstructured":"Rusch DB, Halpern AL, Sutton G, Heidelberg KB, Williamson S, Yooseph S, Wu D, Eisen JA, Hoffman JM, Remington K, et al.: The Sorcerer II Global Ocean Sampling Expedition: Northwest Atlantic through Eastern Tropical Pacific. PLoS Biol 2007, 5(3):e77. 10.1371\/journal.pbio.0050077","journal-title":"PLoS Biol"},{"issue":"5667","key":"3644_CR2","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1126\/science.1093857","volume":"304","author":"JC Venter","year":"2004","unstructured":"Venter JC, Remington K, Heidelberg JF, Halpern AL, Rusch D, Eisen JA, Wu D, Paulsen I, Nelson KE, Nelson W, et al.: Environmental genome shotgun sequencing of the Sargasso Sea. Science 2004, 304(5667):66\u201374. 10.1126\/science.1093857","journal-title":"Science"},{"issue":"5721","key":"3644_CR3","doi-asserted-by":"publisher","first-page":"554","DOI":"10.1126\/science.1107851","volume":"308","author":"SG Tringe","year":"2005","unstructured":"Tringe SG, von Mering C, Kobayashi A, Salamov AA, Chen K, Chang HW, Podar M, Short JM, Mathur EJ, Detter JC, et al.: Comparative metagenomics of microbial communities. Science 2005, 308(5721):554\u2013557. 10.1126\/science.1107851","journal-title":"Science"},{"issue":"5778","key":"3644_CR4","doi-asserted-by":"publisher","first-page":"1355","DOI":"10.1126\/science.1124234","volume":"312","author":"SR Gill","year":"2006","unstructured":"Gill SR, Pop M, Deboy RT, Eckburg PB, Turnbaugh PJ, Samuel BS, Gordon JI, Relman DA, Fraser-Liggett CM, Nelson KE: Metagenomic analysis of the human distal gut microbiome. Science 2006, 312(5778):1355\u20131359. 10.1126\/science.1124234","journal-title":"Science"},{"issue":"6978","key":"3644_CR5","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1038\/nature02340","volume":"428","author":"GW Tyson","year":"2004","unstructured":"Tyson GW, Chapman J, Hugenholtz P, Allen EE, Ram RJ, Richardson PM, Solovyev VV, Rubin EM, Rokhsar DS, Banfield JF: Community structure and metabolism through reconstruction of microbial genomes from the environment. Nature 2004, 428(6978):37\u201343. 10.1038\/nature02340","journal-title":"Nature"},{"issue":"5760","key":"3644_CR6","doi-asserted-by":"publisher","first-page":"496","DOI":"10.1126\/science.1120250","volume":"311","author":"EF DeLong","year":"2006","unstructured":"DeLong EF, Preston CM, Mincer T, Rich V, Hallam SJ, Frigaard NU, Martinez A, Sullivan MB, Edwards R, Brito BR, et al.: Community genomics among stratified microbial assemblages in the ocean's interior. Science 2006, 311(5760):496\u2013503. 10.1126\/science.1120250","journal-title":"Science"},{"issue":"7187","key":"3644_CR7","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1038\/nature06810","volume":"452","author":"EA Dinsdale","year":"2008","unstructured":"Dinsdale EA, Edwards RA, Hall D, Angly F, Breitbart M, Brulc JM, Furlan M, Desnues C, Haynes M, Li L, et al.: Functional metagenomic profiling of nine biomes. Nature 2008, 452(7187):629\u2013632. 10.1038\/nature06810","journal-title":"Nature"},{"issue":"10","key":"3644_CR8","doi-asserted-by":"publisher","first-page":"3805","DOI":"10.1073\/pnas.0708897105","volume":"105","author":"J Frias-Lopez","year":"2008","unstructured":"Frias-Lopez J, Shi Y, Tyson GW, Coleman ML, Schuster SC, Chisholm SW, Delong EF: Microbial community gene expression in ocean surface waters. Proc Natl Acad Sci USA 2008, 105(10):3805\u20133810. 10.1073\/pnas.0708897105","journal-title":"Proc Natl Acad Sci USA"},{"issue":"7228","key":"3644_CR9","doi-asserted-by":"publisher","first-page":"480","DOI":"10.1038\/nature07540","volume":"457","author":"PJ Turnbaugh","year":"2009","unstructured":"Turnbaugh PJ, Hamady M, Yatsunenko T, Cantarel BL, Duncan A, Ley RE, Sogin ML, Jones WJ, Roe BA, Affourtit JP, et al.: A core gut microbiome in obese and lean twins. Nature 2009, 457(7228):480\u2013484. 10.1038\/nature07540","journal-title":"Nature"},{"issue":"10","key":"3644_CR10","doi-asserted-by":"publisher","first-page":"1135","DOI":"10.1038\/nbt1486","volume":"26","author":"J Shendure","year":"2008","unstructured":"Shendure J, Ji H: Next-generation DNA sequencing. Nat Biotechnol 2008, 26(10):1135\u20131145. 10.1038\/nbt1486","journal-title":"Nat Biotechnol"},{"issue":"11","key":"3644_CR11","doi-asserted-by":"publisher","first-page":"1314","DOI":"10.1038\/ismej.2009.72","volume":"3","author":"V Gomez-Alvarez","year":"2009","unstructured":"Gomez-Alvarez V, Teal TK, Schmidt TM: Systematic artifacts in metagenomes from complex microbial communities. Isme J 2009, 3(11):1314\u20131317. 10.1038\/ismej.2009.72","journal-title":"Isme J"},{"issue":"3","key":"3644_CR12","doi-asserted-by":"publisher","first-page":"282","DOI":"10.1093\/bioinformatics\/17.3.282","volume":"17","author":"W Li","year":"2001","unstructured":"Li W, Jaroszewski L, Godzik A: Clustering of highly homologous sequences to reduce the size of large protein databases. Bioinformatics 2001, 17(3):282\u2013283. 10.1093\/bioinformatics\/17.3.282","journal-title":"Bioinformatics"},{"issue":"1","key":"3644_CR13","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1093\/bioinformatics\/18.1.77","volume":"18","author":"W Li","year":"2002","unstructured":"Li W, Jaroszewski L, Godzik A: Tolerating some redundancy significantly speeds up clustering of large protein databases. Bioinformatics 2002, 18(1):77\u201382. 10.1093\/bioinformatics\/18.1.77","journal-title":"Bioinformatics"},{"issue":"13","key":"3644_CR14","doi-asserted-by":"publisher","first-page":"1658","DOI":"10.1093\/bioinformatics\/btl158","volume":"22","author":"W Li","year":"2006","unstructured":"Li W, Godzik A: Cd-hit: a fast program for clustering and comparing large sets of protein or nucleotide sequences. Bioinformatics 2006, 22(13):1658\u20131659. 10.1093\/bioinformatics\/btl158","journal-title":"Bioinformatics"},{"issue":"7057","key":"3644_CR15","doi-asserted-by":"crossref","first-page":"376","DOI":"10.1038\/nature03959","volume":"437","author":"M Margulies","year":"2005","unstructured":"Margulies M, Egholm M, Altman WE, Attiya S, Bader JS, Bemben LA, Berka J, Braverman MS, Chen YJ, Chen Z, et al.: Genome sequencing in microfabricated high-density picolitre reactors. Nature 2005, 437(7057):376\u2013380.","journal-title":"Nature"},{"issue":"7","key":"3644_CR16","doi-asserted-by":"publisher","first-page":"R143","DOI":"10.1186\/gb-2007-8-7-r143","volume":"8","author":"SM Huse","year":"2007","unstructured":"Huse SM, Huber JA, Morrison HG, Sogin ML, Welch DM: Accuracy and quality of massively parallel DNA pyrosequencing. Genome Biol 2007, 8(7):R143. 10.1186\/gb-2007-8-7-r143","journal-title":"Genome Biol"},{"issue":"2","key":"3644_CR17","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1038\/nmeth.1172","volume":"5","author":"AR Quinlan","year":"2008","unstructured":"Quinlan AR, Stewart DA, Stromberg MP, Marth GT: Pyrobayes: an improved base caller for SNP discovery in pyrosequences. Nat Methods 2008, 5(2):179\u2013181. 10.1038\/nmeth.1172","journal-title":"Nat Methods"},{"issue":"1-2","key":"3644_CR18","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1089\/10665270050081478","volume":"7","author":"Z Zhang","year":"2000","unstructured":"Zhang Z, Schwartz S, Wagner L, Miller W: A greedy algorithm for aligning DNA sequences. J Comput Biol 2000, 7(1\u20132):203\u2013214. 10.1089\/10665270050081478","journal-title":"J Comput Biol"},{"issue":"6","key":"3644_CR19","doi-asserted-by":"publisher","first-page":"495","DOI":"10.1038\/nmeth1043","volume":"4","author":"K Mavromatis","year":"2007","unstructured":"Mavromatis K, Ivanova N, Barry K, Shapiro H, Goltsman E, McHardy AC, Rigoutsos I, Salamov A, Korzeniewski F, Land M, et al.: Use of simulated data sets to evaluate the fidelity of metagenomic processing methods. Nat Methods 2007, 4(6):495\u2013500. 10.1038\/nmeth1043","journal-title":"Nat Methods"},{"issue":"6","key":"3644_CR20","doi-asserted-by":"publisher","first-page":"1358","DOI":"10.1111\/j.1462-2920.2008.01863.x","volume":"11","author":"RS Poretsky","year":"2009","unstructured":"Poretsky RS, Hewson I, Sun S, Allen AE, Zehr JP, Moran MA: Comparative day\/night metatranscriptomic analysis of microbial communities in the North Pacific subtropical gyre. Environ Microbiol 2009, 11(6):1358\u20131375. 10.1111\/j.1462-2920.2008.01863.x","journal-title":"Environ Microbiol"},{"issue":"8","key":"3644_CR21","doi-asserted-by":"publisher","first-page":"e3042","DOI":"10.1371\/journal.pone.0003042","volume":"3","author":"JA Gilbert","year":"2008","unstructured":"Gilbert JA, Field D, Huang Y, Edwards R, Li W, Gilna P, Joint I: Detection of large numbers of novel sequences in the metatranscriptomes of complex marine microbial communities. PLoS ONE 2008, 3(8):e3042. 10.1371\/journal.pone.0003042","journal-title":"PLoS ONE"},{"issue":"22","key":"3644_CR22","doi-asserted-by":"publisher","first-page":"4673","DOI":"10.1093\/nar\/22.22.4673","volume":"22","author":"JD Thompson","year":"1994","unstructured":"Thompson JD, Higgins DG, Gibson TJ: CLUSTAL W: improving the sensitivity of progressive multiple sequence alignment through sequence weighting, position-specific gap penalties and weight matrix choice. Nucleic Acids Res 1994, 22(22):4673\u20134680. 10.1093\/nar\/22.22.4673","journal-title":"Nucleic Acids Res"}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/1471-2105-11-187.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,1]],"date-time":"2021-09-01T05:25:21Z","timestamp":1630473921000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/1471-2105-11-187"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,4,13]]},"references-count":22,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2010,12]]}},"alternative-id":["3644"],"URL":"https:\/\/doi.org\/10.1186\/1471-2105-11-187","relation":{},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2010,4,13]]},"assertion":[{"value":"5 January 2010","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 April 2010","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 April 2010","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"187"}}