{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T12:16:24Z","timestamp":1763468184221},"reference-count":22,"publisher":"Oxford University Press (OUP)","issue":"19","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2014,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Motivation: Computational methods are essential to extract actionable information from raw sequencing data, and to thus fulfill the promise of next-generation sequencing technology. Unfortunately, computational tools developed to call variants from human sequencing data disagree on many of their predictions, and current methods to evaluate accuracy and computational performance are ad hoc and incomplete. Agreement on benchmarking variant calling methods would stimulate development of genomic processing tools and facilitate communication among researchers.<\/jats:p><jats:p>Results: We propose SM a SH, a benchmarking methodology for evaluating germline variant calling algorithms. We generate synthetic datasets, organize and interpret a wide range of existing benchmarking data for real genomes and propose a set of accuracy and computational performance metrics for evaluating variant calling methods on these benchmarking data. Moreover, we illustrate the utility of SM a SH to evaluate the performance of some leading single-nucleotide polymorphism, indel and structural variant calling algorithms.<\/jats:p><jats:p>Availability and implementation: We provide free and open access online to the SM a SH tool kit, along with detailed documentation, at smash.cs.berkeley.edu<\/jats:p><jats:p>Contact: \u00a0ameet@cs.berkeley.edu or pattrsn@cs.berkeley.edu<\/jats:p><jats:p>Supplementary information: \u00a0Supplementary data are available at Bioinformatics online.<\/jats:p>","DOI":"10.1093\/bioinformatics\/btu345","type":"journal-article","created":{"date-parts":[[2014,6,4]],"date-time":"2014-06-04T05:56:45Z","timestamp":1401861405000},"page":"2787-2795","source":"Crossref","is-referenced-by-count":32,"title":["SM<scp>a<\/scp>SH: a benchmarking toolkit for human genome variant calling"],"prefix":"10.1093","volume":"30","author":[{"given":"Ameet","family":"Talwalkar","sequence":"first","affiliation":[{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jesse","family":"Liptrap","sequence":"additional","affiliation":[{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Julie","family":"Newcomb","sequence":"additional","affiliation":[{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christopher","family":"Hartl","sequence":"additional","affiliation":[{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"},{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jonathan","family":"Terhorst","sequence":"additional","affiliation":[{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kristal","family":"Curtis","sequence":"additional","affiliation":[{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ma\u2019ayan","family":"Bresler","sequence":"additional","affiliation":[{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yun S.","family":"Song","sequence":"additional","affiliation":[{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"},{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael I.","family":"Jordan","sequence":"additional","affiliation":[{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"},{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Patterson","sequence":"additional","affiliation":[{"name":"1 Department of Electrical Engineering and Computer Science, UC Berkeley, Berkeley, CA 94720, USA, 2 The Broad Institute of Harvard and MIT, Cambridge, MA 02142, USA and 3 Department of Statistics, UC Berkeley, Berkeley, CA 94720, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2014,6,3]]},"reference":[{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"961","DOI":"10.1101\/gr.112326.110","article-title":"Dindel: accurate indel calls from short-read data","volume":"21","author":"Albers","year":"2011","journal-title":"Genome Res."},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1038\/nrg2958","article-title":"Genome structural variation discovery and genotyping","volume":"12","author":"Alkan","year":"2011","journal-title":"Nat. Rev. Genet."},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"677","DOI":"10.1038\/nmeth.1363","article-title":"BreakDancer: an algorithm for high-resolution mapping of genomic structural variation","volume":"6","author":"Chen","year":"2009","journal-title":"Nat. Methods"},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"e1000112","DOI":"10.1371\/journal.pbio.1000112","article-title":"Lineage-specific biology revealed by a finished genome assembly of the mouse","volume":"7","author":"Church","year":"2009","journal-title":"PLoS Biol."},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"1061","DOI":"10.1038\/nature09534","article-title":"A map of human genome variation from population-scale sequencing","volume":"467","author":"The 1000 Genomes Project Consortium","year":"2010","journal-title":"Nature"},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"491","DOI":"10.1038\/ng.806","article-title":"A framework for variation discovery and genotyping using next-generation DNA sequencing data","volume":"43","author":"DePristo","year":"2011","journal-title":"Nat. Genet."},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"2224","DOI":"10.1101\/gr.126599.111","article-title":"Assemblathon 1: a competitive assessment of de novo short read assembly methods","volume":"21","author":"Earl","year":"2011","journal-title":"Genome Res."},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"851","DOI":"10.1038\/nature06258","article-title":"A second generation human HAPlotype map of over 3.1 million SNPs","volume":"449","author":"Frazer","year":"2007","journal-title":"Nature"},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"1513","DOI":"10.1073\/pnas.1017351108","article-title":"High-quality draft assemblies of mammalian genomes from massively parallel sequence data","volume":"108","author":"Gnerre","year":"2011","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1038\/nature09298","article-title":"Integrating common and rare genetic variation in diverse human populations","volume":"467","author":"The HapMap Consortium","year":"2010","journal-title":"Nature"},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"1055","DOI":"10.1038\/ng.988","article-title":"The new date, new format, new goals and new sponsor of the archon genomics x PRIZE competition","volume":"43","author":"Kedes","year":"2011","journal-title":"Nat. Genet."},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1038\/nmeth.1451","article-title":"Characterization of missing human genome sequences and copy-number polymorphic insertions","volume":"7","author":"Kidd","year":"2010","journal-title":"Nat. Methods"},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"837","DOI":"10.1016\/j.cell.2010.10.027","article-title":"A human genome structural variation sequencing resource reveals insights into mutational mechanisms","volume":"143","author":"Kidd","year":"2010","journal-title":"Cell"},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"e254","DOI":"10.1371\/journal.pbio.0050254","article-title":"The diploid genome sequence of an individual human","volume":"5","author":"Levy","year":"2007","journal-title":"PLoS Biol."},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"2078","DOI":"10.1093\/bioinformatics\/btp352","article-title":"The sequence alignment\/map (sam) format and samtools","volume":"25","author":"Li","year":"2009","journal-title":"Bioinformatics"},{"key":"2023041303360178600_","article-title":"Low concordance of variant calling algorithms in exome sequencing","volume-title":"Meeting of The American Society of Human Genetics","author":"Lyon","year":"2012"},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1186\/gm205","article-title":"The $1,000 genome, the $100,000 analysis?","volume":"2","author":"Mardis","year":"2010","journal-title":"Genome Med."},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"667","DOI":"10.1038\/nrg3305","article-title":"Next-generation sequencing data interpretation: enhancing reproducibility and accessibility","volume":"13","author":"Nekrutenko","year":"2012","journal-title":"Nat. Rev. Genet."},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1145\/2209249.2209271","article-title":"For better or worse, benchmarks shape a field: technical perspective","volume":"55","author":"Patterson","year":"2012","journal-title":"Commun. ACM."},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"326","DOI":"10.1038\/nature10432","article-title":"Sequence-based characterization of structural variation in the mouse genome","volume":"477","author":"Yalcin","year":"2011","journal-title":"Nature"},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"2865","DOI":"10.1093\/bioinformatics\/btp394","article-title":"Pindel: a pattern growth approach to detect break points of large deletions and medium sized insertions from paired-end short reads","volume":"25","author":"Ye","year":"2009","journal-title":"Bioinformatics"},{"key":"2023041303360178600_","doi-asserted-by":"crossref","first-page":"P31","DOI":"10.1186\/gb-2011-12-s1-p31","article-title":"Genomes in a bottle: creating standard reference materials for genomic variation - why, what and how?","volume":"12","author":"Zook","year":"2011","journal-title":"Genome Biol."}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/30\/19\/2787\/49872377\/bioinformatics_30_19_2787.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/30\/19\/2787\/49872377\/bioinformatics_30_19_2787.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,13]],"date-time":"2023-07-13T23:07:46Z","timestamp":1689289666000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/30\/19\/2787\/2422156"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,6,3]]},"references-count":22,"journal-issue":{"issue":"19","published-print":{"date-parts":[[2014,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btu345","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2014,10]]},"published":{"date-parts":[[2014,6,3]]}}}