{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,31]],"date-time":"2026-01-31T02:14:14Z","timestamp":1769825654147,"version":"3.49.0"},"reference-count":19,"publisher":"Oxford University Press (OUP)","issue":"19","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2014,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Motivation: FASTQ is a standard file format for DNA sequencing data, which stores both nucleotides and quality scores. A typical sequencing study can easily generate hundreds of gigabytes of FASTQ files, while public archives such as ENA and NCBI and large international collaborations such as the Cancer Genome Atlas can accumulate many terabytes of data in this format. Compression tools such as gzip are often used to reduce the storage burden but have the disadvantage that the data must be decompressed before they can be used.<\/jats:p><jats:p>Here, we present BEETL-fastq, a tool that not only compresses FASTQ-formatted DNA reads more compactly than gzip but also permits rapid search for k -mer queries within the archived sequences. Importantly, the full FASTQ record of each matching read or read pair is returned, allowing the search results to be piped directly to any of the many standard tools that accept FASTQ data as input.<\/jats:p><jats:p>Results: We show that 6.6 terabytes of human reads in FASTQ format can be transformed into 1.7 terabytes of indexed files, from where we can search for 1, 10, 100, 1000 and a million of 30-mers in 3, 8, 14, 45 and 567 s, respectively, plus 20 ms per output read. Useful applications of the search capability are highlighted, including the genotyping of structural variant breakpoints and \u2018 in silico pull-down\u2019 experiments in which only the reads that cover a region of interest are selectively extracted for the purposes of variant calling or visualization.<\/jats:p><jats:p>Availability and implementation: BEETL-fastq is part of the BEETL library, available as a github repository at github.com\/BEETL\/BEETL.<\/jats:p><jats:p>Contact: \u00a0acox@illumina.com<\/jats:p>","DOI":"10.1093\/bioinformatics\/btu387","type":"journal-article","created":{"date-parts":[[2014,6,21]],"date-time":"2014-06-21T00:24:41Z","timestamp":1403310281000},"page":"2796-2801","source":"Crossref","is-referenced-by-count":36,"title":["BEETL-fastq: a searchable compressed archive for DNA reads"],"prefix":"10.1093","volume":"30","author":[{"given":"Lilian","family":"Janin","sequence":"first","affiliation":[{"name":"Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK"}]},{"given":"Ole","family":"Schulz-Trieglaff","sequence":"additional","affiliation":[{"name":"Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK"}]},{"given":"Anthony J.","family":"Cox","sequence":"additional","affiliation":[{"name":"Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK"}]}],"member":"286","published-online":{"date-parts":[[2014,6,20]]},"reference":[{"key":"2023041303360382200_","doi-asserted-by":"crossref","DOI":"10.1007\/978-0-387-78909-5","volume-title":"The Burrows-Wheeler Transform: Data Compression, Suffix Arrays, and Pattern Matching","author":"Adjeroh","year":"2008","edition":"1st edn"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-21458-5_20","article-title":"Lightweight BWT construction for very large string collections","volume-title":"CPM (2011)","author":"Bauer","year":"2011"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1016\/j.tcs.2012.02.002","article-title":"Lightweight algorithms for constructing and inverting the BWT of string collections","volume":"483","author":"Bauer","year":"2013","journal-title":"Theor. Comput. Sci."},{"key":"2023041303360382200_","article-title":"A block sorting data compression algorithm","volume-title":"Technical report","author":"Burrows","year":"1994"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"1767","DOI":"10.1093\/nar\/gkp1137","article-title":"The sanger FASTQ file format for sequences with quality scores, and the Solexa\/Illumina FASTQ variants","volume":"38","author":"Cock","year":"2010","journal-title":"Nucleic Acids Res."},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"1415","DOI":"10.1093\/bioinformatics\/bts173","article-title":"Large-scale compression of genomic sequence databases with the Burrows-Wheeler transform","volume":"28","author":"Cox","year":"2012","journal-title":"Bioinformatics"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"390","DOI":"10.1109\/SFCS.2000.892127","article-title":"Opportunistic data structures with applications","volume-title":"Proceedings of the 41st Annual Symposium on Foundations of Computer Science","author":"Ferragina","year":"2000"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1093\/bioinformatics\/btt257","article-title":"Adaptive reference-free compression of sequence quality scores","volume":"30","author":"Janin","year":"2014","journal-title":"Bioinformatics"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"R25","DOI":"10.1186\/gb-2009-10-3-r25","article-title":"Ultrafast and memory-efficient alignment of short DNA sequences to the human genome","volume":"10","author":"Langmead","year":"2009","journal-title":"Genome Biol"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"1754","DOI":"10.1093\/bioinformatics\/btp324","article-title":"Fast and accurate short read alignment with Burrows-Wheeler transform","volume":"25","author":"Li","year":"2009","journal-title":"Bioinformatics"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"2078","DOI":"10.1093\/bioinformatics\/btp352","article-title":"The sequence alignment\/map format and samtools","volume":"25","author":"Li","year":"2009","journal-title":"Bioinformatics"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1101\/gr.097261.109","article-title":"De novo assembly of human genomes with massively parallel short read sequencing","volume":"20","author":"Li","year":"2010","journal-title":"Genome Res."},{"key":"2023041303360382200_","article-title":"GPU-accelerated BWT construction for large collection of short reads","author":"Liu","year":"2014"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"308","DOI":"10.1093\/bioinformatics\/bts690","article-title":"Musket: a multistage k -mer spectrum-based error corrector for illumina sequence data","volume":"29","author":"Liu","year":"2013","journal-title":"Bioinformatics"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1038\/nature09708","article-title":"Mapping copy number variation by population-scale genome sequencing","volume":"470","author":"Mills","year":"2011","journal-title":"Nature"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1101\/gr.126953.111","article-title":"Efficient de novo assembly of large genomes using compressed data structures","volume":"22","author":"Simpson","year":"2012","journal-title":"Genome Res."},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1038\/nature11632","article-title":"An integrated map of genetic variation from 1,092 human genomes","volume":"491","author":"The 1000 Genomes Project Consortium","year":"2012","journal-title":"Nature"},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"821","DOI":"10.1101\/gr.074492.107","article-title":"Velvet: algorithms for de novo short read assembly using de bruijn graphs","volume":"18","author":"Zerbino","year":"2008","journal-title":"Genome Res."},{"key":"2023041303360382200_","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1038\/nbt.2835","article-title":"Integrating human sequence data sets provides a resource of benchmark SNP and indel genotype calls","volume":"32","author":"Zook","year":"2014","journal-title":"Nat. Biotech."}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/30\/19\/2796\/49872391\/bioinformatics_30_19_2796.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/30\/19\/2796\/49872391\/bioinformatics_30_19_2796.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,14]],"date-time":"2023-07-14T13:14:48Z","timestamp":1689340488000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/30\/19\/2796\/2422232"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,6,20]]},"references-count":19,"journal-issue":{"issue":"19","published-print":{"date-parts":[[2014,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btu387","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2014,10]]},"published":{"date-parts":[[2014,6,20]]}}}