{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T03:28:23Z","timestamp":1777346903421,"version":"3.51.4"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,11,8]],"date-time":"2019-11-08T00:00:00Z","timestamp":1573171200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,11,8]],"date-time":"2019-11-08T00:00:00Z","timestamp":1573171200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100010663","name":"H2020 European Research Council","doi-asserted-by":"publisher","award":["693174"],"award-info":[{"award-number":["693174"]}],"id":[{"id":"10.13039\/100010663","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100008536","name":"Amazon Web Services","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100008536","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"published-print":{"date-parts":[[2019,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n              <jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>With the growth of available sequenced datasets, analysis of heterogeneous processed data can answer increasingly relevant biological and clinical questions. Scientists are challenged in performing efficient and reproducible data extraction and analysis pipelines over heterogeneously processed datasets. Available software packages are suitable for analyzing experimental files from such datasets one by one, but do not scale to thousands of experiments. Moreover, they lack proper support for metadata manipulation.<\/jats:p>\n              <\/jats:sec>\n              <jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>We present PyGMQL, a novel software for the manipulation of region-based genomic files and their relative metadata, built on top of the GMQL genomic big data management system. PyGMQL provides a set of expressive functions for the manipulation of region data and their metadata that can scale to arbitrary clusters and implicitly apply to thousands of files, producing millions of regions. PyGMQL provides data interoperability, distribution transparency and query outsourcing. The PyGMQL package integrates scalable data extraction over the Apache Spark engine underlying the GMQL implementation with native Python support for interactive data analysis and visualization. It supports data interoperability, solving the impedance mismatch between executing set-oriented queries and programming in Python. PyGMQL provides distribution transparency (the ability to address a remote dataset) and query outsourcing (the ability to assign processing to a remote service) in an orthogonal way. Outsourced processing can address cloud-based installations of the GMQL engine.<\/jats:p>\n              <\/jats:sec>\n              <jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>PyGMQL is an effective and innovative tool for supporting tertiary data extraction and analysis pipelines. We demonstrate the expressiveness and performance of PyGMQL through a sequence of biological data analysis scenarios of increasing complexity, which highlight reproducibility, expressive power and scalability.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12859-019-3159-9","type":"journal-article","created":{"date-parts":[[2019,11,8]],"date-time":"2019-11-08T15:36:12Z","timestamp":1573227372000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["PyGMQL: scalable data extraction and analysis for heterogeneous genomic datasets"],"prefix":"10.1186","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7021-2007","authenticated-orcid":false,"given":"Luca","family":"Nanni","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pietro","family":"Pinoli","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Arif","family":"Canakoglu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefano","family":"Ceri","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2019,11,8]]},"reference":[{"issue":"3","key":"3159_CR1","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1038\/gim.2012.116","volume":"15","author":"S Moorthie","year":"2013","unstructured":"Moorthie S, Hall A, Wright CF. Informatics and clinical genome sequencing: opening the black box. Genet Med. 2013; 15(3):165.","journal-title":"Genet Med"},{"key":"3159_CR2","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/j.ymeth.2016.09.002","volume":"111","author":"M Masseroli","year":"2016","unstructured":"Masseroli M, et al.Modeling and interoperability of heterogeneous genomic big data for integrative processing and querying. Methods. 2016; 111:3\u201311. \nhttps:\/\/doi.org\/10.1016\/j.ymeth.2016.09.002\n\n.","journal-title":"Methods"},{"issue":"12","key":"3159_CR3","doi-asserted-by":"publisher","first-page":"1881","DOI":"10.1093\/bioinformatics\/btv048","volume":"31","author":"M Masseroli","year":"2015","unstructured":"Masseroli M, et al.Genometric query language: a novel approach to large-scale genomic data management. Bioinformatics. 2015; 31(12):1881\u20138. \nhttps:\/\/doi.org\/10.1093\/bioinformatics\/btv048\n\n. \nhttps:\/\/oup\/backfile\/content_public\/journal\/bioinformatics\/31\/12\/10.1093_bioinformatics_btv048\/2\/btv048.pdf\n\n.","journal-title":"Bioinformatics"},{"issue":"5","key":"3159_CR4","doi-asserted-by":"publisher","first-page":"729","DOI":"10.1093\/bioinformatics\/bty688","volume":"35","author":"Marco Masseroli","year":"2018","unstructured":"Masseroli M, et al.Processing of big heterogeneous genomic datasets for tertiary analysis of next generation sequencing data. Bioinformatics. 2018; 688. \nhttps:\/\/doi.org\/10.1093\/bioinformatics\/bty688\n\n. \nhttps:\/\/oup\/backfile\/content_public\/journal\/bioinformatics\/pap\/10.1093_bioinformatics_bty688\/3\/bty688.pdf\n\n.","journal-title":"Bioinformatics"},{"issue":"11","key":"3159_CR5","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1145\/2934664","volume":"59","author":"M Zaharia","year":"2016","unstructured":"Zaharia M, et al.Apache spark: a unified engine for big data processing. Commun ACM. 2016; 59(11):56\u201365.","journal-title":"Commun ACM"},{"issue":"6","key":"3159_CR6","doi-asserted-by":"publisher","first-page":"841","DOI":"10.1093\/bioinformatics\/btq033","volume":"26","author":"AR Quinlan","year":"2010","unstructured":"Quinlan AR, et al.Bedtools: a flexible suite of utilities for comparing genomic features. Bioinformatics. 2010; 26(6):841\u20132. \nhttps:\/\/doi.org\/10.1093\/bioinformatics\/btq033\n\n. \nhttps:\/\/oup\/backfile\/content_public\/journal\/bioinformatics\/26\/6\/10.1093_bioinformatics_btq033\/3\/btq033.pdf\n\n.","journal-title":"Bioinformatics"},{"issue":"14","key":"3159_CR7","doi-asserted-by":"publisher","first-page":"1919","DOI":"10.1093\/bioinformatics\/bts277","volume":"28","author":"S Neph","year":"2012","unstructured":"Neph S, et al.Bedops: high-performance genomic feature operations. Bioinformatics. 2012; 28(14):1919\u201320.","journal-title":"Bioinformatics"},{"issue":"24","key":"3159_CR8","doi-asserted-by":"publisher","first-page":"3423","DOI":"10.1093\/bioinformatics\/btr539","volume":"27","author":"RK Dale","year":"2011","unstructured":"Dale RK, Pedersen BS, Quinlan AR. Pybedtools: a flexible python library for manipulating genomic datasets and annotations. Bioinformatics. 2011; 27(24):3423\u20134.","journal-title":"Bioinformatics"},{"issue":"8","key":"3159_CR9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pcbi.1003118","volume":"9","author":"M Lawrence","year":"2013","unstructured":"Lawrence M, Huber W, Pag\u00e8s H, Aboyoun P, Carlson M, Gentleman R, Morgan MT, Carey VJ. Software for computing and annotating genomic ranges. PLOS Comput Biol. 2013; 9(8):1\u201310. \nhttps:\/\/doi.org\/10.1371\/journal.pcbi.1003118\n\n.","journal-title":"PLOS Comput Biol"},{"issue":"11","key":"3159_CR10","doi-asserted-by":"publisher","first-page":"1422","DOI":"10.1093\/bioinformatics\/btp163","volume":"25","author":"PJA Cock","year":"2009","unstructured":"Cock PJA, Antao T, Chang JT, Chapman BA, Cox CJ, Dalke A, Friedberg I, Hamelryck T, Kauff F, Wilczynski B, de Hoon MJL. Biopython: freely available Python tools for computational molecular biology and bioinformatics. Bioinformatics. 2009; 25(11):1422\u20133. \nhttps:\/\/doi.org\/10.1093\/bioinformatics\/btp163\n\n. \nhttp:\/\/oup.prod.sis.lan\/bioinformatics\/article-pdf\/25\/11\/1422\/944180\/btp163.pdf\n\n.","journal-title":"Bioinformatics"},{"issue":"6","key":"3159_CR11","doi-asserted-by":"publisher","first-page":"1251","DOI":"10.1109\/TCBB.2016.2576447","volume":"14","author":"S Ceri","year":"2017","unstructured":"Ceri S, et al.Data management for heterogeneous genomic datasets. IEEE\/ACM Trans Comput Biol Bioinforma. 2017; 14(6):1251\u201364. \nhttps:\/\/doi.org\/10.1109\/TCBB.2016.2576447\n\n.","journal-title":"IEEE\/ACM Trans Comput Biol Bioinforma"},{"key":"3159_CR12","doi-asserted-by":"publisher","unstructured":"Guo R, Zhao Y, Zou Q, Fang X, Peng S. Bioinformatics applications on Apache Spark. GigaScience. 2018; 7(8). \nhttps:\/\/doi.org\/10.1093\/gigascience\/giy098\n\n. \nhttp:\/\/oup.prod.sis.lan\/gigascience\/article-pdf\/7\/8\/giy098\/25597457\/giy098.pdf\n\n.","DOI":"10.1093\/gigascience\/giy098"},{"issue":"W1","key":"3159_CR13","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1093\/nar\/gkw343","volume":"44","author":"E Afgan","year":"2016","unstructured":"Afgan E, Baker D, Van den Beek M, Blankenberg D, Bouvier D, \u010cech M, Chilton J, Clements D, Coraor N, Eberhard C, et al.The galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update. Nucleic Acids Res. 2016; 44(W1):3\u201310.","journal-title":"Nucleic Acids Res"},{"issue":"19","key":"3159_CR14","doi-asserted-by":"publisher","first-page":"2520","DOI":"10.1093\/bioinformatics\/bts480","volume":"28","author":"J K\u00f6ster","year":"2012","unstructured":"K\u00f6ster J, Rahmann S. Snakemake\u2014a scalable bioinformatics workflow engine. Bioinformatics. 2012; 28(19):2520\u20132. \nhttps:\/\/doi.org\/10.1093\/bioinformatics\/bts480\n\n. \nhttp:\/\/oup.prod.sis.lan\/bioinformatics\/article-pdf\/28\/19\/2520\/819790\/bts480.pdf\n\n.","journal-title":"Bioinformatics"},{"issue":"4","key":"3159_CR15","doi-asserted-by":"publisher","first-page":"316","DOI":"10.1038\/nbt.3820","volume":"35","author":"P Di Tommaso","year":"2017","unstructured":"Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017; 35(4):316.","journal-title":"Nat Biotechnol"},{"key":"3159_CR16","doi-asserted-by":"publisher","unstructured":"Birger C, Hanna M, Salinas E, Neff J, Saksena G, Livitz D, Rosebrock D, Stewart C, Leshchiner I, Baumann A, Voet D, Cibulskis K, Banks E, Philippakis A, Getz G. Firecloud, a scalable cloud-based platform for collaborative genome analysis: Strategies for reducing and controlling costs. bioRxiv. 2017. \nhttps:\/\/doi.org\/10.1101\/209494\n\n. \nhttps:\/\/www.biorxiv.org\/content\/early\/2017\/11\/03\/209494.full.pdf\n\n.","DOI":"10.1101\/209494"},{"issue":"9","key":"3159_CR17","doi-asserted-by":"publisher","first-page":"1297","DOI":"10.1101\/gr.107524.110","volume":"20","author":"A McKenna","year":"2010","unstructured":"McKenna A, Hanna M, Banks E, Sivachenko A, Cibulskis K, Kernytsky A, Garimella K, Altshuler D, Gabriel S, Daly M, DePristo MA. The genome analysis toolkit: A mapreduce framework for analyzing next-generation dna sequencing data. Genome Res. 2010; 20(9):1297\u2013303. \nhttps:\/\/doi.org\/10.1101\/gr.107524.110\n\n. \nhttp:\/\/genome.cshlp.org\/content\/20\/9\/1297.full.pdf+html\n\n.","journal-title":"Genome Res"},{"key":"3159_CR18","doi-asserted-by":"publisher","unstructured":"Nothaft FA, Massie M, Danford T, Zhang Z, Laserson U, Yeksigian C, Kottalam J, Ahuja A, Hammerbacher J, Linderman M, Franklin M, Joseph AD, Patterson DA. Rethinking data-intensive science using scalable analytics systems. In: Proceedings of the 2015 International Conference on Management of Data (SIGMOD \u201915). ACM: 2015. \nhttps:\/\/doi.org\/10.1145\/2723372.2742787\n\n.","DOI":"10.1145\/2723372.2742787"},{"issue":"10","key":"3159_CR19","doi-asserted-by":"publisher","first-page":"1113","DOI":"10.1038\/ng.2764","volume":"45","author":"JN Weinstein","year":"2013","unstructured":"Weinstein JN, Collisson EA, Mills GB, Shaw KRM, Ozenberger BA, Ellrott K, Shmulevich I, Sander C, Stuart JM, Network CGAR, et al.The cancer genome atlas pan-cancer analysis project. Nat Genet. 2013; 45(10):1113.","journal-title":"Nat Genet"},{"issue":"4","key":"3159_CR20","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1182\/blood-2017-03-735654","volume":"130","author":"MA Jensen","year":"2017","unstructured":"Jensen MA, Ferretti V, Grossman RL, Staudt LM. The nci genomic data commons as an engine for precision medicine. Blood. 2017; 130(4):453\u20139.","journal-title":"Blood"},{"issue":"7414","key":"3159_CR21","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1038\/nature11247","volume":"489","author":"EP Consortium","year":"2012","unstructured":"Consortium EP, et al.An integrated encyclopedia of dna elements in the human genome. Nature. 2012; 489(7414):57.","journal-title":"Nature"},{"issue":"7539","key":"3159_CR22","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1038\/nature14248","volume":"518","author":"A Kundaje","year":"2015","unstructured":"Kundaje A, Meuleman W, Ernst J, Bilenky M, Yen A, Heravi-Moussavi A, Kheradpour P, Zhang Z, Wang J, Ziller MJ, et al.Integrative analysis of 111 reference human epigenomes. Nature. 2015; 518(7539):317.","journal-title":"Nature"},{"issue":"D1","key":"3159_CR23","doi-asserted-by":"publisher","first-page":"766","DOI":"10.1093\/nar\/gky955","volume":"47","author":"A Frankish","year":"2018","unstructured":"Frankish A, Diekhans M, Ferreira A-M, Johnson R, Jungreis I, Loveland J, Mudge JM, Sisu C, Wright J, Armstrong J, et al.Gencode reference annotation for the human and mouse genomes. Nucleic Acids Res. 2018; 47(D1):766\u201373.","journal-title":"Nucleic Acids Res"},{"issue":"D1","key":"3159_CR24","doi-asserted-by":"publisher","first-page":"733","DOI":"10.1093\/nar\/gkv1189","volume":"44","author":"NA O\u2019Leary","year":"2015","unstructured":"O\u2019Leary NA, Wright MW, Brister JR, Ciufo S, Haddad D, McVeigh R, Rajput B, Robbertse B, Smith-White B, Ako-Adjei D, et al.Reference sequence (refseq) database at ncbi: current status, taxonomic expansion, and functional annotation. Nucleic Acids Res. 2015; 44(D1):733\u201345.","journal-title":"Nucleic Acids Res"},{"issue":"D1","key":"3159_CR25","doi-asserted-by":"publisher","first-page":"D658","DOI":"10.1093\/nar\/gkw983","volume":"45","author":"Shenglin Mei","year":"2016","unstructured":"Mei S, Qin Q, Wu Q, Sun H, Zheng R, Zang C, Zhu M, Wu J, Shi X, Taing L, et al.Cistrome data browser: a data portal for chip-seq and chromatin accessibility data in human and mouse. Nucleic Acids Res. 2016; 983. \nhttps:\/\/doi.org\/10.1093\/nar\/gkw983\n\n.","journal-title":"Nucleic Acids Research"},{"issue":"0","key":"3159_CR26","doi-asserted-by":"publisher","first-page":"bar026","DOI":"10.1093\/database\/bar026","volume":"2011","author":"J. Zhang","year":"2011","unstructured":"Zhang J, Baran J, Cros A, Guberman JM, Haider S, Hsu J, Liang Y, Rivkin E, Wang J, Whitty B, et al.International cancer genome consortium data portal\u2014a one-stop shop for cancer genomics data. Database. 2011; 2011. \nhttps:\/\/doi.org\/10.1093\/database\/bar026\n\n.","journal-title":"Database"},{"key":"3159_CR27","doi-asserted-by":"publisher","DOI":"10.1145\/3214708.3214710","volume-title":"Proceedings of the 5th International Workshop on Exploratory Search in Databases and the Web. ExploreDB 2018","author":"L Nanni","year":"2018","unstructured":"Nanni L, Pinoli P, Canakoglu A, Ceri S. Exploring genomic datasets: From batch to interactive and back. In: Proceedings of the 5th International Workshop on Exploratory Search in Databases and the Web. ExploreDB 2018. New York: ACM: 2018. p. 3\u2013136. \nhttps:\/\/doi.org\/10.1145\/3214708.3214710\n\n. \nhttp:\/\/doi.acm.org\/10.1145\/3214708.3214710\n\n."},{"key":"3159_CR28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.is.2018.10.002","volume":"81","author":"P Pinoli","year":"2019","unstructured":"Pinoli P, Ceri S, Martinenghi D, Nanni L. Metadata management for scientific databases. Inf Syst. 2019; 81:1\u201320.","journal-title":"Inf Syst"},{"issue":"7539","key":"3159_CR29","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1038\/nature13835","volume":"518","author":"Kyle Kai-How Farh","year":"2014","unstructured":"Farh KK-H, Marson A, Zhu J, Kleinewietfeld M, Housley WJ, Beik S, Shoresh N, Whitton H, Ryan RJH, Shishkin AA, Hatan M, Carrasco-Alfonso MJ, Mayer D, Luckey CJ, Patsopoulos NA, De Jager PL, Kuchroo VK, Epstein CB, Daly MJ, Hafler DA, Bernstein BE. Genetic and epigenetic fine mapping of causal autoimmune disease variants. Nature. 2015. \nhttps:\/\/doi.org\/10.1038\/nature13835\n\n.","journal-title":"Nature"},{"key":"3159_CR30","doi-asserted-by":"publisher","DOI":"10.1145\/2523616.2523633","volume-title":"Proceedings of the 4th Annual Symposium on Cloud Computing. SOCC \u201913","author":"VK Vavilapalli","year":"2013","unstructured":"Vavilapalli VK, Murthy AC, Douglas C, Agarwal S, Konar M, Evans R, Graves T, Lowe J, Shah H, Seth S, Saha B, Curino C, O\u2019Malley O, Radia S, Reed B, Baldeschwieler E. Apache hadoop yarn: Yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing. SOCC \u201913. New York: ACM: 2013. p. 5\u20131516. \nhttps:\/\/doi.org\/10.1145\/2523616.2523633\n\n. \nhttp:\/\/doi.acm.org\/10.1145\/2523616.2523633\n\n."},{"issue":"5","key":"3159_CR31","doi-asserted-by":"publisher","first-page":"342","DOI":"10.1016\/j.gpb.2018.05.004","volume":"16","author":"S Perna","year":"2018","unstructured":"Perna S, Pinoli P, Ceri S, Wong L. Tica: Transcriptional interaction and coregulation analyzer. Genom Proteomics Bioinforma. 2018; 16(5):342\u201353.","journal-title":"Genom Proteomics Bioinforma"},{"key":"3159_CR32","volume-title":"Data Mining for Systems Biology","author":"S Perna","year":"2018","unstructured":"Perna S, Canakoglu A, Pinoli P, Ceri S, Wong L. Implementing a transcription factor interaction prediction system using the genometric query language. In: Data Mining for Systems Biology. New York: Springer: 2018. p. 63\u201381."}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-019-3159-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s12859-019-3159-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-019-3159-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,7]],"date-time":"2020-11-07T00:09:49Z","timestamp":1604707789000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-019-3159-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,8]]},"references-count":32,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,12]]}},"alternative-id":["3159"],"URL":"https:\/\/doi.org\/10.1186\/s12859-019-3159-9","relation":{},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11,8]]},"assertion":[{"value":"13 June 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 October 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 November 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"560"}}