{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,5]],"date-time":"2026-05-05T10:25:56Z","timestamp":1777976756054,"version":"3.51.4"},"reference-count":48,"publisher":"Oxford University Press (OUP)","issue":"23","license":[{"start":{"date-parts":[[2021,7,8]],"date-time":"2021-07-08T00:00:00Z","timestamp":1625702400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"UNIMI Partneriat H2020","award":["PSR2015-1720GVALE_01"],"award-info":[{"award-number":["PSR2015-1720GVALE_01"]}]},{"name":"Machine Learning and Big Data Analysis for Bioinformatics","award":["PSR2019_DIP_010_GVALE"],"award-info":[{"award-number":["PSR2019_DIP_010_GVALE"]}]},{"name":"University of Milano"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,12,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>Automated protein function prediction is a complex multi-class, multi-label, structured classification problem in which protein functions are organized in a controlled vocabulary, according to the Gene Ontology (GO). \u2018Hierarchy-unaware\u2019 classifiers, also known as \u2018flat\u2019 methods, predict GO terms without exploiting the inherent structure of the ontology, potentially violating the True-Path-Rule (TPR) that governs the GO, while \u2018hierarchy-aware\u2019 approaches, even if they obey the TPR, do not always show clear improvements with respect to flat methods, or do not scale well when applied to the full GO.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>To overcome these limitations, we propose Hierarchical Ensemble Methods for Directed Acyclic Graphs (HEMDAG), a family of highly modular hierarchical ensembles of classifiers, able to build upon any flat method and to provide \u2018TPR-safe\u2019 predictions, by leveraging a combination of isotonic regression and TPR learning strategies. Extensive experiments on synthetic and real data across several organisms firstly show that HEMDAG can be used as a general tool to improve the predictions of flat classifiers, and secondly that HEMDAG is competitive versus state-of-the-art hierarchy-aware learning methods proposed in the last CAFA international challenges.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Availability and implementation<\/jats:title>\n                  <jats:p>Fully tested R code freely available at https:\/\/anaconda.org\/bioconda\/r-hemdag. Tutorial and documentation at https:\/\/hemdag.readthedocs.io.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Supplementary information<\/jats:title>\n                  <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btab485","type":"journal-article","created":{"date-parts":[[2021,7,4]],"date-time":"2021-07-04T19:06:29Z","timestamp":1625425589000},"page":"4526-4533","source":"Crossref","is-referenced-by-count":7,"title":["HEMDAG: a family of modular and scalable hierarchical ensemble methods to improve Gene Ontology term prediction"],"prefix":"10.1093","volume":"37","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4309-2200","authenticated-orcid":false,"given":"Marco","family":"Notaro","sequence":"first","affiliation":[{"name":"AnacletoLab \u2013 Dipartimento di Informatica, Universit\u00e0 degli Studi di Milano , Milano 20133, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marco","family":"Frasca","sequence":"additional","affiliation":[{"name":"AnacletoLab \u2013 Dipartimento di Informatica, Universit\u00e0 degli Studi di Milano , Milano 20133, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alessandro","family":"Petrini","sequence":"additional","affiliation":[{"name":"AnacletoLab \u2013 Dipartimento di Informatica, Universit\u00e0 degli Studi di Milano , Milano 20133, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jessica","family":"Gliozzo","sequence":"additional","affiliation":[{"name":"AnacletoLab \u2013 Dipartimento di Informatica, Universit\u00e0 degli Studi di Milano , Milano 20133, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2024-7572","authenticated-orcid":false,"given":"Elena","family":"Casiraghi","sequence":"additional","affiliation":[{"name":"AnacletoLab \u2013 Dipartimento di Informatica, Universit\u00e0 degli Studi di Milano , Milano 20133, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0736-9199","authenticated-orcid":false,"given":"Peter N","family":"Robinson","sequence":"additional","affiliation":[{"name":"The Jackson Laboratory for Genomic Medicine , Farmington, CT 06032, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5694-3919","authenticated-orcid":false,"given":"Giorgio","family":"Valentini","sequence":"additional","affiliation":[{"name":"AnacletoLab \u2013 Dipartimento di Informatica, Universit\u00e0 degli Studi di Milano , Milano 20133, Italy"},{"name":"CINI, National Laboratory in Artificial Intelligence and Intelligent Systems\u2014AIIS , Roma 00185, Italy"},{"name":"Data Science Research Center, Universit\u00e0 degli Studi di Milano , Milano 20133, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2021,7,8]]},"reference":[{"key":"2023061310493338700_btab485-B1","doi-asserted-by":"crossref","first-page":"641","DOI":"10.1214\/aoms\/1177728423","article-title":"An empirical distribution function for sampling with incomplete information","volume":"26","author":"Ayer","year":"1955","journal-title":"Ann. Math. Statist"},{"key":"2023061310493338700_btab485-B2","doi-asserted-by":"crossref","first-page":"285","DOI":"10.3233\/FI-2015-1213","article-title":"Modelling progressive filtering","volume":"138","author":"Armano","year":"2015","journal-title":"Fundam. Inform"},{"key":"2023061310493338700_btab485-B3","volume-title":"Statistical Inference under Order Restrictions: The Theory and Application of Isotonic Regression","author":"Barlow","year":"1972"},{"key":"2023061310493338700_btab485-B4","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1080\/01621459.1972.10481216","article-title":"The isotonic regression problem and its dual","volume":"67","author":"Barlow","year":"1972","journal-title":"J. Am. Stat. Assoc"},{"key":"2023061310493338700_btab485-B5","first-page":"281","article-title":"Random search for hyper-parameter optimization","volume":"12","author":"Bergstra","year":"2012","journal-title":"J. Mach. Learn. Res"},{"key":"2023061310493338700_btab485-B6","first-page":"25","volume-title":"An O(n2) Algorithm for Isotonic Regression","author":"Burdakov","year":"2006"},{"key":"2023061310493338700_btab485-B7","first-page":"1","author":"Cerri","year":"2015"},{"key":"2023061310493338700_btab485-B8","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1186\/s12859-016-1232-1","article-title":"Reduction strategies for hierarchical multi-label classification in protein function prediction","volume":"17","author":"Cerri","year":"2016","journal-title":"BMC Bioinform"},{"key":"2023061310493338700_btab485-B9","doi-asserted-by":"crossref","first-page":"584","DOI":"10.1016\/j.asoc.2019.01.017","article-title":"Inducing hierarchical multi-label classification rules with genetic algorithms","volume":"77","author":"Cerri","year":"2019","journal-title":"Appl. Soft Comput"},{"key":"2023061310493338700_btab485-B10","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1007\/s10994-011-5271-6","article-title":"Synergy of multi-label hierarchical ensembles, data fusion, and cost-sensitive methods for gene functional inference","volume":"88","author":"Cesa-Bianchi","year":"2012","journal-title":"Mach. Learn"},{"key":"2023061310493338700_btab485-B11","doi-asserted-by":"crossref","first-page":"1550","DOI":"10.1109\/TCBB.2017.2684127","article-title":"Multitask protein function prediction through task dissimilarity","volume":"16","author":"Frasca","year":"2016","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform"},{"key":"2023061310493338700_btab485-B12","volume-title":"ECML PKDD 2019, Volume 11907 of Lecture Notes in Computer Science","author":"Frasca","year":"2020"},{"key":"2023061310493338700_btab485-B5865602","first-page":"D330","article-title":"The Gene Ontology Resource: 20 years and still GOing strong","volume":"47","year":"2018","journal-title":"Nucleic Acid Research"},{"key":"2023061310493338700_btab485-B14","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1007\/BF01449044","article-title":"Projections onto order simplexes","volume":"12","author":"Grotzinger","year":"1984","journal-title":"Appl. Math. Optim"},{"key":"2023061310493338700_btab485-B15","doi-asserted-by":"crossref","first-page":"S3","DOI":"10.1186\/gb-2008-9-s1-s3","article-title":"Predicting gene function in a hierarchical context with an ensemble of classifiers","volume":"9","author":"Guan","year":"2008","journal-title":"Genome Biol"},{"key":"2023061310493338700_btab485-B16","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1186\/s13059-016-1037-6","article-title":"An expanded evaluation of protein function prediction methods shows an improvement in accuracy","volume":"17","author":"Jiang","year":"2016","journal-title":"Genome Biol"},{"key":"2023061310493338700_btab485-B17","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1186\/gb-2009-10-2-206","article-title":"Sequence-based feature prediction and annotation of proteins","volume":"10","author":"Juncker","year":"2009","journal-title":"Genome Biol"},{"key":"2023061310493338700_btab485-B18","doi-asserted-by":"crossref","first-page":"259","DOI":"10.12688\/f1000research.6670.1","article-title":"PHENOstruct: prediction of human phenotype ontology terms using heterogeneous data sources","volume":"4","author":"Kahanda","year":"2015","journal-title":"F1000Res"},{"key":"2023061310493338700_btab485-B19","doi-asserted-by":"crossref","first-page":"1343","DOI":"10.1038\/bjc.2016.113","article-title":"Ccr6 expression in colon cancer is associated with advanced disease and supports epithelial-to-mesenchymal transition","volume":"114","author":"Kapur","year":"2016","journal-title":"Br. J. Cancer"},{"key":"2023061310493338700_btab485-B20","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1016\/j.patcog.2012.09.023","article-title":"Tree ensembles for predicting structured outputs","volume":"46","author":"Kocev","year":"2013","journal-title":"Pattern Recognit"},{"key":"2023061310493338700_btab485-B21","doi-asserted-by":"crossref","first-page":"D1018","DOI":"10.1093\/nar\/gky1105","article-title":"Expansion of the Human Phenotype Ontology (HPO) knowledge base and resources","volume":"47","author":"K\u00f6hler","year":"2019","journal-title":"NAR"},{"key":"2023061310493338700_btab485-B22","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1093\/bioinformatics\/btz595","article-title":"DeepGOPlus: improved protein function prediction from sequence","volume":"36","author":"Kulmanov","year":"2019","journal-title":"Bioinformatics"},{"key":"2023061310493338700_btab485-B23","doi-asserted-by":"crossref","first-page":"660","DOI":"10.1093\/bioinformatics\/btx624","article-title":"DeepGO: predicting protein functions from sequence and interactions using a deep ontology-aware classifier","volume":"34","author":"Kulmanov","year":"2018","journal-title":"Bioinformatics"},{"key":"2023061310493338700_btab485-B24","article-title":"Semantic similarity and machine learning with ontologies","author":"Kulmanov","year":"2020","journal-title":"Brief. Bioinform"},{"key":"2023061310493338700_btab485-B25","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1007\/s10994-009-5111-0","article-title":"Structured prediction by joint kernel support estimation","volume":"77","author":"Lampert","year":"2009","journal-title":"Mach. Learn"},{"key":"2023061310493338700_btab485-B26","doi-asserted-by":"crossref","first-page":"4180","DOI":"10.1093\/bioinformatics\/btaa284","article-title":"HPOLabeler: improving prediction of human protein\u2013phenotype associations by learning to rank","volume":"36","author":"Liu","year":"2020","journal-title":"Bioinformatics"},{"key":"2023061310493338700_btab485-B27","doi-asserted-by":"crossref","first-page":"1316","DOI":"10.1287\/opre.33.6.1316","article-title":"Establishing consistent and realistic reorder intervals in production-distribution systems","volume":"33","author":"Maxwell","year":"1985","journal-title":"Oper. Res"},{"key":"2023061310493338700_btab485-B28","doi-asserted-by":"crossref","first-page":"1496","DOI":"10.1007\/s10618-020-00704-w","article-title":"Active learning for hierarchical multi-label classification","volume":"34","author":"Nakano","year":"2020","journal-title":"Data Min. Knowl. Discov"},{"key":"2023061310493338700_btab485-B29","doi-asserted-by":"crossref","first-page":"449:1","DOI":"10.1186\/s12859-017-1854-y","article-title":"Prediction of human phenotype ontology terms by means of hierarchical ensemble methods","volume":"18","author":"Notaro","year":"2017","journal-title":"BMC Bioinform"},{"key":"2023061310493338700_btab485-B30","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1007\/978-3-030-14160-8_8","volume-title":"Computational Intelligence Methods for Bioinformatics and Biostatistics","author":"Notaro","year":"2019"},{"key":"2023061310493338700_btab485-B31","doi-asserted-by":"crossref","first-page":"S6","DOI":"10.1186\/gb-2008-9-s1-s6","article-title":"Consistent probabilistic output for protein function prediction","volume":"9","author":"Obozinski","year":"2008","journal-title":"Genome Biol"},{"key":"2023061310493338700_btab485-B32","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1038\/nmeth.2340","article-title":"A large-scale evaluation of computational protein function prediction","volume":"10","author":"Radivojac","year":"2013","journal-title":"Nat. Methods"},{"key":"2023061310493338700_btab485-B33","doi-asserted-by":"crossref","first-page":"1812","DOI":"10.1109\/TCBB.2012.114","article-title":"A fast ranking algorithm for predicting gene functions in biomolecular networks","volume":"9","author":"Re","year":"2012","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform"},{"key":"2023061310493338700_btab485-B34","doi-asserted-by":"crossref","first-page":"5539","DOI":"10.1093\/nar\/gkh894","article-title":"The FunCat, a functional annotation scheme for systematic classification of proteins from whole genomes","volume":"32","author":"Ruepp","year":"2004","journal-title":"NAR"},{"key":"2023061310493338700_btab485-B35","doi-asserted-by":"crossref","first-page":"D955","DOI":"10.1093\/nar\/gky1032","article-title":"Human Disease Ontology 2018 update: classification, content and workflow expansion","volume":"47","author":"Schriml","year":"2019","journal-title":"NAR"},{"key":"2023061310493338700_btab485-B36","first-page":"407","article-title":"Network-based prediction of protein function","volume":"8","author":"Sharan","year":"2007","journal-title":"Mol. Sys. Biol"},{"key":"2023061310493338700_btab485-B37","doi-asserted-by":"crossref","first-page":"830","DOI":"10.1007\/s10618-010-0175-9","article-title":"A survey of hierarchical classification across different application domains","volume":"22","author":"Silla","year":"2011","journal-title":"Data Min. Knowl. Discov"},{"key":"2023061310493338700_btab485-B38","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1142\/S0219720010004744","article-title":"Hierarchical classification of gene ontology terms using the gostruct method","volume":"8","author":"Sokolov","year":"2010","journal-title":"J. Bioinf. Comp. Biol"},{"key":"2023061310493338700_btab485-B39","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1093\/nar\/gku1003","article-title":"STRING v10: protein-protein interaction networks, integrated over the tree of life","volume":"43","author":"Szklarczyk","year":"2015","journal-title":"NAR"},{"key":"2023061310493338700_btab485-B40","doi-asserted-by":"crossref","first-page":"D607","DOI":"10.1093\/nar\/gky1131","article-title":"STRING v11: protein\u2013protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets","volume":"47","author":"Szklarczyk","year":"2019","journal-title":"NAR"},{"key":"2023061310493338700_btab485-B41","doi-asserted-by":"crossref","first-page":"W84","DOI":"10.1093\/nar\/gky350","article-title":"Pannzer2: a rapid functional annotation web server","volume":"46","author":"T\u00f6r\u00f6nen","year":"2018","journal-title":"NAR"},{"key":"2023061310493338700_btab485-B42","doi-asserted-by":"crossref","first-page":"832","DOI":"10.1109\/TCBB.2010.38","article-title":"True path rule hierarchical ensembles for genome-wide gene function prediction","volume":"8","author":"Valentini","year":"2011","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform"},{"key":"2023061310493338700_btab485-B43","first-page":"901419","article-title":"Hierarchical ensemble methods for protein function prediction","volume":"34","author":"Valentini","year":"2014","journal-title":"ISRN Bioinform"},{"key":"2023061310493338700_btab485-B44","doi-asserted-by":"crossref","first-page":"1577","DOI":"10.12688\/f1000research.16415.1","article-title":"Large-scale protein function prediction using heterogeneous ensembles","volume":"7","author":"Wang","year":"2018","journal-title":"F1000Res"},{"key":"2023061310493338700_btab485-B45","first-page":"207","author":"Widmer","year":"2012"},{"key":"2023061310493338700_btab485-B46","doi-asserted-by":"crossref","first-page":"891","DOI":"10.1109\/TCBB.2014.2323058","article-title":"Genome-wide protein function prediction through multi-instance multi-label learning","volume":"11","author":"Wu","year":"2014","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform"},{"key":"2023061310493338700_btab485-B47","doi-asserted-by":"crossref","first-page":"1:1","DOI":"10.1186\/s12859-014-0430-y","article-title":"Predicting protein functions using incomplete hierarchical labels","volume":"16","author":"Yu","year":"2015","journal-title":"BMC Bioinform"},{"key":"2023061310493338700_btab485-B48","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1186\/s13059-019-1835-8","article-title":"The CAFA challenge reports improved protein function prediction and new functional annotations for hundreds of genes through experimental screens","volume":"20","author":"Zhou","year":"2019","journal-title":"Genome Biol"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btab485\/39510865\/btab485.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/37\/23\/4526\/50579686\/btab485.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/37\/23\/4526\/50579686\/btab485.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,13]],"date-time":"2023-06-13T10:51:49Z","timestamp":1686653509000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/37\/23\/4526\/6317663"}},"subtitle":[],"editor":[{"given":"Jonathan","family":"Wren","sequence":"additional","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2021,7,8]]},"references-count":48,"journal-issue":{"issue":"23","published-print":{"date-parts":[[2021,12,7]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btab485","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,12,1]]},"published":{"date-parts":[[2021,7,8]]}}}