{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T23:19:02Z","timestamp":1768087142940,"version":"3.49.0"},"reference-count":39,"publisher":"Oxford University Press (OUP)","issue":"5","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2015,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Motivation: Network-based gene function inference methods have proliferated in recent years, but measurable progress remains elusive. We wished to better explore performance trends by controlling data and algorithm implementation, with a particular focus on the performance of aggregate predictions.<\/jats:p>\n               <jats:p>Results: Hypothesizing that popular methods would perform well without hand-tuning, we used well-characterized algorithms to produce verifiably \u2018untweaked\u2019 results. We find that most state-of-the-art machine learning methods obtain \u2018gold standard\u2019 performance as measured in critical assessments in defined tasks. Across a broad range of tests, we see close alignment in algorithm performances after controlling for the underlying data being used. We find that algorithm aggregation provides only modest benefits, with a 17% increase in area under the ROC (AUROC) above the mean AUROC. In contrast, data aggregation gains are enormous with an 88% improvement in mean AUROC. Altogether, we find substantial evidence to support the view that additional algorithm development has little to offer for gene function prediction.<\/jats:p>\n               <jats:p>Availability and implementation: The supplementary information contains a description of the algorithms, the network data parsed from different biological data resources and a guide to the source code (available at: http:\/\/gillislab.cshl.edu\/supplements\/).<\/jats:p>\n               <jats:p>Contact: \u00a0jgillis@cshl.edu<\/jats:p>","DOI":"10.1093\/bioinformatics\/btu715","type":"journal-article","created":{"date-parts":[[2014,10,31]],"date-time":"2014-10-31T00:44:56Z","timestamp":1414716296000},"page":"745-752","source":"Crossref","is-referenced-by-count":20,"title":["Measuring the wisdom of the crowds in network-based gene function inference"],"prefix":"10.1093","volume":"31","author":[{"given":"W.","family":"Verleyen","sequence":"first","affiliation":[{"name":"Stanley Institute for Cognitive Genomics, Cold Spring Harbor Laboratory, Woodbury, NY 11797, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"S.","family":"Ballouz","sequence":"additional","affiliation":[{"name":"Stanley Institute for Cognitive Genomics, Cold Spring Harbor Laboratory, Woodbury, NY 11797, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"J.","family":"Gillis","sequence":"additional","affiliation":[{"name":"Stanley Institute for Cognitive Genomics, Cold Spring Harbor Laboratory, Woodbury, NY 11797, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2014,10,26]]},"reference":[{"key":"2023020116165558300_btu715-B1","doi-asserted-by":"crossref","first-page":"3389","DOI":"10.1093\/nar\/25.17.3389","article-title":"Gapped BLAST and PSI-BLAST: a new generation of protein database search programs","volume":"25","author":"Altschul","year":"1997","journal-title":"Nucleic Acids Res."},{"key":"2023020116165558300_btu715-B2","first-page":"499","article-title":"Stability and generalization","volume":"2","author":"Bousquet","year":"2002","journal-title":"J. Mach. Learn. Res."},{"key":"2023020116165558300_btu715-B3","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"2023020116165558300_btu715-B4","doi-asserted-by":"crossref","first-page":"2076","DOI":"10.1093\/bioinformatics\/bti273","article-title":"Online predicted human interaction database","volume":"21","author":"Brown","year":"2005","journal-title":"Bioinformatics"},{"key":"2023020116165558300_btu715-B5","doi-asserted-by":"crossref","first-page":"D816","DOI":"10.1093\/nar\/gks1158","article-title":"The BioGRID interaction database: 2013 update","volume":"41","author":"Chatr-Aryamontri","year":"2013","journal-title":"Nucleic Acids Res."},{"key":"2023020116165558300_btu715-B6","doi-asserted-by":"crossref","first-page":"D700","DOI":"10.1093\/nar\/gkr1029","article-title":"Saccharomyces Genome Database: the genomics resource of budding yeast","volume":"40","author":"Cherry","year":"2012","journal-title":"Nucleic Acids Res."},{"issue":"Suppl. 3","key":"2023020116165558300_btu715-B7","doi-asserted-by":"crossref","first-page":"S1","DOI":"10.1186\/1471-2105-14-S3-S1","article-title":"Protein function prediction by massive integration of evolutionary analyses and multiple data sources","volume":"14","author":"Cozzetto","year":"2013","journal-title":"BMC Bioinformatics"},{"key":"2023020116165558300_btu715-B8","first-page":"551","article-title":"Online passive-aggressive algorithms","volume":"7","author":"Crammer","year":"2006","journal-title":"J. Mach. Learn. Res."},{"key":"2023020116165558300_btu715-B9","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1145\/2347736.2347755","article-title":"A few useful things to know about machine learning","volume":"55","author":"Domingos","year":"2012","journal-title":"Commun. ACM"},{"key":"2023020116165558300_btu715-B10","doi-asserted-by":"crossref","first-page":"823","DOI":"10.1038\/35015694","article-title":"Protein function in the post-genomic era","volume":"405","author":"Eisenberg","year":"2000","journal-title":"Nature"},{"key":"2023020116165558300_btu715-B11","doi-asserted-by":"crossref","first-page":"1575","DOI":"10.1093\/nar\/30.7.1575","article-title":"An efficient algorithm for large-scale detection of protein families","volume":"30","author":"Enright","year":"2002","journal-title":"Nucleic Acids Res."},{"key":"2023020116165558300_btu715-B12","first-page":"1871","article-title":"LIBLINEAR: a library for large linear classification","volume":"9","author":"Fan","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"2023020116165558300_btu715-B13","doi-asserted-by":"crossref","first-page":"D247","DOI":"10.1093\/nar\/gkj149","article-title":"Pfam: clans, web tools and services","volume":"34","author":"Finn","year":"2006","journal-title":"Nucleic Acids Res."},{"key":"2023020116165558300_btu715-B14","doi-asserted-by":"crossref","first-page":"e17258","DOI":"10.1371\/journal.pone.0017258","article-title":"The impact of multifunctional genes on \u201cguilt by association\u201d analysis","volume":"6","author":"Gillis","year":"2011","journal-title":"PLoS One"},{"key":"2023020116165558300_btu715-B15","doi-asserted-by":"crossref","first-page":"1860","DOI":"10.1093\/bioinformatics\/btr288","article-title":"The role of indirect connections in gene networks in predicting function","volume":"27","author":"Gillis","year":"2011","journal-title":"Bioinformatics"},{"key":"2023020116165558300_btu715-B16","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1093\/bioinformatics\/bts727","article-title":"Assessing identity, redundancy and confounds in Gene Ontology annotations over time","volume":"29","author":"Gillis","year":"2013","journal-title":"Bioinformatics"},{"issue":"Suppl. 3","key":"2023020116165558300_btu715-B17","doi-asserted-by":"crossref","first-page":"S15","DOI":"10.1186\/1471-2105-14-S3-S15","article-title":"Characterizing the state of the art in the computational assignment of gene function: lessons from the first critical assessment of functional annotation (CAFA)","volume":"14","author":"Gillis","year":"2013","journal-title":"BMC Bioinformatics"},{"key":"2023020116165558300_btu715-B18","volume-title":"Correlation-based Feature Selection for Machine Learning","author":"Hall","year":"1998"},{"key":"2023020116165558300_btu715-B19","doi-asserted-by":"crossref","first-page":"D514","DOI":"10.1093\/nar\/gki033","article-title":"Online Mendelian Inheritance in Man (OMIM), a knowledgebase of human genes and genetic disorders","volume":"33","author":"Hamosh","year":"2005","journal-title":"Nucleic Acids Res."},{"issue":"Suppl. 3","key":"2023020116165558300_btu715-B20","doi-asserted-by":"crossref","first-page":"S7","DOI":"10.1186\/1471-2105-14-S3-S7","article-title":"Homology-based inference sets the bar high for protein function prediction","volume":"14","author":"Hamp","year":"2013","journal-title":"BMC Bioinformatics"},{"key":"2023020116165558300_btu715-B21","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1002\/prot.22172","article-title":"PFP: automated prediction of gene ontology functional annotations with confidence scores using protein sequence data","volume":"74","author":"Hawkins","year":"2009","journal-title":"Proteins"},{"key":"2023020116165558300_btu715-B22","doi-asserted-by":"crossref","first-page":"e1003545","DOI":"10.1371\/journal.pcbi.1003545","article-title":"Prediction and prioritization of rare oncogenic mutations in the cancer kinome using novel features and multiple classifiers","volume":"10","author":"ManChon","year":"2014","journal-title":"PLoS Comput. Biol."},{"key":"2023020116165558300_btu715-B23","doi-asserted-by":"crossref","first-page":"796","DOI":"10.1038\/nmeth.2016","article-title":"Wisdom of crowds for robust gene network inference","volume":"9","author":"Marbach","year":"2012","journal-title":"Nat. Methods"},{"key":"2023020116165558300_btu715-B24","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1186\/1471-2105-9-327","article-title":"Gene Ontology term overlap as a measure of gene functional similarity","volume":"9","author":"Mistry","year":"2008","journal-title":"BMC Bioinformatics"},{"issue":"Suppl. 1","key":"2023020116165558300_btu715-B25","doi-asserted-by":"crossref","first-page":"S4","DOI":"10.1186\/gb-2008-9-s1-s4","article-title":"GeneMANIA: a real-time multiple association network integration algorithm for predicting gene function","volume":"9","author":"Mostafavi","year":"2008","journal-title":"Genome Biol."},{"key":"2023020116165558300_btu715-B26","doi-asserted-by":"crossref","first-page":"D201","DOI":"10.1093\/nar\/gki106","article-title":"InterPro, progress and status in 2005","volume":"33","author":"Mulder","year":"2005","journal-title":"Nucleic Acids Res."},{"key":"2023020116165558300_btu715-B27","doi-asserted-by":"crossref","first-page":"D476","DOI":"10.1093\/nar\/gki107","article-title":"Inparanoid: a comprehensive database of eukaryotic orthologs","volume":"33","author":"O'Brien","year":"2005","journal-title":"Nucleic Acids Res."},{"key":"2023020116165558300_btu715-B28","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1093\/nar\/27.1.29","article-title":"KEGG: kyoto encyclopedia of genes and genomes","volume":"27","author":"Ogata","year":"1999","journal-title":"Nucleic Acids Res."},{"key":"2023020116165558300_btu715-B29","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1613\/jair.614","article-title":"Popular ensemble methods: an empirical study","volume":"11","author":"Opitz","year":"1999","journal-title":"J. Artif. Intell. Res."},{"key":"2023020116165558300_btu715-B30","doi-asserted-by":"crossref","first-page":"401","DOI":"10.1089\/10665270252935539","article-title":"Learning gene functional classifications from multiple data types","volume":"9","author":"Pavlidis","year":"2002","journal-title":"J. Comput. Biol."},{"key":"2023020116165558300_btu715-B31","first-page":"2825","article-title":"Scikit-learn: machine learning in python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res."},{"key":"2023020116165558300_btu715-B32","doi-asserted-by":"crossref","first-page":"S2","DOI":"10.1186\/gb-2008-9-s1-s2","article-title":"A critical assessment of Mus musculus gene function prediction using integrated genomic evidence","volume":"9","author":"Pena-Castillo","year":"2008","journal-title":"Genome Biol."},{"key":"2023020116165558300_btu715-B33","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1038\/nmeth.2340","article-title":"A large-scale evaluation of computational protein function prediction","volume":"10","author":"Radivojac","year":"2013","journal-title":"Nat. Methods"},{"key":"2023020116165558300_btu715-B34","doi-asserted-by":"crossref","first-page":"18485","DOI":"10.1073\/pnas.0509455102","article-title":"A mouse atlas of gene expression: large-scale digital gene-expression profiles from precisely defined developing C57BL\/6J mouse tissues and cells","volume":"102","author":"Siddiqui","year":"2005","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"2023020116165558300_btu715-B35","doi-asserted-by":"crossref","first-page":"R7","DOI":"10.1186\/gb-2004-6-1-r7","article-title":"The Mammalian Phenotype Ontology as a tool for annotating, analyzing and comparing phenotypic information","volume":"6","author":"Smith","year":"2005","journal-title":"Genome Biol."},{"key":"2023020116165558300_btu715-B36","doi-asserted-by":"crossref","first-page":"6062","DOI":"10.1073\/pnas.0400782101","article-title":"A gene atlas of the mouse and human protein-encoding transcriptomes","volume":"101","author":"Su","year":"2004","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"2023020116165558300_btu715-B37","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1093\/bib\/bbq007","article-title":"A guide to web tools to prioritize candidate genes","volume":"12","author":"Tranchevent","year":"2011","journal-title":"Brief. Bioinform."},{"key":"2023020116165558300_btu715-B38","doi-asserted-by":"crossref","first-page":"2277","DOI":"10.1016\/j.jprot.2010.07.005","article-title":"It's the machine that matters: predicting gene function and phenotype from protein networks","volume":"73","author":"Wang","year":"2010","journal-title":"J. Proteomics"},{"key":"2023020116165558300_btu715-B39","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1186\/jbiol16","article-title":"The functional landscape of mouse gene expression","volume":"3","author":"Zhang","year":"2004","journal-title":"J. Biol."}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/31\/5\/745\/49011391\/bioinformatics_31_5_745.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/31\/5\/745\/49011391\/bioinformatics_31_5_745.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,2]],"date-time":"2023-02-02T00:30:05Z","timestamp":1675297805000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/31\/5\/745\/317877"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,10,26]]},"references-count":39,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2015,3,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btu715","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2015,3,1]]},"published":{"date-parts":[[2014,10,26]]}}}