{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,3]],"date-time":"2024-08-03T18:58:26Z","timestamp":1722711506322},"reference-count":51,"publisher":"Oxford University Press (OUP)","issue":"7","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2016,4,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Motivation: Gene networks have become a central tool in the analysis of genomic data but are widely regarded as hard to interpret. This has motivated a great deal of comparative evaluation and research into best practices. We explore the possibility that this may lead to overfitting in the field as a whole.<\/jats:p>\n               <jats:p>Results: We construct a model of \u2018research communities\u2019 sampling from real gene network data and machine learning methods to characterize performance trends. Our analysis reveals an important principle limiting the value of replication, namely that targeting it directly causes \u2018easy\u2019 or uninformative replication to dominate analyses. We find that when sampling across network data and algorithms with similar variability, the relationship between replicability and accuracy is positive (Spearman\u2019s correlation, rs \u223c0.33) but where no such constraint is imposed, the relationship becomes negative for a given gene function (rs \u223c \u22120.13). We predict factors driving replicability in some prior analyses of gene networks and show that they are unconnected with the correctness of the original result, instead reflecting replicable biases. Without these biases, the original results also vanish replicably. We show these effects can occur quite far upstream in network data and that there is a strong tendency within protein\u2013protein interaction data for highly replicable interactions to be associated with poor quality control.<\/jats:p>\n               <jats:p>Availability and implementation: Algorithms, network data and a guide to the code available at: https:\/\/github.com\/wimverleyen\/AggregateGeneFunctionPrediction.<\/jats:p>\n               <jats:p>Contact: \u00a0jgillis@cshl.edu<\/jats:p>\n               <jats:p>Supplementary information: \u00a0Supplementary data are available at Bioinformatics online.<\/jats:p>","DOI":"10.1093\/bioinformatics\/btv734","type":"journal-article","created":{"date-parts":[[2015,12,15]],"date-time":"2015-12-15T02:54:41Z","timestamp":1450148081000},"page":"1065-1073","source":"Crossref","is-referenced-by-count":10,"title":["Positive and negative forms of replicability in gene network analysis"],"prefix":"10.1093","volume":"32","author":[{"given":"W.","family":"Verleyen","sequence":"first","affiliation":[{"name":"Stanley Institute for Cognitive Genomics, Cold Spring Harbor Laboratory, 500 Sunnyside Boulevard Woodbury, NY 11797, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"S.","family":"Ballouz","sequence":"additional","affiliation":[{"name":"Stanley Institute for Cognitive Genomics, Cold Spring Harbor Laboratory, 500 Sunnyside Boulevard Woodbury, NY 11797, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"J.","family":"Gillis","sequence":"additional","affiliation":[{"name":"Stanley Institute for Cognitive Genomics, Cold Spring Harbor Laboratory, 500 Sunnyside Boulevard Woodbury, NY 11797, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2015,12,14]]},"reference":[{"key":"2023020112010698100_btv734-B1","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1016\/S0022-2836(05)80360-2","article-title":"Basic local alignment search tool","volume":"215","author":"Altschul","year":"1990","journal-title":"J. Mol. Biol"},{"key":"2023020112010698100_btv734-B2","doi-asserted-by":"crossref","first-page":"1039","DOI":"10.1038\/nbt.2017","article-title":"Comprehensive assay of kinase catalytic activity reveals features of kinase inhibitor selectivity","volume":"29","author":"Anastassiadis","year":"2011","journal-title":"Nat. Biotechnol"},{"key":"2023020112010698100_btv734-B3","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1038\/75556","article-title":"Gene ontology: tool for the unification of biology. The Gene Ontology Consortium","volume":"25","author":"Ashburner","year":"2000","journal-title":"Nat. Genet"},{"key":"2023020112010698100_btv734-B4","doi-asserted-by":"crossref","first-page":"2123","DOI":"10.1093\/bioinformatics\/btv118","article-title":"Guidance for RNA-seq co-expression network construction and analysis: safety in numbers","volume":"31","author":"Ballouz","year":"2015","journal-title":"Bioinformatics"},{"key":"2023020112010698100_btv734-B5","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1038\/483531a","article-title":"Drug development: raise standards for preclinical cancer research","volume":"483","author":"Begley","year":"2012","journal-title":"Nature"},{"key":"2023020112010698100_btv734-B6","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1038\/nature09204","article-title":"Network organization of the human autophagy system","volume":"466","author":"Behrends","year":"2010","journal-title":"Nature"},{"key":"2023020112010698100_btv734-B7","doi-asserted-by":"crossref","first-page":"398","DOI":"10.1126\/science.187.4175.398","article-title":"Sex bias in graduate admissions: data from Berkeley","volume":"187","author":"Bickel","year":"1975","journal-title":"Science"},{"key":"2023020112010698100_btv734-B8","doi-asserted-by":"crossref","first-page":"3081","DOI":"10.1093\/bioinformatics\/bts581","article-title":"An unbiased evaluation of gene prioritization tools","volume":"28","author":"Bornigen","year":"2012","journal-title":"Bioinformatics"},{"key":"2023020112010698100_btv734-B9","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/BF00058655","article-title":"Bagging predictors","volume":"24","author":"Breiman","year":"1996","journal-title":"J. Mach. Learn. Res"},{"key":"2023020112010698100_btv734-B10","article-title":"Unequal evolutionary conservation of human protein interactions in interologous networks","volume":"8, R95","author":"Brown","year":"2007","journal-title":"Genome Biol"},{"key":"2023020112010698100_btv734-B11","doi-asserted-by":"crossref","first-page":"D816","DOI":"10.1093\/nar\/gks1158","article-title":"The BioGRID interaction database: 2013 update","volume":"41","author":"Chatr-aryamontri","year":"2013","journal-title":"Nucleic Acids Res"},{"key":"2023020112010698100_btv734-B12","first-page":"1","article-title":"Statistical comparisons of classifiers over multiple data sets","volume":"7","author":"Demsar","year":"2006","journal-title":"J. Mach. Learn. Res"},{"key":"2023020112010698100_btv734-B13","doi-asserted-by":"crossref","first-page":"1895","DOI":"10.1162\/089976698300017197","article-title":"Approximate statistical tests for comparing supervised classification learning algorithms","volume":"10","author":"Dietterich","year":"1998","journal-title":"Neural Comput"},{"key":"2023020112010698100_btv734-B14","doi-asserted-by":"crossref","first-page":"D222","DOI":"10.1093\/nar\/gkt1223","article-title":"Pfam: the protein families database","volume":"42","author":"Finn","year":"2014","journal-title":"Nucleic Acids Res"},{"key":"2023020112010698100_btv734-B15","volume-title":"The Design of Experiments","author":"Fisher","year":"1935"},{"key":"2023020112010698100_btv734-B16","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1016\/j.cell.2008.10.016","article-title":"Autism: many genes, common pathways?","volume":"135","author":"Geschwind","year":"2008","journal-title":"Cell"},{"key":"2023020112010698100_btv734-B17","doi-asserted-by":"crossref","first-page":"e17258","DOI":"10.1371\/journal.pone.0017258","article-title":"The impact of multifunctional genes on \u201cguilt by association\u201d analysis","volume":"6","author":"Gillis","year":"2011","journal-title":"PLoS One"},{"key":"2023020112010698100_btv734-B18","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1093\/bioinformatics\/bts727","article-title":"Assessing identity, redundancy and confounds in Gene Ontology annotations over time","volume":"29","author":"Gillis","year":"2013","journal-title":"Bioinformatics"},{"key":"2023020112010698100_btv734-B19","volume-title":"Problems of Monetary Management: The UK Experience","author":"Goodhart","year":"1975"},{"key":"2023020112010698100_btv734-B20","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1111\/j.1749-6632.2011.06383.x","article-title":"Accurate evaluation and analysis of functional genomics data and methods","volume":"1260","author":"Greene","year":"2012","journal-title":"Ann. N. Y. Acad. Sci"},{"key":"2023020112010698100_btv734-B21","doi-asserted-by":"crossref","first-page":"518","DOI":"10.1016\/j.cell.2013.06.049","article-title":"Spatial and temporal mapping of de novo mutations in schizophrenia to a fetal prefrontal cortical network","volume":"154","author":"Gulsuner","year":"2013","journal-title":"Cell"},{"key":"2023020112010698100_btv734-B22","first-page":"359","volume-title":"Proceedings of the Seventeenth International Conference on Machine Learning","author":"Hall","year":"2000"},{"key":"2023020112010698100_btv734-B23","doi-asserted-by":"crossref","first-page":"D306","DOI":"10.1093\/nar\/gkr948","article-title":"InterPro in 2011: new developments in the family and domain prediction database","volume":"40","author":"Hunter","year":"2012","journal-title":"Nucleic Acids Res"},{"key":"2023020112010698100_btv734-B24","doi-asserted-by":"crossref","first-page":"218","DOI":"10.1001\/jama.294.2.218","article-title":"Contradicted and initially stronger effects in highly cited clinical research","volume":"294","author":"Ioannidis","year":"2005","journal-title":"JAMA"},{"key":"2023020112010698100_btv734-B25","doi-asserted-by":"crossref","first-page":"D428","DOI":"10.1093\/nar\/gki072","article-title":"Reactome: a knowledgebase of biological pathways","volume":"33","author":"Joshi-Tope","year":"2005","journal-title":"Nucleic Acids Res"},{"key":"2023020112010698100_btv734-B26","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1002\/prot.24448","article-title":"CASP10 results compared to those of previous CASP experiments","volume":"82 (suppl.)","author":"Kryshtafovych","year":"2014","journal-title":"Proteins"},{"key":"2023020112010698100_btv734-B27","doi-asserted-by":"crossref","first-page":"796","DOI":"10.1038\/nmeth.2016","article-title":"Wisdom of crowds for robust gene network inference","volume":"9","author":"Marbach","year":"2012","journal-title":"Nat. Methods"},{"key":"2023020112010698100_btv734-B28","doi-asserted-by":"crossref","first-page":"910","DOI":"10.1126\/science.1065103","article-title":"Specificity and stability in topology of protein networks","volume":"296","author":"Maslov","year":"2002","journal-title":"Science"},{"key":"2023020112010698100_btv734-B29","doi-asserted-by":"crossref","first-page":"730","DOI":"10.1038\/nmeth.2557","article-title":"The CRAPome: a contaminant repository for affinity purification-mass spectrometry data","volume":"10","author":"Mellacheruvu","year":"2013","journal-title":"Nat. Methods"},{"key":"2023020112010698100_btv734-B30","doi-asserted-by":"crossref","first-page":"1106","DOI":"10.1177\/0956797614524255","article-title":"Psychological strategies for winning a geopolitical forecasting tournament","volume":"25","author":"Mellers","year":"2014","journal-title":"Psychol. Sci"},{"key":"2023020112010698100_btv734-B31","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1186\/1471-2105-9-327","article-title":"Gene Ontology term overlap as a measure of gene functional similarity","volume":"9","author":"Mistry","year":"2008","journal-title":"BMC Bioinformatics"},{"key":"2023020112010698100_btv734-B32","doi-asserted-by":"crossref","first-page":"523","DOI":"10.1038\/nrg3253","article-title":"Computational tools for prioritizing candidate genes: boosting disease gene discovery","volume":"13","author":"Moreau","year":"2012","journal-title":"Nat. Rev. Genet"},{"key":"2023020112010698100_btv734-B33","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1038\/nature10989","article-title":"Sporadic autism exomes reveal a highly interconnected protein network of de novo mutations","volume":"485","author":"O'Roak","year":"2012","journal-title":"Nature"},{"key":"2023020112010698100_btv734-B34","doi-asserted-by":"crossref","first-page":"e38937","DOI":"10.1371\/journal.pone.0038937","article-title":"Improving disease gene prioritization by comparing the semantic similarity of phenotypes in mice with those of human diseases","volume":"7","author":"Oellrich","year":"2012","journal-title":"PLoS One"},{"key":"2023020112010698100_btv734-B35","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1093\/nar\/27.1.29","article-title":"KEGG: Kyoto Encyclopedia of Genes and Genomes","volume":"27","author":"Ogata","year":"1999","journal-title":"Nucleic Acids Res"},{"key":"2023020112010698100_btv734-B36","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1038\/35001165","article-title":"Guilt-by-association goes global","volume":"403","author":"Oliver","year":"2000","journal-title":"Nature"},{"key":"2023020112010698100_btv734-B37","doi-asserted-by":"crossref","first-page":"D358","DOI":"10.1093\/nar\/gkt1115","article-title":"The MIntAct project-IntAct as a common curation platform for 11 molecular interaction databases","volume":"42","author":"Orchard","year":"2014","journal-title":"Nucleic Acids Res"},{"key":"2023020112010698100_btv734-B38","doi-asserted-by":"crossref","first-page":"1008","DOI":"10.1016\/j.cell.2013.10.031","article-title":"Integrative functional genomic analyses implicate specific molecular pathways and circuits in autism","volume":"155","author":"Parikshak","year":"2013","journal-title":"Cell"},{"key":"2023020112010698100_btv734-B39","doi-asserted-by":"crossref","first-page":"230","DOI":"10.12688\/f1000research.2-230.v1","article-title":"Progress and challenges in the computational prediction of gene function using networks: 2012-2013 update","volume":"2","author":"Pavlidis","year":"2013","journal-title":"F1000Res"},{"key":"2023020112010698100_btv734-B40","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn. Res"},{"key":"2023020112010698100_btv734-B41","doi-asserted-by":"crossref","first-page":"S2","DOI":"10.1186\/gb-2008-9-s1-s2","article-title":"A critical assessment of Mus musculus gene function prediction using integrated genomic evidence","volume":"9 (suppl.)","author":"Pena-Castillo","year":"2008","journal-title":"Genome Biol"},{"key":"2023020112010698100_btv734-B42","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1111\/j.1559-1816.2004.tb02547.x","article-title":"The persuasiveness of source credibility: a critical review of five decades' evidence","volume":"34","author":"Pornpitakpan","year":"2004","journal-title":"J. Appl. Soc. Psychol"},{"key":"2023020112010698100_btv734-B43","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1186\/1471-2164-14-129","article-title":"Neurocarta: aggregating and sharing disease-gene relations for the neurosciences","volume":"14","author":"Portales-Casamar","year":"2013","journal-title":"BMC Genomics"},{"key":"2023020112010698100_btv734-B44","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1038\/nmeth.2340","article-title":"A large-scale evaluation of computational protein function prediction","volume":"10","author":"Radivojac","year":"2013","journal-title":"Nat. Methods"},{"key":"2023020112010698100_btv734-B45","doi-asserted-by":"crossref","first-page":"e31826","DOI":"10.1371\/journal.pone.0031826","article-title":"HIPPIE: integrating protein interaction networks with experiment based quality scores","volume":"7","author":"Schaefer","year":"2012","journal-title":"PLoS One"},{"key":"2023020112010698100_btv734-B46","doi-asserted-by":"crossref","first-page":"13585","DOI":"10.1073\/pnas.1317516111","article-title":"Science communication as political communication","volume":"111 (suppl.)","author":"Scheufele","year":"2014","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023020112010698100_btv734-B47","doi-asserted-by":"crossref","first-page":"D535","DOI":"10.1093\/nar\/gkj109","article-title":"BioGRID: a general repository for interaction datasets","volume":"34","author":"Stark","year":"2006","journal-title":"Nucleic Acids Res"},{"key":"2023020112010698100_btv734-B48","doi-asserted-by":"crossref","first-page":"745","DOI":"10.1093\/bioinformatics\/btu715","article-title":"Measuring the wisdom of the crowds in network-based gene function inference","volume":"31","author":"Verleyen","year":"2015","journal-title":"Bioinformatics"},{"key":"2023020112010698100_btv734-B49","doi-asserted-by":"crossref","first-page":"2277","DOI":"10.1016\/j.jprot.2010.07.005","article-title":"It's the machine that matters: predicting gene function and phenotype from protein networks","volume":"73","author":"Wang","year":"2010","journal-title":"J. Proteomics"},{"key":"2023020112010698100_btv734-B50","doi-asserted-by":"crossref","first-page":"468","DOI":"10.1006\/geno.1997.4979","article-title":"Guidelines for human gene nomenclature (1997). HUGO Nomenclature Committee","volume":"45","author":"White","year":"1997","journal-title":"Genomics"},{"key":"2023020112010698100_btv734-B51","doi-asserted-by":"crossref","first-page":"W115","DOI":"10.1093\/nar\/gkt533","article-title":"GeneMANIA prediction server 2013 update","volume":"41","author":"Zuberi","year":"2013","journal-title":"Nucleic Acids Res"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/32\/7\/1065\/49018551\/bioinformatics_32_7_1065.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/32\/7\/1065\/49018551\/bioinformatics_32_7_1065.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,1]],"date-time":"2023-02-01T22:25:44Z","timestamp":1675290344000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/32\/7\/1065\/1744280"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,12,14]]},"references-count":51,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2016,4,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btv734","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2016,4,1]]},"published":{"date-parts":[[2015,12,14]]}}}