{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,21]],"date-time":"2026-05-21T16:41:45Z","timestamp":1779381705218,"version":"3.53.1"},"reference-count":37,"publisher":"Oxford University Press (OUP)","issue":"21","license":[{"start":{"date-parts":[[2022,9,12]],"date-time":"2022-09-12T00:00:00Z","timestamp":1662940800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"MRC SHIELD consortium","award":["MR\/N02995X\/1"],"award-info":[{"award-number":["MR\/N02995X\/1"]}]},{"name":"Edinburgh Global Research Scholarship from the University of Edinburgh"},{"name":"Institute Strategic funding provided to the Roslin Institute by the BBSRC","award":["BBS\/E\/D\/10002070"],"award-info":[{"award-number":["BBS\/E\/D\/10002070"]}]},{"name":"Institute Strategic funding provided to the Roslin Institute by the BBSRC","award":["BBS\/E\/D\/10002071"],"award-info":[{"award-number":["BBS\/E\/D\/10002071"]}]},{"name":"Wellcome Trust Senior Research Fellowship","award":["223164\/Z\/21\/Z"],"award-info":[{"award-number":["223164\/Z\/21\/Z"]}]},{"name":"Wellcome Trust Senior Research Fellowship","award":["MC_PC_20004"],"award-info":[{"award-number":["MC_PC_20004"]}]},{"name":"Wellcome Trust Senior Research Fellowship","award":["MC_PC_19025"],"award-info":[{"award-number":["MC_PC_19025"]}]},{"name":"Wellcome Trust Senior Research Fellowship","award":["MC_PC_1905"],"award-info":[{"award-number":["MC_PC_1905"]}]},{"name":"Wellcome Trust Senior Research Fellowship","award":["MRNO2995X\/1"],"award-info":[{"award-number":["MRNO2995X\/1"]}]},{"name":"Wellcome Trust Senior Research Fellowship","award":["MC_PC_20029"],"award-info":[{"award-number":["MC_PC_20029"]}]},{"name":"Sepsis Research (Fiona Elizabeth Agnew Trust), a BBSRC Institute Strategic Programme"},{"name":"Roslin Institute","award":["BB\/P013732\/1"],"award-info":[{"award-number":["BB\/P013732\/1"]}]},{"name":"Roslin Institute","award":["BB\/P013759\/1"],"award-info":[{"award-number":["BB\/P013759\/1"]}]},{"name":"UK Intensive Care Society"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,10,31]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>A common experimental output in biomedical science is a list of genes implicated in a given biological process or disease. The gene lists resulting from a group of studies answering the same, or similar, questions can be combined by ranking aggregation methods to find a consensus or a more reliable answer. Evaluating a ranking aggregation method on a specific type of data before using it is required to support the reliability since the property of a dataset can influence the performance of an algorithm. Such evaluation on gene lists is usually based on a simulated database because of the lack of a known truth for real data. However, simulated datasets tend to be too small compared to experimental data and neglect key features, including heterogeneity of quality, relevance and the inclusion of unranked lists.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>In this study, a group of existing methods and their variations that are suitable for meta-analysis of gene lists are compared using simulated and real data. Simulated data were used to explore the performance of the aggregation methods as a function of emulating the common scenarios of real genomic data, with various heterogeneity of quality, noise level and a mix of unranked and ranked data using 20\u00a0000 possible entities. In addition to the evaluation with simulated data, a comparison using real genomic data on the SARS-CoV-2 virus, cancer (non-small cell lung cancer) and bacteria (macrophage apoptosis) was performed. We summarize the results of our evaluation in a simple flowchart to select a ranking aggregation method, and in an automated implementation using the meta-analysis by information content algorithm to infer heterogeneity of data quality across input datasets.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>The code for simulated data generation and running edited version of algorithms: https:\/\/github.com\/baillielab\/comparison_of_RA_methods. Code to perform an optimal selection of methods based on the results of this review, using the MAIC algorithm to infer the characteristics of an input dataset, can be downloaded here: https:\/\/github.com\/baillielab\/maic. An online service for running MAIC: https:\/\/baillielab.net\/maic.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Supplementary information<\/jats:title>\n                    <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btac621","type":"journal-article","created":{"date-parts":[[2022,9,12]],"date-time":"2022-09-12T11:53:32Z","timestamp":1662983612000},"page":"4927-4933","source":"Crossref","is-referenced-by-count":15,"title":["Systematic comparison of ranking aggregation methods for gene lists in experimental results"],"prefix":"10.1093","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1580-797X","authenticated-orcid":false,"given":"Bo","family":"Wang","sequence":"first","affiliation":[{"name":"Roslin Institute, University of Edinburgh , Edinburgh EH25 9RG, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andy","family":"Law","sequence":"additional","affiliation":[{"name":"Roslin Institute, University of Edinburgh , Edinburgh EH25 9RG, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0979-7875","authenticated-orcid":false,"given":"Tim","family":"Regan","sequence":"additional","affiliation":[{"name":"Roslin Institute, University of Edinburgh , Edinburgh EH25 9RG, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicholas","family":"Parkinson","sequence":"additional","affiliation":[{"name":"Roslin Institute, University of Edinburgh , Edinburgh EH25 9RG, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joby","family":"Cole","sequence":"additional","affiliation":[{"name":"University of Sheffield , Sheffield S10 2NT, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9873-8243","authenticated-orcid":false,"given":"Clark D","family":"Russell","sequence":"additional","affiliation":[{"name":"Centre for Inflammation Research, The Queen\u2019s Medical Research Institute, University of Edinburgh , Edinburgh EH16 4TJ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David H","family":"Dockrell","sequence":"additional","affiliation":[{"name":"Centre for Inflammation Research, The Queen\u2019s Medical Research Institute, University of Edinburgh , Edinburgh EH16 4TJ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5329-9910","authenticated-orcid":false,"given":"Michael U","family":"Gutmann","sequence":"additional","affiliation":[{"name":"School of Informatics, University of Edinburgh , Edinburgh EH8 9AB, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"J Kenneth","family":"Baillie","sequence":"additional","affiliation":[{"name":"Roslin Institute, University of Edinburgh , Edinburgh EH25 9RG, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,9,12]]},"reference":[{"key":"2022103112473829300_btac621-B1","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1002\/eji.200939856","article-title":"Expression of apoptosis-related genes in an Ethiopian cohort study correlates with tuberculosis clinical status","volume":"40","author":"Abebe","year":"2010","journal-title":"Eur. J. Immunol"},{"key":"2022103112473829300_btac621-B2","doi-asserted-by":"crossref","first-page":"284","DOI":"10.1007\/s00453-008-9211-1","article-title":"Aggregation of partial rankings, p-ratings and top-m lists","volume":"57","author":"Ailon","year":"2010","journal-title":"Algorithmica"},{"key":"2022103112473829300_btac621-B3","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1038\/nature12787","article-title":"An atlas of active enhancers across human cell types and tissues","volume":"507","author":"Andersson","year":"2014","journal-title":"Nature"},{"key":"2022103112473829300_btac621-B4","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1093\/bioinformatics\/btu518","article-title":"Hybrid Bayesian-rank integration approach improves the predictive power of genomic dataset aggregation","volume":"31","author":"Badgeley","year":"2015","journal-title":"Bioinformatics"},{"key":"2022103112473829300_btac621-B5","doi-asserted-by":"crossref","first-page":"1949","DOI":"10.1016\/S0002-9440(10)63553-5","article-title":"Non-small-cell lung cancer molecular signatures recapitulate lung developmental pathways","volume":"163","author":"Borczuk","year":"2003","journal-title":"Am. J. Pathol"},{"key":"2022103112473829300_btac621-B6","doi-asserted-by":"crossref","first-page":"1202","DOI":"10.14778\/2809974.2809982","article-title":"Rank aggregation with ties: experiments and analysis","volume":"8","author":"Brancotte","year":"2015","journal-title":"Proc. VLDB Endow"},{"key":"2022103112473829300_btac621-B7","doi-asserted-by":"crossref","first-page":"862","DOI":"10.1093\/bioinformatics\/btt068","article-title":"A powerful Bayesian meta-analysis method to integrate multiple gene set enrichment studies","volume":"29","author":"Chen","year":"2013","journal-title":"Bioinformatics"},{"key":"2022103112473829300_btac621-B8","first-page":"657","article-title":"M\u00e9moire sur les \u00e9lections au scrutin, 1781","author":"de Borda","year":"1781"},{"key":"2022103112473829300_btac621-B9","doi-asserted-by":"crossref","first-page":"1023","DOI":"10.1080\/01621459.2013.878660","article-title":"Bayesian aggregation of order-based rank data","volume":"109","author":"Deng","year":"2014","journal-title":"J. Am. Stat. Assoc"},{"key":"2022103112473829300_btac621-B10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13567-020-00828-3","article-title":"Salmonella enterica serovar Typhimurium inhibits the innate immune response and promotes apoptosis in a ribosomal\/TRP53-dependent manner in swine neutrophils","volume":"51","author":"Huang","year":"2020","journal-title":"Vet. Res"},{"key":"2022103112473829300_btac621-B11","first-page":"D498","article-title":"The reactome pathway knowledgebase","volume":"48","author":"Jassal","year":"2020","journal-title":"Nucleic Acids Res"},{"key":"2022103112473829300_btac621-B12","doi-asserted-by":"crossref","first-page":"D590","DOI":"10.1093\/nar\/gky962","article-title":"New approach for understanding genome variations in KEGG","volume":"47","author":"Kanehisa","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"2022103112473829300_btac621-B13","doi-asserted-by":"crossref","first-page":"251","DOI":"10.3389\/fonc.2014.00251","article-title":"Hidden treasures in \u201cancient\u201d microarrays: gene-expression portrays biology and potential resistance pathways of major lung cancer subtypes and normal tissue","volume":"4","author":"Kerkentzes","year":"2014","journal-title":"Front. Oncol"},{"key":"2022103112473829300_btac621-B14","doi-asserted-by":"crossref","first-page":"e55596","DOI":"10.1371\/journal.pone.0055596","article-title":"A high-dimensional, deep-sequencing study of lung adenocarcinoma in female never-smokers","volume":"8","author":"Kim","year":"2013","journal-title":"PLoS One"},{"key":"2022103112473829300_btac621-B15","doi-asserted-by":"crossref","first-page":"573","DOI":"10.1093\/bioinformatics\/btr709","article-title":"Robust rank aggregation for gene list integration and meta-analysis","volume":"28","author":"Kolde","year":"2012","journal-title":"Bioinformatics"},{"key":"2022103112473829300_btac621-B16","doi-asserted-by":"crossref","first-page":"731","DOI":"10.1016\/j.cell.2010.02.012","article-title":"Genome-wide analysis of the host intracellular network that regulates survival of mycobacterium tuberculosis","volume":"140","author":"Kumar","year":"2010","journal-title":"Cell"},{"key":"2022103112473829300_btac621-B17","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1016\/j.cels.2020.08.010","article-title":"Illuminating host-mycobacterial interactions with genome-wide CRISPR knockout and CRISPRI screens","volume":"11","author":"Lai","year":"2020","journal-title":"Cell Syst"},{"key":"2022103112473829300_btac621-B18","first-page":"1","article-title":"Genome-wide CRISPR screen identifies host dependency factors for influenza a virus infection","volume":"11","author":"Li","year":"2020","journal-title":"Nat. Commun"},{"key":"2022103112473829300_btac621-B19","doi-asserted-by":"crossref","first-page":"4266","DOI":"10.1002\/sim.7920","article-title":"A Bayesian latent variable approach to aggregation of partial and top-ranked lists in genomic studies","volume":"37","author":"Li","year":"2018","journal-title":"Stat. Med"},{"key":"2022103112473829300_btac621-B20","doi-asserted-by":"crossref","first-page":"178","DOI":"10.1093\/bib\/bbx101","article-title":"A comparative study of rank aggregation methods for partial and top ranked lists in genomic applications","volume":"20","author":"Li","year":"2019","journal-title":"Brief. Bioinformatics"},{"key":"2022103112473829300_btac621-B21","doi-asserted-by":"crossref","first-page":"8993","DOI":"10.1007\/s13277-015-3576-y","article-title":"RNA-Seq analysis of lung adenocarcinomas reveals different gene expression profiles between smoking and nonsmoking patients","volume":"36","author":"Li","year":"2015","journal-title":"Tumour Biol"},{"key":"2022103112473829300_btac621-B22","doi-asserted-by":"crossref","first-page":"555","DOI":"10.1002\/wics.111","article-title":"Rank aggregation methods","volume":"2","author":"Lin","year":"2010","journal-title":"Wiley Interdiscip. Rev. Comput. Stat"},{"key":"2022103112473829300_btac621-B23","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1111\/j.1541-0420.2008.01044.x","article-title":"Integration of ranked lists via cross entropy Monte Carlo with applications to mRNA and microRNA studies","volume":"65","author":"Lin","year":"2009","journal-title":"Biometrics"},{"key":"2022103112473829300_btac621-B24","first-page":"481","author":"Liu","year":"2007"},{"key":"2022103112473829300_btac621-B25","doi-asserted-by":"crossref","first-page":"2177","DOI":"10.1084\/jem.20060766","article-title":"NF-\u03baB translocation prevents host cell death after low-dose challenge by Legionella pneumophila","volume":"203","author":"Losick","year":"2006","journal-title":"J. Exp. Med"},{"key":"2022103112473829300_btac621-B26","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1186\/1297-9716-43-25","article-title":"Pan-genomic analysis of bovine monocyte-derived macrophage gene expression in response to in vitro infection with Mycobacterium avium subspecies paratuberculosis","volume":"43","author":"MacHugh","year":"2012","journal-title":"Vet. Res"},{"key":"2022103112473829300_btac621-B27","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1038\/gene.2010.51","article-title":"Human gene expression profiles of susceptibility and resistance in tuberculosis","volume":"12","author":"Maertzdorf","year":"2011","journal-title":"Genes Immun"},{"key":"2022103112473829300_btac621-B28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-020-79033-3","article-title":"Dynamic data-driven meta-analysis for prioritisation of host genes implicated in covid-19","volume":"10","author":"Parkinson","year":"2020","journal-title":"Sci. Rep"},{"key":"2022103112473829300_btac621-B29","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1164\/rccm.201804-0646OC","article-title":"Alveolar macrophage apoptosis-associated bacterial killing helps prevent murine pneumonia","volume":"200","author":"Preston","year":"2019","journal-title":"Am. J. Respir. Crit. Care Med"},{"key":"2022103112473829300_btac621-B30","doi-asserted-by":"crossref","first-page":"D661","DOI":"10.1093\/nar\/gkx1064","article-title":"WikiPathways: a multifaceted pathway database bridging metabolomics to other omics research","volume":"46","author":"Slenter","year":"2018","journal-title":"Nucleic Acids Res"},{"key":"2022103112473829300_btac621-B31","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1126\/science.1087447","article-title":"A gene-coexpression network for global discovery of conserved genetic modules","volume":"302","author":"Stuart","year":"2003","journal-title":"Science"},{"key":"2022103112473829300_btac621-B32","article-title":"The string database in 2017: quality-controlled protein\u2013protein association networks, made broadly accessible","author":"Szklarczyk","year":"2016","journal-title":"Nucleic Acids Research"},{"key":"2022103112473829300_btac621-B33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1852102.1852106","article-title":"A similarity measure for indefinite rankings","volume":"28","author":"Webber","year":"2010","journal-title":"ACM Trans. Inf. Syst"},{"key":"2022103112473829300_btac621-B34","first-page":"811","author":"Wu","year":"2003"},{"key":"2022103112473829300_btac621-B35","doi-asserted-by":"crossref","first-page":"e02169","DOI":"10.1128\/mBio.02169-19","article-title":"A genome-wide knockout screen in human macrophages identified host factors modulating Salmonella infection","volume":"10","author":"Yeung","year":"2019","journal-title":"MBio"},{"key":"2022103112473829300_btac621-B36","volume-title":"arXiv preprint arXiv:1607.06051.","author":"Yi","year":"2016"},{"key":"2022103112473829300_btac621-B37","doi-asserted-by":"crossref","first-page":"731","DOI":"10.1038\/onc.2016.242","article-title":"microRNAs with AAGUGC seed motif constitute an integral part of an oncogenic signaling network","volume":"36","author":"Zhou","year":"2017","journal-title":"Oncogene"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btac621\/46205348\/btac621.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/38\/21\/4927\/46697906\/btac621.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/38\/21\/4927\/46697906\/btac621.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T08:48:47Z","timestamp":1667206127000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/38\/21\/4927\/6696211"}},"subtitle":[],"editor":[{"given":"Zhiyong","family":"Lu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2022,9,12]]},"references-count":37,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2022,9,12]]},"published-print":{"date-parts":[[2022,10,31]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btac621","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2022.01.09.475491","asserted-by":"object"}]},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,11,1]]},"published":{"date-parts":[[2022,9,12]]}}}