{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T19:13:42Z","timestamp":1781118822862,"version":"3.54.1"},"reference-count":79,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2020,2,6]],"date-time":"2020-02-06T00:00:00Z","timestamp":1580947200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"German Research Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,1,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>Although gene set enrichment analysis has become an integral part of high-throughput gene expression data analysis, the assessment of enrichment methods remains rudimentary and ad hoc. In the absence of suitable gold standards, evaluations are commonly restricted to selected datasets and biological reasoning on the relevance of resulting enriched gene sets.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We develop an extensible framework for reproducible benchmarking of enrichment methods based on defined criteria for applicability, gene set prioritization and detection of relevant processes. This framework incorporates a curated compendium of 75 expression datasets investigating 42 human diseases. The compendium features microarray and RNA-seq measurements, and each dataset is associated with a precompiled GO\/KEGG relevance ranking for the corresponding disease under investigation. We perform a comprehensive assessment of 10 major enrichment methods, identifying significant differences in runtime and applicability to RNA-seq data, fraction of enriched gene sets depending on the null hypothesis tested and recovery of the predefined relevance rankings. We make practical recommendations on how methods originally developed for microarray data can efficiently be applied to RNA-seq data, how to interpret results depending on the type of gene set test conducted and which methods are best suited to effectively prioritize gene sets with high phenotype relevance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability<\/jats:title>\n                    <jats:p>http:\/\/bioconductor.org\/packages\/GSEABenchmarkeR<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Contact<\/jats:title>\n                    <jats:p>ludwig.geistlinger@sph.cuny.edu<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bib\/bbz158","type":"journal-article","created":{"date-parts":[[2019,11,11]],"date-time":"2019-11-11T15:11:50Z","timestamp":1573485110000},"page":"545-556","source":"Crossref","is-referenced-by-count":124,"title":["Toward a gold standard for benchmarking gene set enrichment analysis"],"prefix":"10.1093","volume":"22","author":[{"given":"Ludwig","family":"Geistlinger","sequence":"first","affiliation":[{"name":"Graduate School of Public Health and Health Policy, City University of New York, New York, NY 10027, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gergely","family":"Csaba","sequence":"additional","affiliation":[{"name":"Institute for Implementation Science and Population Health, City University of New York, New York, NY 10027, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mara","family":"Santarelli","sequence":"additional","affiliation":[{"name":"Institute for Bioinformatics, Ludwig-Maximilians-Universit\u00e4t M\u00fcnchen, 80333 Munich, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marcel","family":"Ramos","sequence":"additional","affiliation":[{"name":"Roswell Park Cancer Institute, Buffalo, NY 14203, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lucas","family":"Schiffer","sequence":"additional","affiliation":[{"name":"Graduate School of Arts and Sciences, Boston University, Boston, MA 02215, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nitesh","family":"Turaga","sequence":"additional","affiliation":[{"name":"Epigenetics and Development Division, The Walter and Eliza Hall Institute of Medical Research, Parkville, Victoria 3052, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charity","family":"Law","sequence":"additional","affiliation":[{"name":"Department of Medical Biology, The University of Melbourne, Parkville, Victoria 3010, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sean","family":"Davis","sequence":"additional","affiliation":[{"name":"Center for Cancer Research, National Cancer Institute, Bethesda, MD 20892, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vincent","family":"Carey","sequence":"additional","affiliation":[{"name":"Harvard Medical School, Boston, MA 02215, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martin","family":"Morgan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ralf","family":"Zimmer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Levi","family":"Waldron","sequence":"additional","affiliation":[{"name":"Graduate School of Public Health and Health Policy, City University of New York, New York, NY 10027, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2020,3,9]]},"reference":[{"key":"2021012203310373000_ref1","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1186\/1741-7007-9-34","article-title":"Microarrays, deep sequencing and the true measure of the transcriptome","volume":"9","author":"Malone","year":"2011","journal-title":"BMC Biol"},{"key":"2021012203310373000_ref2","doi-asserted-by":"crossref","first-page":"D1049","DOI":"10.1093\/nar\/gku1179","article-title":"Gene ontology consortium: going forward","volume":"43","author":"Gene Ontology Consortium","year":"2015","journal-title":"Nucleic Acids Res"},{"key":"2021012203310373000_ref3","doi-asserted-by":"crossref","first-page":"D199","DOI":"10.1093\/nar\/gkt1076","article-title":"Data, information, knowledge and principle: back to metabolism in KEGG","volume":"42","author":"Kanehisa","year":"2014","journal-title":"Nucleic Acids Res"},{"key":"2021012203310373000_ref4","doi-asserted-by":"crossref","first-page":"D691","DOI":"10.1093\/nar\/gkq1018","article-title":"Reactome: a database of reactions, pathways and biological processes","volume":"39","author":"Croft","year":"2011","journal-title":"Nucleic Acids Res"},{"issue":"12","key":"2021012203310373000_ref5","doi-asserted-by":"crossref","first-page":"1739","DOI":"10.1093\/bioinformatics\/btr260","article-title":"Molecular signatures database (MSigDB) 3.0","volume":"27","author":"Liberzon","year":"2011","journal-title":"Bioinformatics"},{"issue":"8","key":"2021012203310373000_ref6","doi-asserted-by":"crossref","first-page":"980","DOI":"10.1093\/bioinformatics\/btm051","article-title":"Analyzing gene expression data in terms of gene sets: methodological issues","volume":"23","author":"Goeman","year":"2007","journal-title":"Bioinformatics"},{"issue":"43","key":"2021012203310373000_ref7","doi-asserted-by":"crossref","first-page":"15545","DOI":"10.1073\/pnas.0506580102","article-title":"Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles","volume":"102","author":"Subramanian","year":"2005","journal-title":"Proc Natl Acad Sci USA"},{"issue":"1","key":"2021012203310373000_ref8","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1038\/nprot.2008.211","article-title":"Systematic and integrative analysis of large gene lists using DAVID bioinformatics resources","volume":"4","author":"Huang","year":"2009","journal-title":"Nat Protoc"},{"key":"2021012203310373000_ref9","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1186\/1471-2105-14-128","article-title":"Enrichr: interactive and collaborative HTML5 gene list enrichment analysis tool","volume":"14","author":"Chen","year":"2013","journal-title":"BMC Bioinformatics"},{"issue":"5","key":"2021012203310373000_ref10","doi-asserted-by":"crossref","first-page":"284","DOI":"10.1089\/omi.2011.0118","article-title":"ClusterProfiler: an R package for comparing biological themes among gene clusters","volume":"6","author":"Yu","year":"2012","journal-title":"OMICS"},{"issue":"1","key":"2021012203310373000_ref11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/nar\/gkn923","article-title":"Bioinformatics enrichment tools: paths toward the comprehensive functional analysis of large gene lists","volume":"37","author":"Huang","year":"2009","journal-title":"Nucleic Acids Res"},{"key":"2021012203310373000_ref12","article-title":"Simultaneous enrichment analysis of all possible gene-sets: unifying self-contained and competitive methods","author":"Ebrahimpoor","year":"2019","journal-title":"Brief Bioinform"},{"issue":"2","key":"2021012203310373000_ref13","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1002375","article-title":"Ten years of pathway analysis: current approaches and outstanding challenges","volume":"8","author":"Khatri","year":"2012","journal-title":"PLoS Comput Biol"},{"issue":"13","key":"2021012203310373000_ref14","doi-asserted-by":"crossref","first-page":"i366","DOI":"10.1093\/bioinformatics\/btr228","article-title":"From sets to graphs: towards a realistic enrichment analysis of transcriptomic systems","volume":"27","author":"Geistlinger","year":"2011","journal-title":"Bioinformatics"},{"key":"2021012203310373000_ref15","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1038\/nature08460","article-title":"Systematic RNA interference reveals that oncogenic KRAS-driven cancers require TBK1","volume":"462","author":"Barbie","year":"2009","journal-title":"Nature"},{"issue":"1","key":"2021012203310373000_ref16","doi-asserted-by":"crossref","first-page":"404","DOI":"10.1186\/s12859-018-2435-4","article-title":"Single sample scoring of molecular phenotypes","volume":"19","author":"Foroutan","year":"2018","journal-title":"BMC Bioinformatics"},{"key":"2021012203310373000_ref17","doi-asserted-by":"crossref","first-page":"278","DOI":"10.3389\/fphys.2013.00278","article-title":"Methods and approaches in the topology-based analysis of biological pathways","volume":"4","author":"Mitrea","year":"2013","journal-title":"Front Physiol"},{"issue":"4","key":"2021012203310373000_ref18","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1093\/bib\/bbt002","article-title":"Gene set analysis methods: statistical models and methodological differences","volume":"15","author":"Maciejewski","year":"2014","journal-title":"Brief Bioinform"},{"issue":"2","key":"2021012203310373000_ref19","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1016\/j.gene.2013.08.063","article-title":"Assessment of gene set analysis methods based on microarray data","volume":"534","author":"Alavi-Majd","year":"2014","journal-title":"Gene"},{"key":"2021012203310373000_ref20","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1186\/1471-2105-10-275","article-title":"Comparative study of gene set enrichment methods","volume":"10","author":"Abatangelo","year":"2009","journal-title":"BMC Bioinformatics"},{"issue":"3","key":"2021012203310373000_ref21","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1093\/bib\/bbr049","article-title":"Gene set enrichment analysis: performance evaluation and usage guidelines","volume":"13","author":"Hung","year":"2012","journal-title":"Brief Bioinform"},{"key":"2021012203310373000_ref22","first-page":"1","article-title":"Evaluating gene set enrichment analysis via a hybrid data model","volume":"13","author":"Hua","year":"2014","journal-title":"Cancer Inform"},{"issue":"3","key":"2021012203310373000_ref23","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1093\/bioinformatics\/btw623","article-title":"Combining multiple tools outperforms individual methods in gene set enrichment analyses","volume":"33","author":"Alhamdoosh","year":"2017","journal-title":"Bioinformatics"},{"issue":"11","key":"2021012203310373000_ref24","doi-asserted-by":"crossref","first-page":"1480","DOI":"10.1093\/bioinformatics\/bts164","article-title":"Rigorous assessment of gene set enrichment tests","volume":"28","author":"Naeem","year":"2012","journal-title":"Bioinformatics"},{"key":"2021012203310373000_ref25","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1186\/1471-2105-13-136","article-title":"Down-weighting overlapping genes improves gene set analysis","volume":"13","author":"Tarca","year":"2012","journal-title":"BMC Bioinformatics"},{"issue":"11","key":"2021012203310373000_ref26","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0079217","article-title":"A comparison of gene set analysis methods in terms of sensitivity, prioritization and specificity","volume":"8","author":"Tarca","year":"2013","journal-title":"PLoS One"},{"key":"2021012203310373000_ref27","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1186\/s12859-015-0751-5","article-title":"Comparative study on gene set and pathway topology-based enrichment methods","volume":"16","author":"Bayerlova","year":"2015","journal-title":"BMC Bioinformatics"},{"key":"2021012203310373000_ref28","doi-asserted-by":"crossref","first-page":"18871","DOI":"10.1038\/srep18871","article-title":"Lego: a novel method for gene set over-representation analysis by incorporating network-based gene weights","volume":"6","author":"Dong","year":"2016","journal-title":"Sci Rep"},{"key":"2021012203310373000_ref29","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1186\/s12859-016-0884-1","article-title":"Bioconductor\u2019s EnrichmentBrowser: seamless navigation through combined results of set- & network-based enrichment analysis","volume":"17","author":"Geistlinger","year":"2016","journal-title":"BMC Bioinformatics"},{"issue":"1","key":"2021012203310373000_ref30","doi-asserted-by":"crossref","first-page":"256","DOI":"10.1186\/s12859-017-1674-0","article-title":"Ranking metrics in gene set enrichment analysis: do they matter?","volume":"18","author":"Zyla","year":"2017","journal-title":"BMC Bioinformatics"},{"issue":"1","key":"2021012203310373000_ref31","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1186\/s13059-019-1790-4","article-title":"Identifying significantly impacted pathways: a comprehensive review and assessment","volume":"20","author":"Nguyen","year":"2019","journal-title":"Genome Biol"},{"issue":"10","key":"2021012203310373000_ref32","doi-asserted-by":"crossref","first-page":"1113","DOI":"10.1038\/ng.2764","article-title":"The Cancer Genome Atlas Pan-Cancer analysis project","volume":"45","author":"The Cancer Genome Atlas Research Network","year":"2013","journal-title":"Nat Genet"},{"key":"2021012203310373000_ref33","doi-asserted-by":"crossref","first-page":"1.24.1","DOI":"10.1002\/0471250953.bi0124s47","article-title":"MalaCards: a comprehensive automatically-mined database of human diseases","volume":"47","author":"Rappaport","year":"2014","journal-title":"Curr Protoc Bioinformatics"},{"issue":"3","key":"2021012203310373000_ref34","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1089\/omi.2015.0168","article-title":"GeneAnalytics: an integrative gene set analysis tool for next generation sequencing, RNAseq and microarray data","volume":"20","author":"Fuchs","year":"2016","journal-title":"OMICS"},{"issue":"1","key":"2021012203310373000_ref35","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1093\/bioinformatics\/btp616","article-title":"EdgeR: a Bioconductor package for differential expression analysis of digital gene expression data","volume":"26","author":"Robinson","year":"2010","journal-title":"Bioinformatics"},{"key":"2021012203310373000_ref36","article-title":"Anscombe\u2019s 1948 variance stabilizing transformation for the negative binomial distribution is well suited to RNA-seq expression data","author":"Harrison","year":"2015","journal-title":"F1000Res"},{"issue":"3\u20134","key":"2021012203310373000_ref37","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1093\/biomet\/35.3-4.246","article-title":"The transformation of Poisson, binomial and negative-binomial data","volume":"35","author":"Anscombe","year":"1948","journal-title":"Biometrika"},{"issue":"1","key":"2021012203310373000_ref38","doi-asserted-by":"crossref","first-page":"1","DOI":"10.2202\/1544-6115.1027","article-title":"Linear models and empirical Bayes for assessing differential expression in microarray experiments","volume":"3","author":"Smyth","year":"2004","journal-title":"Stat Appl Genet Mol Biol"},{"key":"2021012203310373000_ref39","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1186\/gb-2014-15-2-r29","article-title":"Voom: precision weights unlock linear model analysis tools for RNA-seq read counts","volume":"15","author":"Law","year":"2014","journal-title":"Genome Biol"},{"key":"2021012203310373000_ref40","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1186\/s13059-014-0550-8","article-title":"Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2","volume":"15","author":"Love","year":"2014","journal-title":"Genome Biol"},{"issue":"1","key":"2021012203310373000_ref41","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","article-title":"Controlling the false discovery rate: a practical and powerful approach to multiple testing","volume":"57","author":"Benjamini","year":"1995","journal-title":"J Royal Stat Soc"},{"issue":"5439","key":"2021012203310373000_ref42","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1126\/science.286.5439.531","article-title":"Molecular classification of cancer: class discovery and class prediction by gene expression monitoring","volume":"286","author":"Golub","year":"1999","journal-title":"Science"},{"key":"2021012203310373000_ref43","article-title":"GolubEsets: exprSets for golub leukemia data","author":"Golub","year":"2019","journal-title":"R package"},{"key":"2021012203310373000_ref44","article-title":"Vsn: variance stabilization and calibration for microarray data","author":"Huber","year":"2019","journal-title":"R package"},{"key":"2021012203310373000_ref45","article-title":"R: a language and environment for statistical computing","author":"R Core Team","year":"2019"},{"issue":"2","key":"2021012203310373000_ref46","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1038\/nmeth.3252","article-title":"Orchestrating high-throughput genomic analysis with Bioconductor","volume":"12","author":"Huber","year":"2015","journal-title":"Nat Methods"},{"issue":"18","key":"2021012203310373000_ref47","doi-asserted-by":"crossref","first-page":"8452","DOI":"10.1093\/nar\/gkt631","article-title":"A comprehensive gene regulatory network for the diauxic shift in Saccharomyces cerevisiae","volume":"41","author":"Geistlinger","year":"2013","journal-title":"Nucleic Acids Res"},{"key":"2021012203310373000_ref48","doi-asserted-by":"crossref","first-page":"D991","DOI":"10.1093\/nar\/gks1193","article-title":"NCBI GEO: archive for functional genomics data sets\u2014update","volume":"41","author":"Barrett","year":"2013","journal-title":"Nucleic Acids Res"},{"key":"2021012203310373000_ref49","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1186\/1471-2105-12-323","article-title":"RSEM: accurate transcript quantification from RNA-seq data with or without a reference genome","volume":"12","author":"Li","year":"2011","journal-title":"BMC Bioinformatics"},{"issue":"3","key":"2021012203310373000_ref50","doi-asserted-by":"crossref","first-page":"562","DOI":"10.1038\/nprot.2012.016","article-title":"Differential gene and transcript expression analysis of RNA-seq experiments with TopHat and Cufflinks","volume":"7","author":"Trapnell","year":"2012","journal-title":"Nat Protoc"},{"issue":"4","key":"2021012203310373000_ref51","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1007\/s12064-012-0162-3","article-title":"Measurement of mRNA abundance using RNA-seq data: RPKM measure is inconsistent among samples","volume":"131","author":"Wagner","year":"2012","journal-title":"Theory Biosci"},{"issue":"2","key":"2021012203310373000_ref52","doi-asserted-by":"crossref","first-page":"R14","DOI":"10.1186\/gb-2010-11-2-r14","article-title":"Gene ontology analysis for RNA-seq: accounting for selection bias","volume":"11","author":"Young","year":"2010","journal-title":"Genome Biol"},{"key":"2021012203310373000_ref53","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1186\/s12859-014-0397-8","article-title":"Comparative evaluation of gene set analysis approaches for RNA-seq data","volume":"15","author":"Rahmatallah","year":"2014","journal-title":"BMC Bioinformatics"},{"key":"2021012203310373000_ref54","doi-asserted-by":"crossref","first-page":"1943","DOI":"10.1093\/bioinformatics\/bti260","article-title":"Significance analysis of functional categories in gene expression studies: a structured permutation approach","volume":"21","author":"Barry","year":"2005","journal-title":"Bioinformatics"},{"key":"2021012203310373000_ref55","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1146\/annurev.ps.46.020195.003021","article-title":"Multiple hypothesis testing","volume":"46","author":"Shaffer","year":"1995","journal-title":"Ann Rev Psychol"},{"issue":"2","key":"2021012203310373000_ref56","doi-asserted-by":"crossref","first-page":"371","DOI":"10.1016\/j.cell.2018.02.060","article-title":"Comprehensive characterization of cancer driver genes and mutations","volume":"173","author":"Bailey","year":"2018","journal-title":"Cell"},{"issue":"2","key":"2021012203310373000_ref57","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1016\/j.cell.2018.03.035","article-title":"Oncogenic signaling pathways in The Cancer Genome Atlas","volume":"173","author":"Sanchez-Vega","year":"2018","journal-title":"Cell"},{"key":"2021012203310373000_ref58","doi-asserted-by":"crossref","first-page":"574","DOI":"10.1186\/1471-2164-11-574","article-title":"Heading down the wrong pathway: on the influence of correlation within gene sets","volume":"11","author":"Gatti","year":"2010","journal-title":"BMC Genomics"},{"issue":"6","key":"2021012203310373000_ref59","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1177\/0962280209351908","article-title":"Gene set enrichment analysis made simple","volume":"18","author":"Irizarry","year":"2009","journal-title":"Stat Methods Med Res"},{"key":"2021012203310373000_ref60","article-title":"An algorithm for fast preranked gene set enrichment analysis using cumulative statistic calculation","author":"Sergushichev","year":"2016","journal-title":"bioRxiv"},{"issue":"1","key":"2021012203310373000_ref61","doi-asserted-by":"crossref","first-page":"472","DOI":"10.1177\/0962280212460441","article-title":"The limitations of simple gene set enrichment analysis assuming gene independence","volume":"25","author":"Tamayo","year":"2016","journal-title":"Stat Methods Med Res"},{"issue":"1","key":"2021012203310373000_ref62","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1214\/07-AOAS101","article-title":"On testing the significance of sets of genes","volume":"1","author":"Efron","year":"2007","journal-title":"Ann Appl Stat"},{"key":"2021012203310373000_ref63","doi-asserted-by":"crossref","first-page":"A39","DOI":"10.2202\/1544-6115.1585","article-title":"Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn","volume":"9","author":"Phipson","year":"2010","journal-title":"Stat Appl Genet Mol Biol"},{"issue":"1","key":"2021012203310373000_ref64","doi-asserted-by":"crossref","first-page":"8.25.1","DOI":"10.1002\/cpbi.42","article-title":"Network-based approaches for pathway level analysis","volume":"61","author":"Nguyen","year":"2018","journal-title":"Curr Protoc Bioinformatics"},{"issue":"1","key":"2021012203310373000_ref65","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1186\/s13059-019-1716-1","article-title":"A practical guide to methods controlling false discoveries in computational biology","volume":"20","author":"Korthauer","year":"2019","journal-title":"Genome Biol"},{"issue":"17","key":"2021012203310373000_ref66","doi-asserted-by":"crossref","first-page":"e133","DOI":"10.1093\/nar\/gks461","article-title":"Camera: a competitive gene set test accounting for inter-gene correlation","volume":"40","author":"Wu","year":"2012","journal-title":"Nucleic Acids Res"},{"issue":"16","key":"2021012203310373000_ref67","doi-asserted-by":"crossref","first-page":"2155","DOI":"10.1093\/bioinformatics\/btg290","article-title":"Pathway studio\u2014the analysis and navigation of molecular networks","volume":"19","author":"Nikitin","year":"2003","journal-title":"Bioinformatics"},{"issue":"1","key":"2021012203310373000_ref68","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1093\/bioinformatics\/btg382","article-title":"A global test for groups of genes: testing association with a clinical outcome","volume":"20","author":"Goeman","year":"2004","journal-title":"Bioinformatics"},{"key":"2021012203310373000_ref69","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1186\/1471-2105-8-242","article-title":"Improving gene set analysis of microarray data by SAM-GS","volume":"8","author":"Dinu","year":"2007","journal-title":"BMC Bioinformatics"},{"issue":"17","key":"2021012203310373000_ref70","doi-asserted-by":"crossref","first-page":"2176","DOI":"10.1093\/bioinformatics\/btq401","article-title":"ROAST: rotation gene set tests for complex microarray experiments","volume":"26","author":"Wu","year":"2010","journal-title":"Bioinformatics"},{"key":"2021012203310373000_ref71","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1186\/1471-2105-14-7","article-title":"GSVA: gene set variation analysis for microarray and RNA-seq data","volume":"14","author":"H\u00e4nzelmann","year":"2013","journal-title":"BMC Bioinformatics"},{"issue":"9","key":"2021012203310373000_ref72","doi-asserted-by":"crossref","first-page":"5116","DOI":"10.1073\/pnas.091062498","article-title":"Significance analysis of microarrays applied to the ionizing radiation response","volume":"98","author":"Tusher","year":"2001","journal-title":"Proc Natl Acad Sci U S A"},{"key":"2021012203310373000_ref73","doi-asserted-by":"crossref","first-page":"W741","DOI":"10.1093\/nar\/gki475","article-title":"WebGestalt: an integrated system for exploring gene sets in various biological contexts","volume":"33","author":"Zhang","year":"2005","journal-title":"Nucleic Acids Res"},{"issue":"2","key":"2021012203310373000_ref74","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1093\/bioinformatics\/btl567","article-title":"Using GOstats to test gene lists for GO term association","volume":"23","author":"Falcon","year":"2007","journal-title":"Bioinformatics"},{"key":"2021012203310373000_ref75","doi-asserted-by":"crossref","first-page":"W193","DOI":"10.1093\/nar\/gkm226","article-title":"G:profiler\u2014a web-based toolset for functional profiling of gene lists from large-scale experiments","volume":"35","author":"Reimand","year":"2007","journal-title":"Nucleic Acids Res"},{"key":"2021012203310373000_ref76","doi-asserted-by":"crossref","first-page":"W186","DOI":"10.1093\/nar\/gkm323","article-title":"GeneTrail\u2014advanced gene set enrichment analysis","volume":"35","author":"Backes","year":"2007","journal-title":"Nucleic Acids Res"},{"key":"2021012203310373000_ref77","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1186\/1471-2105-10-48","article-title":"GOrilla: a tool for discovery and visualization of enriched GO terms in ranked gene lists","volume":"10","author":"Eden","year":"2009","journal-title":"BMC Bioinformatics"},{"key":"2021012203310373000_ref78","doi-asserted-by":"crossref","first-page":"W305","DOI":"10.1093\/nar\/gkp427","article-title":"ToppGene Suite for gene list enrichment analysis and candidate gene prioritization","volume":"37","author":"Chen","year":"2009","journal-title":"Nucleic Acids Res"},{"issue":"8","key":"2021012203310373000_ref79","doi-asserted-by":"crossref","first-page":"1551","DOI":"10.1038\/nprot.2013.092","article-title":"Large-scale gene function analysis with the PANTHER classification system","volume":"8","author":"Mi","year":"2013","journal-title":"Nat Protoc"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/22\/1\/545\/35935074\/bbz158.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/bib\/article-pdf\/22\/1\/545\/35935074\/bbz158.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,26]],"date-time":"2024-07-26T14:46:39Z","timestamp":1722005199000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/22\/1\/545\/5722384"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,9]]},"references-count":79,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,3,9]]},"published-print":{"date-parts":[[2021,1,18]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbz158","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/674267","asserted-by":"object"}]},"ISSN":["1477-4054"],"issn-type":[{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,1]]},"published":{"date-parts":[[2020,3,9]]}}}