{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,19]],"date-time":"2026-03-19T18:20:55Z","timestamp":1773944455920,"version":"3.50.1"},"reference-count":52,"publisher":"Oxford University Press (OUP)","issue":"12","license":[{"start":{"date-parts":[[2017,2,13]],"date-time":"2017-02-13T00:00:00Z","timestamp":1486944000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"HFSP CDA"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2017,6,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>Phosphoproteomic experiments are increasingly used to study the changes in signaling occurring across different conditions. It has been proposed that changes in phosphorylation of kinase target sites can be used to infer when a kinase activity is under regulation. However, these approaches have not yet been benchmarked due to a lack of appropriate benchmarking strategies.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We used curated phosphoproteomic experiments and a gold standard dataset containing a total of 184 kinase-condition pairs where regulation is expected to occur to benchmark and compare different kinase activity inference strategies: Z-test, Kolmogorov Smirnov test, Wilcoxon rank sum test, gene set enrichment analysis (GSEA), and a multiple linear regression model. We also tested weighted variants of the Z-test and GSEA that include information on kinase sequence specificity as proxy for affinity. Finally, we tested how the number of known substrates and the type of evidence (in vivo, in vitro or in silico) supporting these influence the predictions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Most models performed well with the Z-test and the GSEA performing best as determined by the area under the ROC curve (Mean AUC = 0.722). Weighting kinase targets by the kinase target sequence preference improves the results marginally. However, the number of known substrates and the evidence supporting the interactions has a strong effect on the predictions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and Implementation<\/jats:title>\n                    <jats:p>The KSEA implementation is available in https:\/\/github.com\/ evocellnet\/ksea. Additional data is available in http:\/\/phosfate.com<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Supplementary information<\/jats:title>\n                    <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btx082","type":"journal-article","created":{"date-parts":[[2017,2,15]],"date-time":"2017-02-15T03:58:08Z","timestamp":1487131088000},"page":"1845-1851","source":"Crossref","is-referenced-by-count":87,"title":["Benchmarking substrate-based kinase activity inference using phosphoproteomic data"],"prefix":"10.1093","volume":"33","author":[{"given":"Claudia","family":"Hernandez-Armenta","sequence":"first","affiliation":[{"name":"European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI), Hinxton, UK"}]},{"given":"David","family":"Ochoa","sequence":"additional","affiliation":[{"name":"European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI), Hinxton, UK"}]},{"given":"Emanuel","family":"Gon\u00e7alves","sequence":"additional","affiliation":[{"name":"European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI), Hinxton, UK"}]},{"given":"Julio","family":"Saez-Rodriguez","sequence":"additional","affiliation":[{"name":"European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI), Hinxton, UK"},{"name":"RWTH Aachen University, Faculty of Medicine, Joint Research Center for Computational Biomedicine (JRC-COMBINE), Wendlingweg 2, Aachen, Germany"}]},{"given":"Pedro","family":"Beltrao","sequence":"additional","affiliation":[{"name":"European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI), Hinxton, UK"}]}],"member":"286","published-online":{"date-parts":[[2017,2,13]]},"reference":[{"key":"2023020205313689500_btx082-B1","doi-asserted-by":"crossref","first-page":"1622","DOI":"10.1074\/mcp.M116.058354","article-title":"Reduced-representation phosphosignatures measured by quantitative targeted MS capture cellular states and enable large-scale comparison of drug-induced phenotypes","volume":"15","author":"Abelin","year":"2016","journal-title":"Mol. Cell. Proteomics"},{"key":"2023020205313689500_btx082-B2","doi-asserted-by":"crossref","first-page":"e1","DOI":"10.1182\/blood-2013-07-512384","article-title":"Time-resolved characterization of cAMP\/PKA-dependent signaling reveals that platelet inhibition is a concerted process involving multiple signaling pathways","volume":"123","author":"Beck","year":"2014","journal-title":"Blood"},{"key":"2023020205313689500_btx082-B3","doi-asserted-by":"crossref","first-page":"212","DOI":"10.1016\/j.molcel.2012.01.026","article-title":"Proteomic investigations reveal a role for RNA processing factor THRAP3 in the DNA damage response","volume":"46","author":"Beli","year":"2012","journal-title":"Molecular Cell"},{"key":"2023020205313689500_btx082-B4","doi-asserted-by":"crossref","first-page":"rs6.","DOI":"10.1126\/scisignal.2003573","article-title":"Kinase-substrate enrichment analysis provides insights into the heterogeneity of signaling pathway activation in leukemia cells","volume":"6","author":"Casado","year":"2013","journal-title":"Sci. Signal"},{"key":"2023020205313689500_btx082-B5","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1038\/nrm2900","article-title":"Decoding signalling networks by mass spectrometry-based proteomics","volume":"11","author":"Choudhary","year":"2010","journal-title":"Nat. Rev.. Mol. Cell Biol"},{"key":"2023020205313689500_btx082-B6","doi-asserted-by":"crossref","first-page":"10762","DOI":"10.1073\/pnas.0805139105","article-title":"A quantitative atlas of mitotic phosphorylation","volume":"105","author":"Dephoure","year":"2008","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023020205313689500_btx082-B7","doi-asserted-by":"crossref","first-page":"D261","DOI":"10.1093\/nar\/gkq1104","article-title":"Phospho.ELM: a database of phosphorylation sites\u2013update 2011","volume":"39(Database issue)","author":"Dinkel","year":"2011","journal-title":"Nucleic Acids Res"},{"key":"2023020205313689500_btx082-B8","doi-asserted-by":"crossref","first-page":"1643","DOI":"10.1073\/pnas.1120985109","article-title":"Oncogene-specific activation of tyrosine kinase networks during prostate cancer progression","volume":"109","author":"Drake","year":"2012","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023020205313689500_btx082-B100"},{"key":"2023020205313689500_btx082-B9","doi-asserted-by":"crossref","first-page":"5383","DOI":"10.1021\/pr200641x","article-title":"Multidimensional strategy for sensitive phosphoproteomics incorporating protein prefractionation combined with SIMAC, HILIC, and TiO(2) chromatography applied to proximal EGF signaling","volume":"10","author":"Engholm-Keller","year":"2011","journal-title":"J. Proteome Res"},{"key":"2023020205313689500_btx082-B10","doi-asserted-by":"crossref","first-page":"608","DOI":"10.1038\/nsmb.3218","article-title":"Multilayered proteomics reveals molecular switches dictating ligand-dependent EGFR trafficking","volume":"23","author":"Francavilla","year":"2016","journal-title":"Nat. Struct. Mol. Biol"},{"key":"2023020205313689500_btx082-B11","doi-asserted-by":"crossref","first-page":"M111.008540.","DOI":"10.1074\/mcp.M111.008540","article-title":"Quantitative phospho-proteomics to investigate the polo-like kinase 1-dependent phospho-proteome","volume":"10","author":"Grosstessner-Hain","year":"2011","journal-title":"Mol. Cell. Proteomics"},{"key":"2023020205313689500_btx082-B12","doi-asserted-by":"crossref","first-page":"rs9.","DOI":"10.1126\/scisignal.2003664","article-title":"Comparative phosphoproteomic analysis of checkpoint recovery identifies new regulators of the DNA damage response","volume":"6","author":"Halim","year":"2013","journal-title":"Sci. Signal"},{"key":"2023020205313689500_btx082-B13","doi-asserted-by":"crossref","first-page":"D512","DOI":"10.1093\/nar\/gku1267","article-title":"PhosphoSitePlus, 2014: mutations, PTMs and recalibrations","volume":"43(Database issue)","author":"Hornbeck","year":"2015","journal-title":"Nucleic Acids Res"},{"key":"2023020205313689500_btx082-B14","doi-asserted-by":"crossref","first-page":"1317","DOI":"10.1126\/science.1199498","article-title":"The mTOR-regulated phosphoproteome reveals a mechanism of mTORC1-mediated inhibition of growth factor signaling","volume":"332","author":"Hsu","year":"2011","journal-title":"Science"},{"key":"2023020205313689500_btx082-B15","doi-asserted-by":"crossref","first-page":"990","DOI":"10.1038\/nbt.3327","article-title":"High-throughput phosphoproteomics reveals in vivo insulin signaling dynamics","volume":"33","author":"Humphrey","year":"2015","journal-title":"Nat. Biotechnol"},{"key":"2023020205313689500_btx082-B16","doi-asserted-by":"crossref","first-page":"1202","DOI":"10.1016\/j.celrep.2015.01.052","article-title":"A cell-signaling network temporally resolves specific versus promiscuous phosphorylation","volume":"10","author":"Kanshin","year":"2015","journal-title":"Cell Rep"},{"key":"2023020205313689500_btx082-B17","doi-asserted-by":"crossref","first-page":"3576","DOI":"10.1093\/nar\/gkg585","article-title":"MATCH: A tool for searching transcription factor binding sites in DNA sequences","volume":"31","author":"Kel","year":"2003","journal-title":"Nucleic Acids Res"},{"key":"2023020205313689500_btx082-B18","doi-asserted-by":"crossref","first-page":"964","DOI":"10.1002\/pmic.200800548","article-title":"Plant phosphoproteomics: an update","volume":"9","author":"Kersten","year":"2009","journal-title":"Proteomics"},{"key":"2023020205313689500_btx082-B19","doi-asserted-by":"crossref","first-page":"rs5.","DOI":"10.1126\/scisignal.2001497","article-title":"Quantitative phosphoproteomics identifies substrates and functional modules of Aurora and Polo-like kinase activities in mitotic cells","volume":"4","author":"Kettenbach","year":"2011","journal-title":"Sci. Signal"},{"key":"2023020205313689500_btx082-B20","doi-asserted-by":"crossref","first-page":"D695","DOI":"10.1093\/nar\/gkm902","article-title":"NetworKIN: a resource for exploring cellular phosphorylation networks","volume":"36(Database issue)","author":"Linding","year":"2007","journal-title":"Nucleic Acids Res"},{"key":"2023020205313689500_btx082-B21","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1111\/jnc.12483","article-title":"Phosphoproteomic evaluation of pharmacological inhibition of leucine-rich repeat kinase 2 reveals significant off-target effects of LRRK-2-IN-1","volume":"128","author":"Luerman","year":"2014","journal-title":"J. Neurochem"},{"key":"2023020205313689500_btx082-B22","doi-asserted-by":"crossref","first-page":"1160","DOI":"10.1126\/science.1140321","article-title":"ATM and ATR substrate analysis reveals extensive protein networks responsive to DNA damage","volume":"316","author":"Matsuoka","year":"2007","journal-title":"Science"},{"key":"2023020205313689500_btx082-B23","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1093\/nar\/gkg034","article-title":"STRING: a database of predicted functional associations between proteins","volume":"31","author":"Mering","year":"2003","journal-title":"Nucleic Acids Res"},{"key":"2023020205313689500_btx082-B24","doi-asserted-by":"crossref","first-page":"M111.014423.","DOI":"10.1074\/mcp.M111.014423","article-title":"iTRAQ labeling is superior to mTRAQ for quantitative global proteomics and phosphoproteomics","volume":"11","author":"Mertins","year":"2012","journal-title":"Mol. Cell. Proteomics"},{"key":"2023020205313689500_btx082-B25","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1038\/nature18003","article-title":"Proteogenomics connects somatic mutations to signalling in breast cancer","volume":"534","author":"Mertins","year":"2016","journal-title":"Nature"},{"key":"2023020205313689500_btx082-B26","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1038\/nature06001","article-title":"Common effector processing mediates cell-specific responses to stimuli","volume":"448","author":"Miller-Jensen","year":"2007","journal-title":"Nature"},{"key":"2023020205313689500_btx082-B27","doi-asserted-by":"crossref","first-page":"ra2.","DOI":"10.1126\/scisignal.1159433","article-title":"Linear motif atlas for phosphorylation-dependent signaling","volume":"1","author":"Miller","year":"2008","journal-title":"Sci. Signal"},{"key":"2023020205313689500_btx082-B28","doi-asserted-by":"crossref","first-page":"424","DOI":"10.1093\/bioinformatics\/btv699","article-title":"IKAP: a heuristic framework for inference of kinase activities from Phosphoproteomics data","volume":"32","author":"Mischnik","year":"2016","journal-title":"Bioinformatics"},{"key":"2023020205313689500_btx082-B29","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1016\/j.jprot.2015.10.029","article-title":"The phosphoproteome of human Jurkat T cell clones upon costimulation with anti-CD3\/anti-CD28 antibodies","volume":"131","author":"Nguyen","year":"2016","journal-title":"J. Proteomics"},{"key":"2023020205313689500_btx082-B30","doi-asserted-by":"crossref","first-page":"888.","DOI":"10.15252\/msb.20167295","article-title":"An atlas of human kinase regulation","volume":"12","author":"Ochoa","year":"2016","journal-title":"Mol. Syst. Biol"},{"key":"2023020205313689500_btx082-B31","doi-asserted-by":"crossref","DOI":"10.1038\/msb4100014","article-title":"A comprehensive pathway map of epidermal growth factor receptor signaling","volume":"1","author":"Oda","year":"2005","journal-title":"Mol. Syst. Biol"},{"key":"2023020205313689500_btx082-B32","doi-asserted-by":"crossref","first-page":"635","DOI":"10.1016\/j.cell.2006.09.026","article-title":"Global, in vivo, and site-specific phosphorylation dynamics in signaling networks","volume":"127","author":"Olsen","year":"2006","journal-title":"Cell"},{"key":"2023020205313689500_btx082-B33","doi-asserted-by":"crossref","first-page":"ra3.","DOI":"10.1126\/scisignal.2000475","article-title":"Quantitative phosphoproteomics reveals widespread full phosphorylation site occupancy during mitosis","volume":"3","author":"Olsen","year":"2010","journal-title":"Sci. Signal"},{"key":"2023020205313689500_btx082-B34","doi-asserted-by":"crossref","first-page":"4089","DOI":"10.1021\/pr400417g","article-title":"Comparison of SILAC and mTRAQ quantification for phosphoproteomics on a quadrupole orbitrap mass spectrometer","volume":"12","author":"Oppermann","year":"2013","journal-title":"J. Proteome Res"},{"key":"2023020205313689500_btx082-B35","doi-asserted-by":"crossref","first-page":"2796","DOI":"10.1074\/mcp.M900285-MCP200","article-title":"Global effects of kinase inhibitors on signaling networks revealed by quantitative phosphoproteomics","volume":"8","author":"Pan","year":"2009","journal-title":"Mol. Cell. Proteomics"},{"key":"2023020205313689500_btx082-B36","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J. Mach. Learn Res"},{"key":"2023020205313689500_btx082-B37","doi-asserted-by":"crossref","first-page":"D548","DOI":"10.1093\/nar\/gkv1048","article-title":"SIGNOR: a database of causal relationships between biological entities","volume":"44","author":"Perfetto","year":"2016","journal-title":"Nucleic Acids Res"},{"key":"2023020205313689500_btx082-B38","doi-asserted-by":"crossref","first-page":"D497","DOI":"10.1093\/nar\/gkh070","article-title":"Human protein reference database as a discovery resource for proteomics","volume":"32(Database issue)","author":"Peri","year":"2004","journal-title":"Nucleic Acids Res"},{"key":"2023020205313689500_btx082-B39","doi-asserted-by":"crossref","first-page":"rs3.","DOI":"10.1126\/scisignal.2001570","article-title":"System-wide temporal characterization of the proteome and phosphoproteome of human embryonic stem cell differentiation","volume":"4","author":"Rigbolt","year":"2011","journal-title":"Sci. Signal"},{"key":"2023020205313689500_btx082-B40","doi-asserted-by":"crossref","first-page":"e77423","DOI":"10.1371\/journal.pone.0077423","article-title":"Quantitative phosphoproteome analysis unveils LAT as a modulator of CD3\u03b6 and ZAP-70 tyrosine phosphorylation","volume":"8","author":"Salek","year":"2013","journal-title":"PloS One"},{"key":"2023020205313689500_btx082-B41","doi-asserted-by":"crossref","first-page":"12007","DOI":"10.3390\/ijms150712007","article-title":"Radiosensitization of human leukemic HL-60 cells by ATR kinase inhibitor (VE-821): phosphoproteomic analysis","volume":"15","author":"\u0160alovsk\u00e1","year":"2014","journal-title":"Int. J. Mol. Sci"},{"key":"2023020205313689500_btx082-B42","doi-asserted-by":"crossref","first-page":"716","DOI":"10.1038\/leu.2013.347","article-title":"Global phosphoproteome analysis of human bone marrow reveals predictive phosphorylation markers for the treatment of acute myeloid leukemia with quizartinib","volume":"28","author":"Schaab","year":"2014","journal-title":"Leukemia"},{"key":"2023020205313689500_btx082-B43","doi-asserted-by":"crossref","first-page":"1599","DOI":"10.1074\/mcp.M114.047233","article-title":"A phosphoproteomic comparison of B-RAFV600E and MKK1\/2 inhibitors in melanoma cells","volume":"14","author":"Stuart","year":"2015","journal-title":"Mol. Cell. Proteomics"},{"key":"2023020205313689500_btx082-B44","doi-asserted-by":"crossref","first-page":"15545","DOI":"10.1073\/pnas.0506580102","article-title":"Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles","volume":"102","author":"Subramanian","year":"2005","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023020205313689500_btx082-B45","doi-asserted-by":"crossref","first-page":"8033.","DOI":"10.1038\/ncomms9033","article-title":"Large-scale models of signal propagation in human cells derived from discovery phosphoproteomic data","volume":"6","author":"Terfve","year":"2015","journal-title":"Nat. Commun"},{"key":"2023020205313689500_btx082-B46","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1038\/nmeth.3396","article-title":"MIMP: predicting the impact of mutations on kinase-substrate phosphorylation","volume":"12","author":"Wagih","year":"2015","journal-title":"Nat. Methods"},{"key":"2023020205313689500_btx082-B47","doi-asserted-by":"crossref","first-page":"1343","DOI":"10.1016\/j.jprot.2011.11.004","article-title":"Dual phosphoproteomics and chemical proteomics analysis of erlotinib and gefitinib interference in acute myeloid leukemia cells","volume":"75","author":"Weber","year":"2012","journal-title":"J. Proteomics"},{"key":"2023020205313689500_btx082-B48","doi-asserted-by":"crossref","first-page":"4329","DOI":"10.1158\/0008-5472.CAN-12-0136","article-title":"Global quantitative phosphoproteome analysis of human tumor xenografts treated with a CD44 antagonist","volume":"72","author":"Weigand","year":"2012","journal-title":"Cancer Res"},{"key":"2023020205313689500_btx082-B49","doi-asserted-by":"crossref","first-page":"7719","DOI":"10.1073\/pnas.1423344112","article-title":"Empirical inference of circuitry and plasticity in a kinase signaling network","volume":"112","author":"Wilkes","year":"2015","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023020205313689500_btx082-B50","doi-asserted-by":"crossref","first-page":"4961.","DOI":"10.1038\/ncomms5961","article-title":"Activation of diverse signalling pathways by oncogenic PIK3CA mutations","volume":"5","author":"Wu","year":"2014","journal-title":"Nat. Commun"},{"key":"2023020205313689500_btx082-B51","doi-asserted-by":"crossref","first-page":"ra25.","DOI":"10.1126\/scisignal.2003572","article-title":"Phosphoproteomic analysis implicates the mTORC2-FoxO1 axis in VEGF signaling and feedback activation of receptor tyrosine kinases","volume":"6","author":"Zhuang","year":"2013","journal-title":"Sci. Signal"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/33\/12\/1845\/49039843\/bioinformatics_33_12_1845.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/33\/12\/1845\/49039843\/bioinformatics_33_12_1845.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,2]],"date-time":"2023-02-02T00:35:34Z","timestamp":1675298134000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/33\/12\/1845\/2991427"}},"subtitle":[],"editor":[{"given":"Jonathen","family":"Wren","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2017,2,13]]},"references-count":52,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2017,6,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btx082","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/080978","asserted-by":"object"}]},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2017,6,15]]},"published":{"date-parts":[[2017,2,13]]}}}