{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T02:59:54Z","timestamp":1774061994516,"version":"3.50.1"},"reference-count":41,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2022,4,26]],"date-time":"2022-04-26T00:00:00Z","timestamp":1650931200000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000265","name":"UK Medical Research Council","doi-asserted-by":"publisher","award":["MC_UU_12010\/3"],"award-info":[{"award-number":["MC_UU_12010\/3"]}],"id":[{"id":"10.13039\/501100000265","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012190","name":"Ministry of Science and Higher Education of the Russian Federation","doi-asserted-by":"publisher","award":["075-15-2020-807"],"award-info":[{"award-number":["075-15-2020-807"]}],"id":[{"id":"10.13039\/501100012190","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,5,13]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>T cell recognition of a cognate peptide\u2013major histocompatibility complex (pMHC) presented on the surface of infected or malignant cells is of the utmost importance for mediating robust and long-term immune responses. Accurate predictions of cognate pMHC targets for T cell receptors would greatly facilitate identification of vaccine targets for both pathogenic diseases and personalized cancer immunotherapies. Predicting immunogenic peptides therefore has been at the center of intensive research for the past decades but has proven challenging. Although numerous models have been proposed, performance of these models has not been systematically evaluated and their success rate in predicting epitopes in the context of human pathology has not been measured and compared. In this study, we evaluated the performance of several publicly available models, in identifying immunogenic CD8+ T cell targets in the context of pathogens and cancers. We found that for predicting immunogenic peptides from an emerging virus such as severe acute respiratory syndrome coronavirus 2, none of the models perform substantially better than random or offer considerable improvement beyond HLA ligand prediction. We also observed suboptimal performance for predicting cancer neoantigens. Through investigation of potential factors associated with ill performance of models, we highlight several data- and model-associated issues. In particular, we observed that cross-HLA variation in the distribution of immunogenic and non-immunogenic peptides in the training data of the models seems to substantially confound the predictions. We additionally compared key parameters associated with immunogenicity between pathogenic peptides and cancer neoantigens and observed evidence for differences in the thresholds of binding affinity and stability, which suggested the need to modulate different features in identifying immunogenic pathogen versus cancer peptides. Overall, we demonstrate that accurate and reliable predictions of immunogenic CD8+ T cell targets remain unsolved; thus, we hope our work will guide users and model developers regarding potential pitfalls and unsettled questions in existing immunogenicity predictors.<\/jats:p>","DOI":"10.1093\/bib\/bbac141","type":"journal-article","created":{"date-parts":[[2022,3,28]],"date-time":"2022-03-28T07:06:50Z","timestamp":1648451210000},"source":"Crossref","is-referenced-by-count":32,"title":["Evaluating performance of existing computational models in predicting CD8+ T cell pathogenic epitopes and cancer neoantigens"],"prefix":"10.1093","volume":"23","author":[{"given":"Paul R","family":"Buckley","sequence":"first","affiliation":[{"name":"MRC Human Immunology Unit, Medical Research Council (MRC) Human Immunology Unit, MRC Weatherall Institute of Molecular Medicine (WIMM), John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"},{"name":"MRC WIMM Centre for Computational Biology, Medical Research Council (MRC) Weatherall Institute of Molecular Medicine, John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"}]},{"given":"Chloe H","family":"Lee","sequence":"additional","affiliation":[{"name":"MRC Human Immunology Unit, Medical Research Council (MRC) Human Immunology Unit, MRC Weatherall Institute of Molecular Medicine (WIMM), John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"},{"name":"MRC WIMM Centre for Computational Biology, Medical Research Council (MRC) Weatherall Institute of Molecular Medicine, John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"}]},{"given":"Ruichong","family":"Ma","sequence":"additional","affiliation":[{"name":"MRC Human Immunology Unit, Medical Research Council (MRC) Human Immunology Unit, MRC Weatherall Institute of Molecular Medicine (WIMM), John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"},{"name":"Department of Neurosurgery, Oxford University Hospitals NHS Foundation Trust, John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"},{"name":"Nuffield Department of Surgical Sciences, University of Oxford, Oxford, United Kingdom"}]},{"given":"Isaac","family":"Woodhouse","sequence":"additional","affiliation":[{"name":"Centre for Immuno-Oncology, Nuffield Department of Medicine, University of Oxford, Oxford, United Kingdom"}]},{"given":"Jeongmin","family":"Woo","sequence":"additional","affiliation":[{"name":"MRC Human Immunology Unit, Medical Research Council (MRC) Human Immunology Unit, MRC Weatherall Institute of Molecular Medicine (WIMM), John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"},{"name":"MRC WIMM Centre for Computational Biology, Medical Research Council (MRC) Weatherall Institute of Molecular Medicine, John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"}]},{"given":"Vasily O","family":"Tsvetkov","sequence":"additional","affiliation":[{"name":"Chief Scientific Officer, ImmunoMind Inc, Berkeley, CA, USA"}]},{"given":"Dmitrii S","family":"Shcherbinin","sequence":"additional","affiliation":[{"name":"Department of Genomics of Adaptive Immunity, Shemyakin-Ovchinnikov Institute of Bioorganic Chemistry, Moscow, 117997, Russia"},{"name":"Institute of Translational Medicine, Pirogov Russian National Research Medical University, Moscow, 117997, Russia"}]},{"given":"Agne","family":"Antanaviciute","sequence":"additional","affiliation":[{"name":"MRC Human Immunology Unit, Medical Research Council (MRC) Human Immunology Unit, MRC Weatherall Institute of Molecular Medicine (WIMM), John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"},{"name":"MRC WIMM Centre for Computational Biology, Medical Research Council (MRC) Weatherall Institute of Molecular Medicine, John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"}]},{"given":"Mikhail","family":"Shughay","sequence":"additional","affiliation":[{"name":"Department of Genomics of Adaptive Immunity, Shemyakin-Ovchinnikov Institute of Bioorganic Chemistry, Moscow, 117997, Russia"},{"name":"Institute of Translational Medicine, Pirogov Russian National Research Medical University, Moscow, 117997, Russia"}]},{"given":"Margarida","family":"Rei","sequence":"additional","affiliation":[{"name":"The Ludwig Institute for Cancer Research, Old Road Campus Research Building, University of Oxford, Oxford, United Kingdom"}]},{"given":"Alison","family":"Simmons","sequence":"additional","affiliation":[{"name":"MRC Human Immunology Unit, Medical Research Council (MRC) Human Immunology Unit, MRC Weatherall Institute of Molecular Medicine (WIMM), John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3640-7043","authenticated-orcid":false,"given":"Hashem","family":"Koohy","sequence":"additional","affiliation":[{"name":"MRC Human Immunology Unit, Medical Research Council (MRC) Human Immunology Unit, MRC Weatherall Institute of Molecular Medicine (WIMM), John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"},{"name":"MRC WIMM Centre for Computational Biology, Medical Research Council (MRC) Weatherall Institute of Molecular Medicine, John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom"},{"name":"Alan Turning Fellow, University of Oxford, Oxford, United Kingdom"}]}],"member":"286","published-online":{"date-parts":[[2022,4,25]]},"reference":[{"key":"2022051813181822600_ref1","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.immuni.2011.07.010","article-title":"CD8+ T cells: foot soldiers of the immune system","volume":"35","author":"Zhang","year":"2011","journal-title":"Immunity"},{"key":"2022051813181822600_ref2","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1152\/advan.00066.2013","article-title":"T cell responses: naive to memory and everything in between","volume":"37","author":"Pennock","year":"2013","journal-title":"Adv Physiol Educ"},{"key":"2022051813181822600_ref3","doi-asserted-by":"crossref","DOI":"10.1016\/j.cell.2020.09.015","article-title":"Key parameters of tumor epitope immunogenicity revealed through a consortium approach improve Neoantigen prediction","volume":"183","author":"Wells","year":"2020","journal-title":"Cell"},{"key":"2022051813181822600_ref4","doi-asserted-by":"crossref","first-page":"2492","DOI":"10.4049\/jimmunol.1600808","article-title":"Unsupervised HLA peptidome deconvolution improves ligand prediction accuracy and predicts cooperative effects in peptide\u2013HLA interactions","volume":"197","author":"Bassani-Sternberg","year":"2016","journal-title":"J Immunol"},{"key":"2022051813181822600_ref5","article-title":"HLA binding of self-peptides is biased towards proteins with specific molecular functions","author":"Karnaukhov","year":"2021","journal-title":"bioRxiv"},{"key":"2022051813181822600_ref6","doi-asserted-by":"crossref","first-page":"W449","DOI":"10.1093\/nar\/gkaa379","article-title":"NetMHCpan-4.1 and NetMHCIIpan-4.0: improved predictions of MHC antigen presentation by concurrent motif deconvolution and integration of MS MHC eluted ligand data","volume":"48","author":"Reynisson","year":"2020","journal-title":"Nucleic Acids Res"},{"key":"2022051813181822600_ref7","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1005725","article-title":"Deciphering HLA-I motifs across HLA peptidomes improves neo-antigen predictions and identifies allostery regulating HLA specificity","volume":"13","author":"Bassani-Sternberg","year":"2017","journal-title":"PLoS Comput Biol"},{"key":"2022051813181822600_ref8","doi-asserted-by":"crossref","DOI":"10.3389\/fimmu.2020.565096","article-title":"Predicting cross-reactivity and antigen specificity of T cell receptors","volume":"11","author":"Lee","year":"2020","journal-title":"Front Immunol"},{"key":"2022051813181822600_ref9","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1038\/s41577-020-0391-5","article-title":"Constitutive immune mechanisms: mediators of host defence and immune regulation","volume":"21","author":"Paludan","year":"2020","journal-title":"Nat Rev Immunol"},{"key":"2022051813181822600_ref10","doi-asserted-by":"crossref","first-page":"873","DOI":"10.1038\/s41592-020-0867-z","article-title":"T cell antigen discovery","volume":"18","author":"Joglekar","year":"2021","journal-title":"Nat Methods"},{"key":"2022051813181822600_ref11","doi-asserted-by":"crossref","DOI":"10.3389\/fimmu.2019.00827","article-title":"Quantitative prediction of the landscape of T cell epitope immunogenicity in sequence space","volume":"10","author":"Ogishi","year":"2019","journal-title":"Front Immunol"},{"key":"2022051813181822600_ref12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13073-018-0577-7","article-title":"Exploring the pre-immune landscape of antigen-specific T cells","volume":"10","author":"Pogorelyy","year":"2018","journal-title":"Genome Med"},{"key":"2022051813181822600_ref13","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1007\/s00251-014-0779-0","article-title":"NetTepi: an integrated method for the prediction of T cell epitopes","volume":"66","author":"Trolle","year":"2014","journal-title":"Immunogenetics"},{"key":"2022051813181822600_ref14","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1003266","article-title":"Properties of MHC class I presented peptides that enhance immunogenicity","volume":"9","author":"Calis","year":"2013","journal-title":"PLoS Comput Biol"},{"key":"2022051813181822600_ref15","first-page":"1","article-title":"DeepImmuno: deep learning-empowered prediction and generation of immunogenic peptides for T-cell immunity","volume":"00","author":"Li","year":"2021","journal-title":"Brief Bioinform"},{"key":"2022051813181822600_ref16","doi-asserted-by":"crossref","first-page":"517","DOI":"10.1038\/nature24473","article-title":"A neoantigen fitness model predicts tumour response to checkpoint blockade immunotherapy","volume":"551","author":"Luksza","year":"2017","journal-title":"Nature"},{"key":"2022051813181822600_ref17","doi-asserted-by":"crossref","first-page":"1566","DOI":"10.3389\/fimmu.2017.01566","article-title":"An analysis of natural T cell responses to predicted tumor neoepitopes","volume":"8","author":"Bjerregaard","year":"2017","journal-title":"Front Immunol"},{"key":"2022051813181822600_ref18","doi-asserted-by":"crossref","first-page":"2047","DOI":"10.3389\/fimmu.2019.02047","article-title":"Structure based prediction of neoantigen immunogenicity","volume":"10","author":"Riley","year":"2019","journal-title":"Front Immunol"},{"key":"2022051813181822600_ref19","article-title":"Prediction of neo-epitope immunogenicity reveals TCR recognition determinants and provides insight into immunoediting","volume":"2","author":"Schmidt","year":"2021","journal-title":"Cell Rep Med"},{"key":"2022051813181822600_ref20","doi-asserted-by":"crossref","DOI":"10.1084\/jem.20190179","article-title":"Mutation position is an important determinant for predicting cancer neoantigens","volume-title":"J Exp Med","author":"Capietto","year":"2020"},{"key":"2022051813181822600_ref21","article-title":"To what extent does MHC binding translate to immunogenicity in humans?","volume":"3\u20134","author":"Lee","year":"2021","journal-title":"ImmunoInformatics"},{"key":"2022051813181822600_ref22","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1016\/j.cels.2019.08.009","article-title":"Neoantigen dissimilarity to the self-proteome predicts immunogenicity and response to immune checkpoint blockade in brief","volume":"9","author":"Richman","year":"2019","journal-title":"Cell Systems"},{"key":"2022051813181822600_ref23","doi-asserted-by":"crossref","first-page":"1269","DOI":"10.1038\/s41589-020-0610-1","article-title":"Structural dissimilarity from self drives neoepitope escape from immune tolerance","volume":"16","author":"Devlin","year":"2020","journal-title":"Nat Chem Biol"},{"key":"2022051813181822600_ref24","doi-asserted-by":"crossref","first-page":"3112","DOI":"10.1073\/pnas.1815239116","article-title":"Most viral peptides displayed by class I MHC on infected cells are immunogenic","volume":"116","author":"Croft","year":"2019","journal-title":"Proc Natl Acad Sci U S A"},{"key":"2022051813181822600_ref25","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1007757","article-title":"Benchmarking predictions of MHC class I restricted T cell epitopes in a comprehensively studied model system","volume":"16","author":"Paul","year":"2020","journal-title":"PLoS Comput Biol"},{"key":"2022051813181822600_ref26","article-title":"Predicting the immunogenicity of T cell epitopes: from HIV to SARS-CoV-2","author":"Gao","year":"2020","journal-title":"bioRxiv"},{"key":"2022051813181822600_ref27","article-title":"HLA-dependent variation in SARS-CoV-2 CD8+ T cell cross-reactivity with human coronaviruses","volume":"0","author":"Buckley","year":"2022","journal-title":"Immunology"},{"key":"2022051813181822600_ref28","doi-asserted-by":"crossref","first-page":"2878","DOI":"10.3389\/fimmu.2020.579480","article-title":"Potential CD8+ T cell Cross-reactivity against SARS-CoV-2 conferred by other coronavirus strains","volume":"11","author":"Lee","year":"2020","journal-title":"Front Immunol"},{"key":"2022051813181822600_ref29","doi-asserted-by":"crossref","first-page":"W502","DOI":"10.1093\/nar\/gkz452","article-title":"IEDB-AR: immune epitope database - analysis resource in 2019","volume":"47","author":"Dhanda","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"2022051813181822600_ref30","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1093\/nar\/gkr859","article-title":"ViPR: an open bioinformatics database and analysis resource for virology research","volume":"40","author":"Pickett","year":"2012","journal-title":"Nucleic Acids Res"},{"key":"2022051813181822600_ref31","doi-asserted-by":"crossref","first-page":"27","DOI":"10.3389\/fimmu.2020.00027","article-title":"Computational prediction and validation of tumor-associated neoantigens","volume":"11","author":"Roudko","year":"2020","journal-title":"Front Immunol"},{"key":"2022051813181822600_ref32","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1038\/nature14001","article-title":"Predicting immunogenic tumour mutations by combining mass spectrometry and exome sequencing","volume":"515","author":"Yadav","year":"2014","journal-title":"Nature"},{"key":"2022051813181822600_ref33","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0118432","article-title":"The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets","volume":"10","author":"Saito","year":"2015","journal-title":"PLoS One"},{"key":"2022051813181822600_ref34","doi-asserted-by":"crossref","first-page":"2559","DOI":"10.3389\/fimmu.2019.02559","article-title":"DeepHLApan: a deep learning approach for Neoantigen prediction considering both HLA-peptide binding and immunogenicity","volume":"10","author":"Wu","year":"2019","journal-title":"Front Immunol"},{"key":"2022051813181822600_ref35","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2100542118","article-title":"Self-mediated positive selection of T cells sets an obstacle to the recognition of nonself","volume":"118","author":"Koncz","year":"2021","journal-title":"Proc Natl Acad Sci U S A"},{"key":"2022051813181822600_ref36","doi-asserted-by":"crossref","first-page":"5831","DOI":"10.4049\/jimmunol.1302101","article-title":"HLA class I alleles are associated with peptide-binding repertoires of different size, affinity, and immunogenicity","volume":"191","author":"Paul","year":"2013","journal-title":"J Immunol"},{"key":"2022051813181822600_ref37","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1007\/s00262-017-2001-3","article-title":"MuPeXI: prediction of neo-epitopes from tumor sequencing data","volume":"66","author":"Bjerregaard","year":"2017","journal-title":"Cancer Immunol Immunother"},{"key":"2022051813181822600_ref38","doi-asserted-by":"crossref","first-page":"E590","DOI":"10.1002\/ijc.29118","article-title":"NY-ESO-1 specific antibody and cellular responses in melanoma patients primed with NY-ESO-1 protein in ISCOMATRIX and boosted with recombinant NY-ESO-1 fowlpox virus","volume":"136","author":"Chen","year":"2015","journal-title":"Int J Cancer"},{"key":"2022051813181822600_ref39","doi-asserted-by":"crossref","first-page":"1926","DOI":"10.1038\/s41596-019-0170-6","article-title":"Induction of neoantigen-reactive T cells from healthy donors","volume":"14","author":"Ali","year":"2019","journal-title":"Nat Protoc"},{"key":"2022051813181822600_ref40","doi-asserted-by":"crossref","first-page":"1120","DOI":"10.1038\/nprot.2006.121","article-title":"Generation of peptide-MHC class I complexes through UV-mediated ligand exchange","volume":"1","author":"Rodenko","year":"2006","journal-title":"Nat Protoc"},{"key":"2022051813181822600_ref41","doi-asserted-by":"crossref","first-page":"1517","DOI":"10.4049\/jimmunol.1600582","article-title":"Pan-specific prediction of peptide\u2013MHC class I complex stability, a correlate of T cell immunogenicity","volume":"197","author":"Rasmussen","year":"2016","journal-title":"J Immunol"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/3\/bbac141\/43745774\/bbac141.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/3\/bbac141\/43745774\/bbac141.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,5,18]],"date-time":"2022-05-18T09:27:18Z","timestamp":1652866038000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbac141\/6573960"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,25]]},"references-count":41,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,5,13]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbac141","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2020.12.25.424183","asserted-by":"object"}]},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,5]]},"published":{"date-parts":[[2022,4,25]]},"article-number":"bbac141"}}