{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,22]],"date-time":"2026-01-22T08:04:35Z","timestamp":1769069075329,"version":"3.49.0"},"reference-count":31,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2022,7,6]],"date-time":"2022-07-06T00:00:00Z","timestamp":1657065600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,7,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>In 2014, the Immune Epitope Database automated benchmark was created to compare the performance of the MHC class I binding predictors. However, this is not a straightforward process due to the different and non-standardized outputs of the methods. Additionally, some methods are more restrictive regarding the HLA alleles and epitope sizes for which they predict binding affinities, while others are more comprehensive. To address how these problems impacted the ranking of the predictors, we developed an approach to assess the reliability of different metrics. We found that using percentile-ranked results improved the stability of the ranks and allowed the predictors to be reliably ranked despite not being evaluated on the same data. We also found that given the rate new data are incorporated into the benchmark, a new method must wait for at least 4 years to be ranked against the pre-existing methods. The best-performing tools with statistically indistinguishable scores in this benchmark were NetMHCcons, NetMHCpan4.0, ANN3.4, NetMHCpan3.0 and NetMHCpan2.8. The results of this study will be used to improve the evaluation and display of benchmark performance. We highly encourage anyone working on MHC binding predictions to participate in this benchmark to get an unbiased evaluation of their predictors.<\/jats:p>","DOI":"10.1093\/bib\/bbac259","type":"journal-article","created":{"date-parts":[[2022,7,7]],"date-time":"2022-07-07T03:47:21Z","timestamp":1657165641000},"source":"Crossref","is-referenced-by-count":8,"title":["A comprehensive analysis of the IEDB MHC class-I automated benchmark"],"prefix":"10.1093","volume":"23","author":[{"given":"Raphael","family":"Trevizani","sequence":"first","affiliation":[{"name":"Division of Vaccine Discovery , La Jolla Institute for Immunology, La Jolla, California 92037 , USA"},{"name":"Fiocruz Cear\u00e1 , Funda\u00e7\u00e3o Oswaldo Cruz, Rua S\u00e3o Jos\u00e9 s\/n, Precabura, Eus\u00e9bio\/CE , Brazil"}]},{"given":"Zhen","family":"Yan","sequence":"additional","affiliation":[{"name":"Bioinformatics Core , La Jolla Institute for Immunology, La Jolla, California 92037 , USA"}]},{"given":"Jason A","family":"Greenbaum","sequence":"additional","affiliation":[{"name":"Bioinformatics Core , La Jolla Institute for Immunology, La Jolla, California 92037 , USA"}]},{"given":"Alessandro","family":"Sette","sequence":"additional","affiliation":[{"name":"Division of Vaccine Discovery , La Jolla Institute for Immunology, La Jolla, California 92037 , USA"},{"name":"Department of Medicine , University of California San Diego, La Jolla, California 92093 , USA"}]},{"given":"Morten","family":"Nielsen","sequence":"additional","affiliation":[{"name":"Department of Health Technology , Technical University of Denmark, DK-2800 Kgs. Lyngby , Denmark"},{"name":"Instituto de Investigaciones Biotecnol\u00f3gicas , Universidad Nacional de San Mart\u00edn, B1650 Buenos Aires , Argentina"}]},{"given":"Bjoern","family":"Peters","sequence":"additional","affiliation":[{"name":"Division of Vaccine Discovery , La Jolla Institute for Immunology, La Jolla, California 92037 , USA"},{"name":"Department of Medicine , University of California San Diego, La Jolla, California 92093 , USA"}]}],"member":"286","published-online":{"date-parts":[[2022,7,6]]},"reference":[{"issue":"9","key":"2022071906193509200_ref1","doi-asserted-by":"crossref","first-page":"729","DOI":"10.1080\/14789450.2017.1361825","article-title":"Deciphering the MHC-associated peptidome: a review of naturally processed ligand data","volume":"14","author":"Vaughan","year":"2017","journal-title":"Expert Rev Proteomics"},{"issue":"1","key":"2022071906193509200_ref2","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1146\/annurev-immunol-082119-124838","article-title":"T Cell Epitope Predictions","volume":"38","author":"Peters","year":"2020","journal-title":"Annu Rev Immunol"},{"issue":"1","key":"2022071906193509200_ref3","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1146\/annurev.immunol.17.1.51","article-title":"Immunodominance in major histocompatibility complex class I-restricted T lymphocyte responses","volume":"17","author":"Yewdell","year":"1999","journal-title":"Annu Rev Immunol"},{"issue":"13","key":"2022071906193509200_ref4","doi-asserted-by":"crossref","first-page":"2174","DOI":"10.1093\/bioinformatics\/btv123","article-title":"Automated benchmarking of peptide-MHC class I binding predictions","volume":"31","author":"Trolle","year":"2015","journal-title":"Bioinformatics"},{"issue":"11","key":"2022071906193509200_ref5","doi-asserted-by":"crossref","first-page":"1397","DOI":"10.1093\/bioinformatics\/btn128","article-title":"Accurate approximation method for prediction of class I MHC affinities for peptides of lengths 8, 10, and 11 using prediction tools trained on 9mers","volume":"24","author":"Lundegaard","year":"2008","journal-title":"Bioinformatics"},{"issue":"5","key":"2022071906193509200_ref6","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1007\/s00251-005-0798-y","article-title":"Automated generation and evaluation of specific MHC binding predictive tools: ARB matrix applications","volume":"57","author":"Bui","year":"2005","journal-title":"Immunogenetics"},{"issue":"1","key":"2022071906193509200_ref7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-6-132","article-title":"Generating quantitative models describing the sequence specificity of biological processes with the stabilized matrix method","volume":"6","author":"Peters","year":"2005","journal-title":"BMC bioinformatics"},{"issue":"3","key":"2022071906193509200_ref8","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1007\/s00251-011-0579-8","article-title":"NetMHCcons: a consensus method for the major histocompatibility complex class I predictions","volume":"64","author":"Karosiene","year":"2012","journal-title":"Immunogenetics"},{"issue":"1","key":"2022071906193509200_ref9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-10-394","article-title":"Derivation of an amino acid similarity matrix for peptide: MHC binding and its application as a Bayesian prior","volume":"10","author":"Kim","year":"2009","journal-title":"BMC bioinformatics"},{"issue":"7","key":"2022071906193509200_ref10","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1038\/nbt1215","article-title":"A consensus epitope prediction approach identifies the breadth of murine T CD8+-cell responses to vaccinia virus","volume":"24","author":"Moutaftsi","year":"2006","journal-title":"Nat Biotechnol"},{"issue":"10","key":"2022071906193509200_ref11","doi-asserted-by":"crossref","first-page":"1293","DOI":"10.1093\/bioinformatics\/btp137","article-title":"The PickPocket method for predicting binding specificities for receptors based on receptor pocket similarities: application to MHC-peptide binding","volume":"25","author":"Zhang","year":"2009","journal-title":"Bioinformatics"},{"issue":"4","key":"2022071906193509200_ref12","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1093\/bioinformatics\/btv639","article-title":"Gapped sequence alignment using artificial neural networks: application to the MHC class I system","volume":"32","author":"Andreatta","year":"2016","journal-title":"Bioinformatics"},{"issue":"1","key":"2022071906193509200_ref13","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/j.cels.2020.06.010","article-title":"MHCflurry 2.0: Improved Pan-Allele Prediction of MHC Class I-Presented Peptides by Incorporating Antigen Processing","volume":"11","author":"O\u2019Donnell","year":"2020","journal-title":"Cell Systems"},{"issue":"1","key":"2022071906193509200_ref14","article-title":"DeepSeqPan, a novel deep convolutional neural network model for pan-specific class I HLA-peptide binding affinity prediction","volume":"9","author":"Liu","year":"2019","journal-title":"Sci Rep"},{"issue":"1","key":"2022071906193509200_ref15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s00251-008-0341-z","article-title":"NetMHCpan, a method for MHC class I binding prediction beyond humans","volume":"61","author":"Hoof","year":"2009","journal-title":"Immunogenetics"},{"issue":"1","key":"2022071906193509200_ref16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13073-016-0288-x","article-title":"NetMHCpan-3.0; improved prediction of binding to MHC class I molecules integrating information from multiple receptor and peptide length datasets","volume":"8","author":"Nielsen","year":"2016","journal-title":"Genome Med"},{"issue":"9","key":"2022071906193509200_ref17","doi-asserted-by":"crossref","first-page":"3360","DOI":"10.4049\/jimmunol.1700893","article-title":"NetMHCpan-4.0: Improved Peptide-MHC Class I Interaction Predictions Integrating Eluted Ligand and Peptide Binding Affinity Data","volume":"199","author":"Jurtz","year":"2017","journal-title":"The Journal of Immunology"},{"issue":"3\u20134","key":"2022071906193509200_ref18","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1007\/s002510050595","article-title":"SYFPEITHI: database for MHC ligands and peptide motifs","volume":"50","author":"Rammensee","year":"1999","journal-title":"Immunogenetics"},{"issue":"1","key":"2022071906193509200_ref19","doi-asserted-by":"crossref","DOI":"10.1186\/1471-2105-8-238","article-title":"Prediction of MHC class II binding affinity using SMM-align, a novel stabilization matrix alignment method","volume":"8","author":"Nielsen","year":"2007","journal-title":"BMC Bioinformatics"},{"issue":"6","key":"2022071906193509200_ref20","doi-asserted-by":"crossref","DOI":"10.1093\/bib\/bbab216","article-title":"Predicting MHC class I binder: existing approaches and a novel recurrent neural network solution","volume":"22","author":"Jiang","year":"2021","journal-title":"Brief Bioinform"},{"issue":"11","key":"2022071906193509200_ref21","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1006457","article-title":"Systematically benchmarking peptide-MHC binding predictors: From synthetic to naturally processed epitopes","volume":"14","author":"Zhao","year":"2018","journal-title":"PLoS Comput Biol"},{"issue":"1","key":"2022071906193509200_ref22","doi-asserted-by":"crossref","DOI":"10.1186\/1471-2172-9-8","article-title":"Evaluation of MHC class I peptide binding prediction servers: Applications for vaccine research","volume":"9","author":"Lin","year":"2008","journal-title":"BMC Immunol"},{"issue":"5","key":"2022071906193509200_ref23","doi-asserted-by":"crossref","first-page":"719","DOI":"10.1158\/2326-6066.CIR-18-0584","article-title":"Performance Evaluation of MHC Class-I Binding Prediction Tools Based on an Experimentally Validated MHC-Peptide Binding dataset","volume":"7","author":"Bonsack","year":"2019","journal-title":"Cancer Immunol Res"},{"issue":"4","key":"2022071906193509200_ref24","doi-asserted-by":"crossref","first-page":"1119","DOI":"10.1093\/bib\/bbz051","article-title":"A comprehensive review and performance evaluation of bioinformatics tools for HLA class I peptide-binding prediction","volume":"21","author":"Mei","year":"2020","journal-title":"Brief Bioinform"},{"issue":"7","key":"2022071906193509200_ref25","doi-asserted-by":"crossref","DOI":"10.1080\/2162402X.2016.1177691","article-title":"Current tools for predicting cancer-specific T cell immunity","volume":"5","author":"Gfeller","year":"2016","journal-title":"OncoImmunology"},{"issue":"14","key":"2022071906193509200_ref26","doi-asserted-by":"crossref","first-page":"e227","DOI":"10.1093\/bioinformatics\/btl255","article-title":"Learning MHC I-peptide binding","volume":"22","author":"Jojic","year":"2006","journal-title":"Bioinformatics"},{"issue":"3","key":"2022071906193509200_ref27","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1093\/bioinformatics\/btm611","article-title":"Efficient peptide-MHC-I binding prediction for alleles with few known binders","volume":"24","author":"Jacob","year":"2007","journal-title":"Bioinformatics"},{"issue":"8","key":"2022071906193509200_ref28","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0000796","article-title":"NetMHCpan, a method for quantitative predictions of peptide binding to any HLA-A and -B locus protein of known sequence","volume":"2","author":"Nielsen","year":"2007","journal-title":"PloS one"},{"issue":"8","key":"2022071906193509200_ref29","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0041710","article-title":"Predicting HLA Class I Non-Permissive Amino Acid Residues Substitutions","volume":"7","author":"Binkowski","year":"2012","journal-title":"PLoS ONE"},{"issue":"Web Server","key":"2022071906193509200_ref30","doi-asserted-by":"crossref","first-page":"W172","DOI":"10.1093\/nar\/gki452","article-title":"MULTIPRED: a computational system for prediction of promiscuous HLA binding peptides","volume":"33","author":"Zhang","year":"2005","journal-title":"Nucleic Acids Res"},{"issue":"3\u20134","key":"2022071906193509200_ref31","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1007\/s002510050594","article-title":"Nine major HLA class I supertypes account for the vast preponderance of HLA-A and -B polymorphism","volume":"50","author":"Sette","year":"1999","journal-title":"Immunogenetics"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/4\/bbac259\/45017883\/bbac259.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/4\/bbac259\/45017883\/bbac259.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,7,19]],"date-time":"2022-07-19T06:23:27Z","timestamp":1658211807000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbac259\/6632617"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,6]]},"references-count":31,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,7,18]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbac259","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,7,18]]},"published":{"date-parts":[[2022,7,6]]},"article-number":"bbac259"}}