{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,26]],"date-time":"2026-10-26T11:12:37Z","timestamp":1793013157236,"version":"4.5.0"},"reference-count":35,"publisher":"Oxford University Press (OUP)","issue":"13","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2015,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Motivation: Numerous in silico methods predicting peptide binding to major histocompatibility complex (MHC) class I molecules have been developed over the last decades. However, the multitude of available prediction tools makes it non-trivial for the end-user to select which tool to use for a given task. To provide a solid basis on which to compare different prediction tools, we here describe a framework for the automated benchmarking of peptide-MHC class I binding prediction tools. The framework runs weekly benchmarks on data that are newly entered into the Immune Epitope Database (IEDB), giving the public access to frequent, up-to-date performance evaluations of all participating tools. To overcome potential selection bias in the data included in the IEDB, a strategy was implemented that suggests a set of peptides for which different prediction methods give divergent predictions as to their binding capability. Upon experimental binding validation, these peptides entered the benchmark study.<\/jats:p>\n               <jats:p>Results: The benchmark has run for 15 weeks and includes evaluation of 44 datasets covering 17 MHC alleles and more than 4000 peptide-MHC binding measurements. Inspection of the results allows the end-user to make educated selections between participating tools. Of the four participating servers, NetMHCpan performed the best, followed by ANN, SMM and finally ARB.<\/jats:p>\n               <jats:p>Availability and implementation: Up-to-date performance evaluations of each server can be found online at http:\/\/tools.iedb.org\/auto_bench\/mhci\/weekly. All prediction tool developers are invited to participate in the benchmark. Sign-up instructions are available at http:\/\/tools.iedb.org\/auto_bench\/mhci\/join.<\/jats:p>\n               <jats:p>Contact: \u00a0mniel@cbs.dtu.dk or bpeters@liai.org<\/jats:p>\n               <jats:p>Supplementary information: \u00a0Supplementary data are available at Bioinformatics online.<\/jats:p>","DOI":"10.1093\/bioinformatics\/btv123","type":"journal-article","created":{"date-parts":[[2015,2,26]],"date-time":"2015-02-26T15:34:19Z","timestamp":1424964859000},"page":"2174-2181","source":"Crossref","is-referenced-by-count":137,"title":["Automated benchmarking of peptide-MHC class I binding predictions"],"prefix":"10.1093","volume":"31","author":[{"given":"Thomas","family":"Trolle","sequence":"first","affiliation":[{"name":"1 Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, 2Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and 3Instituto de Investigaciones Biotecnol\u00f3gicas, Universidad Nacional de San Mart\u00edn, San Mart\u00edn, Buenos Aires, Argentina"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Imir G.","family":"Metushi","sequence":"additional","affiliation":[{"name":"1 Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, 2Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and 3Instituto de Investigaciones Biotecnol\u00f3gicas, Universidad Nacional de San Mart\u00edn, San Mart\u00edn, Buenos Aires, Argentina"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason A.","family":"Greenbaum","sequence":"additional","affiliation":[{"name":"1 Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, 2Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and 3Instituto de Investigaciones Biotecnol\u00f3gicas, Universidad Nacional de San Mart\u00edn, San Mart\u00edn, Buenos Aires, Argentina"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yohan","family":"Kim","sequence":"additional","affiliation":[{"name":"1 Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, 2Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and 3Instituto de Investigaciones Biotecnol\u00f3gicas, Universidad Nacional de San Mart\u00edn, San Mart\u00edn, Buenos Aires, Argentina"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"Sidney","sequence":"additional","affiliation":[{"name":"1 Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, 2Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and 3Instituto de Investigaciones Biotecnol\u00f3gicas, Universidad Nacional de San Mart\u00edn, San Mart\u00edn, Buenos Aires, Argentina"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ole","family":"Lund","sequence":"additional","affiliation":[{"name":"1 Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, 2Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and 3Instituto de Investigaciones Biotecnol\u00f3gicas, Universidad Nacional de San Mart\u00edn, San Mart\u00edn, Buenos Aires, Argentina"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessandro","family":"Sette","sequence":"additional","affiliation":[{"name":"1 Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, 2Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and 3Instituto de Investigaciones Biotecnol\u00f3gicas, Universidad Nacional de San Mart\u00edn, San Mart\u00edn, Buenos Aires, Argentina"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bjoern","family":"Peters","sequence":"additional","affiliation":[{"name":"1 Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, 2Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and 3Instituto de Investigaciones Biotecnol\u00f3gicas, Universidad Nacional de San Mart\u00edn, San Mart\u00edn, Buenos Aires, Argentina"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Morten","family":"Nielsen","sequence":"additional","affiliation":[{"name":"1 Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, 2Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and 3Instituto de Investigaciones Biotecnol\u00f3gicas, Universidad Nacional de San Mart\u00edn, San Mart\u00edn, Buenos Aires, Argentina"},{"name":"1 Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, 2Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and 3Instituto de Investigaciones Biotecnol\u00f3gicas, Universidad Nacional de San Mart\u00edn, San Mart\u00edn, Buenos Aires, Argentina"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2015,2,25]]},"reference":[{"key":"2023020202135070300_btv123-B1","doi-asserted-by":"crossref","first-page":"9130","DOI":"10.1073\/pnas.90.19.9130","article-title":"Evidence that transporters associated with antigen processing translocate a major histocompatibility complex class I-binding peptide into the endoplasmic reticulum in an ATP-dependent manner","volume":"90","author":"Androlewicz","year":"1993","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"2023020202135070300_btv123-B2","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1007\/s00251-005-0798-y","article-title":"Automated generation and evaluation of specific MHC binding predictive tools: ARB matrix applications","volume":"57","author":"Bui","year":"2005","journal-title":"Immunogenetics"},{"key":"2023020202135070300_btv123-B3","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1002\/prot.22561","article-title":"Evaluation of template-based models in CASP8 with standard measures","volume":"77","author":"Cozzetto","year":"2009","journal-title":"Proteins"},{"key":"2023020202135070300_btv123-B4","doi-asserted-by":"crossref","first-page":"1242","DOI":"10.1093\/bioinformatics\/17.12.1242","article-title":"EVA: continuous automatic evaluation of protein structure prediction servers","volume":"17","author":"Eyrich","year":"2001","journal-title":"Bioinformatics"},{"key":"2023020202135070300_btv123-B5","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1016\/S1093-3263(03)00160-8","article-title":"Quantitative online prediction of peptide binding to the major histocompatibility complex","volume":"22","author":"Hattotuwagama","year":"2004","journal-title":"J. Mol. Graph. Model."},{"key":"2023020202135070300_btv123-B6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s00251-008-0341-z","article-title":"NetMHCpan, a method for MHC class I binding prediction beyond humans","volume":"61","author":"Hoof","year":"2009","journal-title":"Immunogenetics"},{"key":"2023020202135070300_btv123-B7","doi-asserted-by":"crossref","first-page":"4135","DOI":"10.4049\/jimmunol.1102661","article-title":"The role of naive T cell precursor frequency and recruitment in dictating immune response magnitude","volume":"188","author":"Jenkins","year":"2012","journal-title":"J. Immunol."},{"key":"2023020202135070300_btv123-B8","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1007\/s00251-011-0579-8","article-title":"NetMHCcons: a consensus method for the major histocompatibility complex class I predictions","volume":"64","author":"Karosiene","year":"2012","journal-title":"Immunogenetics"},{"key":"2023020202135070300_btv123-B9","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1186\/1471-2105-10-394","article-title":"Derivation of an amino acid similarity matrix for peptide: MHC binding and its application as a Bayesian prior","volume":"10","author":"Kim","year":"2009","journal-title":"BMC Bioinformatics"},{"key":"2023020202135070300_btv123-B10","doi-asserted-by":"crossref","first-page":"W525","DOI":"10.1093\/nar\/gks438","article-title":"Immune epitope database analysis resource","volume":"40","author":"Kim","year":"2012","journal-title":"Nucleic Acids Res."},{"key":"2023020202135070300_btv123-B11","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1186\/1471-2105-15-241","article-title":"Dataset size and composition impact the reliability of performance benchmarks for peptide-MHC binding predictions","volume":"15","author":"Kim","year":"2014","journal-title":"BMC Bioinformatics"},{"key":"2023020202135070300_btv123-B12","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1002\/prot.24399","article-title":"CASP prediction center infrastructure and evaluation measures in CASP10 and CASP ROLL","volume":"82","author":"Kryshtafovych","year":"2014","journal-title":"Proteins"},{"key":"2023020202135070300_btv123-B13","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1186\/1471-2172-9-8","article-title":"Evaluation of MHC class I peptide binding prediction servers: applications for vaccine research","volume":"9","author":"Lin","year":"2008","journal-title":"BMC Immunol."},{"key":"2023020202135070300_btv123-B14","doi-asserted-by":"crossref","first-page":"S22","DOI":"10.1186\/1471-2105-9-S12-S22","article-title":"Evaluation of MHC-II peptide binding prediction servers: applications for vaccine research","volume":"9","author":"Lin","year":"2008","journal-title":"BMC Bioinformatics"},{"key":"2023020202135070300_btv123-B15","doi-asserted-by":"crossref","first-page":"W509","DOI":"10.1093\/nar\/gkn202","article-title":"NetMHC-3.0: accurate web accessible predictions of human, mouse and monkey MHC class I affinities for peptides of length 8-11","volume":"36","author":"Lundegaard","year":"2008","journal-title":"Nucleic Acids Res."},{"key":"2023020202135070300_btv123-B16","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1038\/nbt1215","article-title":"A consensus epitope prediction approach identifies the breadth of murine T(CD8+)-cell responses to vaccinia virus","volume":"24","author":"Moutaftsi","year":"2006","journal-title":"Nat. Biotechnol."},{"key":"2023020202135070300_btv123-B17","doi-asserted-by":"crossref","first-page":"e796","DOI":"10.1371\/journal.pone.0000796","article-title":"NetMHCpan, a method for quantitative predictions of peptide binding to any HLA-A and -B locus protein of known sequence","volume":"2","author":"Nielsen","year":"2007","journal-title":"PLoS One"},{"key":"2023020202135070300_btv123-B18","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1186\/1471-2105-6-132","article-title":"Generating quantitative models describing the sequence specificity of biological processes with the stabilized matrix method","volume":"6","author":"Peters","year":"2005","journal-title":"BMC Bioinformatics"},{"key":"2023020202135070300_btv123-B19","doi-asserted-by":"crossref","first-page":"e65","DOI":"10.1371\/journal.pcbi.0020065","article-title":"A community resource benchmarking predictions of peptide binding to MHC-I molecules","volume":"2","author":"Peters","year":"2006","journal-title":"PLoS Comput. Biol."},{"key":"2023020202135070300_btv123-B20","doi-asserted-by":"crossref","first-page":"739","DOI":"10.1146\/annurev.immunol.17.1.739","article-title":"Degradation of cell proteins and the generation of MHC class I-presented peptides","volume":"17","author":"Rock","year":"1999","journal-title":"Annu. Rev. Immunol."},{"key":"2023020202135070300_btv123-B21","doi-asserted-by":"crossref","first-page":"761","DOI":"10.1016\/S0092-8674(94)90462-6","article-title":"Inhibitors of the proteasome block the degradation of most cell proteins and the generation of peptides presented on MHC class I molecules","volume":"78","author":"Rock","year":"1994","journal-title":"Cell"},{"key":"2023020202135070300_btv123-B22","doi-asserted-by":"crossref","first-page":"480","DOI":"10.1038\/nature01074","article-title":"ERAAP customizes peptides for MHC class I molecules in the endoplasmic reticulum","volume":"419","author":"Serwold","year":"2002","journal-title":"Nature"},{"key":"2023020202135070300_btv123-B23","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.jim.2014.02.007","article-title":"MHC binding prediction with KernelRLSpan and its variations","volume":"406","author":"Shen","year":"2014","journal-title":"J. Immunol. Methods"},{"key":"2023020202135070300_btv123-B24","article-title":"Measurement of MHC\/peptide interactions by gel filtration","author":"Sidney","year":"2001","journal-title":"Curr. Protoc. Immunol."},{"key":"2023020202135070300_btv123-B25","doi-asserted-by":"crossref","first-page":"1491","DOI":"10.1007\/s00018-011-0657-y","article-title":"The role of the proteasome in the generation of MHC class I ligands and immune responses","volume":"68","author":"Sijts","year":"2011","journal-title":"Cell. Mol. Life Sci."},{"key":"2023020202135070300_btv123-B26","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1002\/prot.24470","article-title":"Assessment of template-free modeling in CASP10 and ROLL","volume":"82","author":"Tai","year":"2014","journal-title":"Proteins"},{"key":"2023020202135070300_btv123-B27","doi-asserted-by":"crossref","first-page":"D854","DOI":"10.1093\/nar\/gkp1004","article-title":"The immune epitope database 2.0","volume":"38","author":"Vita","year":"2010","journal-title":"Nucleic Acids Res."},{"key":"2023020202135070300_btv123-B28","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1186\/1471-2105-7-463","article-title":"SVRMHC prediction server for MHC-binding peptides","volume":"7","author":"Wan","year":"2006","journal-title":"BMC Bioinformatics"},{"key":"2023020202135070300_btv123-B29","doi-asserted-by":"crossref","first-page":"807","DOI":"10.1007\/s00262-006-0230-y","article-title":"Selective identification of HLA-DP4 binding T cell epitopes encoded by the MAGE-A gene family","volume":"56","author":"Wang","year":"2007","journal-title":"Cancer Immunol. Immunother."},{"key":"2023020202135070300_btv123-B30","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1016\/j.coi.2012.10.001","article-title":"Peptidases trimming MHC class I ligands","volume":"25","author":"Weimershaus","year":"2013","journal-title":"Curr. Opin. Immunol."},{"key":"2023020202135070300_btv123-B31","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1146\/annurev.immunol.17.1.51","article-title":"Immunodominance in major histocompatibility complex class I-restricted T lymphocyte responses","volume":"17","author":"Yewdell","year":"1999","journal-title":"Annu. Rev. Immunol."},{"key":"2023020202135070300_btv123-B32","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1093\/bioinformatics\/btn579","article-title":"Pan-specific MHC class I predictors: a benchmark of HLA class I pan-specific prediction methods","volume":"25","author":"Zhang","year":"2009","journal-title":"Bioinformatics"},{"key":"2023020202135070300_btv123-B33","doi-asserted-by":"crossref","first-page":"1293","DOI":"10.1093\/bioinformatics\/btp137","article-title":"The PickPocket method for predicting binding specificities for receptors based on receptor pocket similarities: application to MHC-peptide binding","volume":"25","author":"Zhang","year":"2009","journal-title":"Bioinformatics"},{"key":"2023020202135070300_btv123-B34","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jim.2011.09.010","article-title":"Machine learning competition in immunology\u2014prediction of HLA class I binding peptides","volume":"374","author":"Zhang","year":"2011","journal-title":"J. Immunol. Methods"},{"key":"2023020202135070300_btv123-B35","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1093\/bib\/bbr060","article-title":"Toward more accurate pan-specific MHC-peptide binding prediction: a review of current methods and tools","volume":"13","author":"Zhang","year":"2012","journal-title":"Brief. Bioinform."}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/31\/13\/2174\/49034668\/bioinformatics_31_13_2174.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/31\/13\/2174\/49034668\/bioinformatics_31_13_2174.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,2]],"date-time":"2023-02-02T03:41:00Z","timestamp":1675309260000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/31\/13\/2174\/196331"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,2,25]]},"references-count":35,"journal-issue":{"issue":"13","published-print":{"date-parts":[[2015,7,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btv123","relation":{},"ISSN":["1367-4811","1367-4803"],"issn-type":[{"value":"1367-4811","type":"electronic"},{"value":"1367-4803","type":"print"}],"subject":[],"published-other":{"date-parts":[[2015,7,1]]},"published":{"date-parts":[[2015,2,25]]}}}