{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,21]],"date-time":"2026-02-21T02:25:05Z","timestamp":1771640705765,"version":"3.50.1"},"reference-count":15,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2024,8,8]],"date-time":"2024-08-08T00:00:00Z","timestamp":1723075200000},"content-version":"vor","delay-in-days":7,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000268","name":"Biotechnology and Biological Sciences Research Council","doi-asserted-by":"publisher","award":["BB\/T018496\/1"],"award-info":[{"award-number":["BB\/T018496\/1"]}],"id":[{"id":"10.13039\/501100000268","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,8,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>Despite an increase in protein modelling accuracy following the development of AlphaFold2, there remains an accuracy gap between predicted and observed model quality assessment (MQA) scores. In CASP15, variations in AlphaFold2 model accuracy prediction were noticed for quaternary models of very similar observed quality. In this study, we compare plDDT and pTM to their observed counterparts the local distance difference test (lDDT) and TM-score for both tertiary and quaternary models to examine whether reliability is retained across the scoring range under normal modelling conditions and in situations where AlphaFold2 functionality is customized. We also explore plDDT and pTM ranking accuracy in comparison with the published independent MQA programmes ModFOLD9 and ModFOLDdock.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>plDDT was found to be an accurate descriptor of tertiary model quality compared to observed lDDT-C\u03b1 scores (Pearson r\u2009=\u20090.97), and achieved a ranking agreement true positive rate (TPR) of 0.34 with observed scores, which ModFOLD9 could not improve. However, quaternary structure accuracy was reduced (plDDT r\u2009=\u20090.67, pTM r\u2009=\u20090.70) and significant overprediction was seen with both scores for some lower quality models. Additionally, ModFOLDdock was able to improve upon AF2-Multimer model ranking compared to TM-score (TPR 0.34) and oligo-lDDT score (TPR 0.43). Finally, evidence is presented for increased variability in plDDT and pTM when using custom template recycling, which is more pronounced for quaternary structures.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Availability and implementation<\/jats:title>\n                  <jats:p>The ModFOLD9 and ModFOLDdock quality assessment servers are available at https:\/\/www.reading.ac.uk\/bioinf\/ModFOLD\/ and https:\/\/www.reading.ac.uk\/bioinf\/ModFOLDdock\/, respectively. A docker image is available at https:\/\/hub.docker.com\/r\/mcguffin\/multifold.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btae491","type":"journal-article","created":{"date-parts":[[2024,8,8]],"date-time":"2024-08-08T23:56:18Z","timestamp":1723161378000},"source":"Crossref","is-referenced-by-count":17,"title":["Benchmarking of AlphaFold2 accuracy self-estimates as indicators of empirical model quality and ranking: a comparison with independent model quality assessment programmes"],"prefix":"10.1093","volume":"40","author":[{"given":"Nicholas S","family":"Edmunds","sequence":"first","affiliation":[{"name":"School of Biological Sciences, University of Reading , Whiteknights , Reading, RG6 6EX, United Kingdom"}]},{"given":"Ahmet G","family":"Genc","sequence":"additional","affiliation":[{"name":"School of Biological Sciences, University of Reading , Whiteknights , Reading, RG6 6EX, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4501-4767","authenticated-orcid":false,"given":"Liam J","family":"McGuffin","sequence":"additional","affiliation":[{"name":"School of Biological Sciences, University of Reading , Whiteknights , Reading, RG6 6EX, United Kingdom"}]}],"member":"286","published-online":{"date-parts":[[2024,8,8]]},"reference":[{"key":"2024081404512749400_btae491-B1","doi-asserted-by":"crossref","first-page":"vbad078","DOI":"10.1093\/bioadv\/vbad078","article-title":"Improvement of protein tertiary and quaternary structure predictions using the ReFOLD refinement method and the AlphaFold2 recycling process","volume":"3","author":"Adiyaman","year":"2023","journal-title":"Bioinform Adv"},{"key":"2024081404512749400_btae491-B2","author":"Evans","year":"2022"},{"key":"2024081404512749400_btae491-B3","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1038\/s41586-021-03819-2","article-title":"Highly accurate protein structure prediction with AlphaFold","volume":"596","author":"Jumper","year":"2021","journal-title":"Nature"},{"key":"2024081404512749400_btae491-B4","doi-asserted-by":"crossref","first-page":"2722","DOI":"10.1093\/bioinformatics\/btt473","article-title":"lDDT: a local superposition-free score for comparing protein structures and models using distance difference tests","volume":"29","author":"Mariani","year":"2013","journal-title":"Bioinformatics"},{"key":"2024081404512749400_btae491-B5","doi-asserted-by":"crossref","first-page":"168531","DOI":"10.1016\/j.jmb.2024.168531","article-title":"ModFOLD9: a web server for independent estimates of 3D protein model quality","author":"McGuffin","year":"2024","journal-title":"J Molecul Biol"},{"key":"2024081404512749400_btae491-B6","doi-asserted-by":"crossref","first-page":"W274","DOI":"10.1093\/nar\/gkad297","article-title":"Prediction of protein structures, functions and interactions using the IntFOLD7, MultiFOLD and ModFOLDdock servers","volume":"51","author":"McGuffin","year":"2023","journal-title":"Nucleic Acids Res"},{"key":"2024081404512749400_btae491-B7","doi-asserted-by":"crossref","first-page":"679","DOI":"10.1038\/s41592-022-01488-1","article-title":"ColabFold: making protein folding accessible to all","volume":"19","author":"Mirdita","year":"2022","journal-title":"Nat Methods"},{"key":"2024081404512749400_btae491-B8","doi-asserted-by":"crossref","first-page":"238101","DOI":"10.1103\/PhysRevLett.129.238101","article-title":"State-of-the-art estimation of protein model accuracy using AlphaFold","volume":"129","author":"Roney","year":"2022","journal-title":"Phys Rev Lett"},{"key":"2024081404512749400_btae491-B9","doi-asserted-by":"crossref","first-page":"1385","DOI":"10.1016\/j.str.2022.08.004","article-title":"Assessing PDB macromolecular crystal structure confidence at the individual amino acid residue level","volume":"30","author":"Shao","year":"2022","journal-title":"Structure"},{"key":"2024081404512749400_btae491-B10","author":"Takei","year":"2022"},{"key":"2024081404512749400_btae491-B11","doi-asserted-by":"crossref","first-page":"1376","DOI":"10.1038\/s41592-022-01645-6","article-title":"Improved AlphaFold modeling with implicit experimental information","volume":"19","author":"Terwilliger","year":"2022","journal-title":"Nat Methods"},{"key":"2024081404512749400_btae491-B12","doi-asserted-by":"crossref","first-page":"590","DOI":"10.1038\/s41586-021-03828-1","article-title":"Highly accurate protein structure prediction for the human proteome","volume":"596","author":"Tunyasuvunakool","year":"2021","journal-title":"Nature"},{"key":"2024081404512749400_btae491-B13","doi-asserted-by":"crossref","first-page":"D439","DOI":"10.1093\/nar\/gkab1061","article-title":"AlphaFold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models","volume":"50","author":"Varadi","year":"2022","journal-title":"Nucleic Acids Res"},{"key":"2024081404512749400_btae491-B14","doi-asserted-by":"crossref","DOI":"10.1093\/bioinformatics\/btad573","article-title":"AFsample: improving multimer prediction with AlphaFold using massive sampling","volume":"39","author":"Wallner","year":"2023","journal-title":"Bioinformatics"},{"key":"2024081404512749400_btae491-B15","doi-asserted-by":"crossref","first-page":"702","DOI":"10.1002\/prot.20264","article-title":"Scoring function for automated assessment of protein structure template quality","volume":"57","author":"Zhang","year":"2004","journal-title":"Proteins"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btae491\/58776964\/btae491.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/40\/8\/btae491\/58813746\/btae491.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/40\/8\/btae491\/58813746\/btae491.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,14]],"date-time":"2024-08-14T04:51:44Z","timestamp":1723611104000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/doi\/10.1093\/bioinformatics\/btae491\/7730100"}},"subtitle":[],"editor":[{"given":"Lenore","family":"Cowen","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2024,8]]},"references-count":15,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2024,8,2]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btae491","relation":{},"ISSN":["1367-4811"],"issn-type":[{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,8]]},"published":{"date-parts":[[2024,8]]},"article-number":"btae491"}}