{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T11:43:48Z","timestamp":1753875828009,"version":"3.41.2"},"reference-count":38,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2024,1,23]],"date-time":"2024-01-23T00:00:00Z","timestamp":1705968000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004921","name":"Shanghai Jiao Tong University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004921","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["2023YFF1205102","2020YFA0907700"],"award-info":[{"award-number":["2023YFF1205102","2020YFA0907700"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["YG2023LC03"],"award-info":[{"award-number":["YG2023LC03"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["21977068","32171242"],"award-info":[{"award-number":["21977068","32171242"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,2,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Motivation<\/jats:title>\n                  <jats:p>Proteins found in nature represent only a fraction of the vast space of possible proteins. Protein design presents an opportunity to explore and expand this protein landscape. Within protein design, protein sequence design plays a crucial role, and numerous successful methods have been developed. Notably, deep learning-based protein sequence design methods have experienced significant advancements in recent years. However, a comprehensive and systematic comparison and evaluation of these methods have been lacking, with indicators provided by different methods often inconsistent or lacking effectiveness.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>To address this gap, we have designed a diverse set of indicators that cover several important aspects, including sequence recovery, diversity, root-mean-square deviation of protein structure, secondary structure, and the distribution of polar and nonpolar amino acids. In our evaluation, we have employed an improved weighted inferiority\u2013superiority distance method to comprehensively assess the performance of eight widely used deep learning-based protein sequence design methods. Our evaluation not only provides rankings of these methods but also offers optimization suggestions by analyzing the strengths and weaknesses of each method. Furthermore, we have developed a method to select the best temperature parameter and proposed solutions for the common issue of designing sequences with consecutive repetitive amino acids, which is often encountered in protein design methods. These findings can greatly assist users in selecting suitable protein sequence design methods. Overall, our work contributes to the field of protein sequence design by providing a comprehensive evaluation system and optimization suggestions for different methods.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btae037","type":"journal-article","created":{"date-parts":[[2024,1,23]],"date-time":"2024-01-23T18:44:01Z","timestamp":1706035441000},"source":"Crossref","is-referenced-by-count":6,"title":["Multi-indicator comparative evaluation for deep learning-based protein sequence design methods"],"prefix":"10.1093","volume":"40","author":[{"given":"Jinyu","family":"Yu","sequence":"first","affiliation":[{"name":"State Key Laboratory of Microbial Metabolism, Joint International Research Laboratory of Metabolic & Developmental Sciences, Department of Bioinformatics and Biostatistics, National Experimental Teaching Center for Life Sciences and Biotechnology, School of Life Sciences and Biotechnology, Shanghai Center for Systems Biomedicine, Shanghai Jiao Tong University , Shanghai 200240, China"}]},{"given":"Junxi","family":"Mu","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Microbial Metabolism, Joint International Research Laboratory of Metabolic & Developmental Sciences, Department of Bioinformatics and Biostatistics, National Experimental Teaching Center for Life Sciences and Biotechnology, School of Life Sciences and Biotechnology, Shanghai Center for Systems Biomedicine, Shanghai Jiao Tong University , Shanghai 200240, China"}]},{"given":"Ting","family":"Wei","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Microbial Metabolism, Joint International Research Laboratory of Metabolic & Developmental Sciences, Department of Bioinformatics and Biostatistics, National Experimental Teaching Center for Life Sciences and Biotechnology, School of Life Sciences and Biotechnology, Shanghai Center for Systems Biomedicine, Shanghai Jiao Tong University , Shanghai 200240, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7496-4182","authenticated-orcid":false,"given":"Hai-Feng","family":"Chen","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Microbial Metabolism, Joint International Research Laboratory of Metabolic & Developmental Sciences, Department of Bioinformatics and Biostatistics, National Experimental Teaching Center for Life Sciences and Biotechnology, School of Life Sciences and Biotechnology, Shanghai Center for Systems Biomedicine, Shanghai Jiao Tong University , Shanghai 200240, China"}]}],"member":"286","published-online":{"date-parts":[[2024,1,23]]},"reference":[{"key":"2024020806562627900_btae037-B1","doi-asserted-by":"crossref","first-page":"119586","DOI":"10.1016\/j.jclepro.2019.119586","article-title":"A novel plithogenic TOPSIS-CRITIC model for sustainable supply chain risk management","volume":"247","author":"Abdel-Basset","year":"2020","journal-title":"J Clean Prod"},{"key":"2024020806562627900_btae037-B2","doi-asserted-by":"crossref","first-page":"746","DOI":"10.1038\/s41467-022-28313-9","article-title":"Protein sequence design with a learned potential","volume":"13","author":"Anand","year":"2022","journal-title":"Nat Commun"},{"key":"2024020806562627900_btae037-B3","doi-asserted-by":"crossref","first-page":"678","DOI":"10.1002\/pro.3588","article-title":"What has de novo protein design taught us about protein folding and biophysics?","volume":"28","author":"Baker","year":"2019","journal-title":"Protein Sci"},{"year":"2015","author":"Bowman","key":"2024020806562627900_btae037-B4"},{"key":"2024020806562627900_btae037-B5","doi-asserted-by":"crossref","first-page":"1265","DOI":"10.1038\/s41467-022-28865-w","article-title":"Improved prediction of protein-protein interactions using AlphaFold2","volume":"13","author":"Bryant","year":"2022","journal-title":"Nat Commun"},{"key":"2024020806562627900_btae037-B6","doi-asserted-by":"crossref","first-page":"btad027","DOI":"10.1093\/bioinformatics\/btad027","article-title":"PDBench: evaluating computational methods for protein-sequence design","volume":"39","author":"Castorina","year":"2023","journal-title":"Bioinformatics"},{"key":"2024020806562627900_btae037-B7","doi-asserted-by":"crossref","first-page":"259","DOI":"10.3390\/e22020259","article-title":"A novel multi-criteria decision-making model for building material supplier selection based on entropy-AHP weighted TOPSIS","volume":"22","author":"Chen","year":"2020","journal-title":"Entropy"},{"key":"2024020806562627900_btae037-B9","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1126\/science.add2187","article-title":"Robust deep learning\u2013based protein sequence design using ProteinMPNN","volume":"378","author":"Dauparas","year":"2022","journal-title":"Science"},{"key":"2024020806562627900_btae037-B10","doi-asserted-by":"crossref","first-page":"763","DOI":"10.1016\/0305-0548(94)00059-H","article-title":"Determining objective weights in multiple criteria problems: the critic method","volume":"22","author":"Diakoulaki","year":"1995","journal-title":"Comput Oper Res"},{"key":"2024020806562627900_btae037-B11","doi-asserted-by":"crossref","first-page":"238","DOI":"10.1016\/j.csbj.2022.11.014","article-title":"From sequence to function through structure: deep learning for protein design","volume":"21","author":"Ferruz","year":"2022","journal-title":"Comput Struct Biotechnol J"},{"year":"2022","author":"Gao","key":"2024020806562627900_btae037-B12"},{"year":"2022","author":"Hsu","key":"2024020806562627900_btae037-B13"},{"key":"2024020806562627900_btae037-B14","doi-asserted-by":"crossref","first-page":"btad122","DOI":"10.1093\/bioinformatics\/btad122","article-title":"Accurate and efficient protein sequence design through learning concise local environment of residues","volume":"39","author":"Huang","year":"2023","journal-title":"Bioinformatics"},{"year":"2019","author":"Ingraham","key":"2024020806562627900_btae037-B15"},{"key":"2024020806562627900_btae037-B16","doi-asserted-by":"crossref","first-page":"e3","DOI":"10.1017\/S0033583519000131","article-title":"De novo protein design, a retrospective","volume":"53","author":"Korendovych","year":"2020","journal-title":"Q Rev Biophys"},{"key":"2024020806562627900_btae037-B17","doi-asserted-by":"crossref","first-page":"398","DOI":"10.1016\/j.eswa.2016.08.064","article-title":"A state-of the-art survey & testbed of fuzzy AHP (FAHP) applications","volume":"65","author":"Kubler","year":"2016","journal-title":"Expert Syst Appl"},{"key":"2024020806562627900_btae037-B18","doi-asserted-by":"crossref","first-page":"2947","DOI":"10.1093\/bioinformatics\/btm404","article-title":"Clustal W and Clustal X version 2.0","volume":"23","author":"Larkin","year":"2007","journal-title":"Bioinformatics"},{"key":"2024020806562627900_btae037-B19","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1126\/science.ade2574","article-title":"Evolutionary-scale prediction of atomic-level protein structure with a language model","volume":"379","author":"Lin","year":"2023","journal-title":"Science"},{"key":"2024020806562627900_btae037-B20","doi-asserted-by":"crossref","first-page":"113738","DOI":"10.1016\/j.eswa.2020.113738","article-title":"A review of fuzzy AHP methods for decision-making with subjective judgements","volume":"161","author":"Liu","year":"2020","journal-title":"Expert Syst Appl"},{"key":"2024020806562627900_btae037-B21","doi-asserted-by":"crossref","first-page":"W235","DOI":"10.1093\/nar\/gkl163","article-title":"RosettaDesign server for protein design","volume":"34","author":"Liu","year":"2006","journal-title":"Nucleic Acids Res"},{"key":"2024020806562627900_btae037-B22","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1038\/s43588-022-00273-6","article-title":"Rotamer-free protein sequence design based on deep learning and self-consistency","volume":"2","author":"Liu","year":"2022","journal-title":"Nat Comput Sci"},{"key":"2024020806562627900_btae037-B23","doi-asserted-by":"crossref","first-page":"4126","DOI":"10.1016\/j.eswa.2015.01.003","article-title":"Fuzzy multiple criteria decision-making techniques and applications\u2013two decades review from 1994 to 2014","volume":"42","author":"Mardani","year":"2015","journal-title":"Expert Syst Appl"},{"key":"2024020806562627900_btae037-B24","doi-asserted-by":"crossref","first-page":"bbad217","DOI":"10.1093\/bib\/bbad217","article-title":"Improved structure-related prediction for insufficient homologous proteins using MSA enhancement and pre-trained language model","volume":"24","author":"Meng","year":"2023","journal-title":"Brief Bioinform"},{"year":"2023","author":"Mu","key":"2024020806562627900_btae037-B25"},{"key":"2024020806562627900_btae037-B26","doi-asserted-by":"crossref","first-page":"1245","DOI":"10.1021\/acs.jcim.0c00043","article-title":"DenseCPD: improving the accuracy of neural-network-based computational protein sequence design with DenseNet","volume":"60","author":"Qi","year":"2020","journal-title":"J Chem Inf Model"},{"key":"2024020806562627900_btae037-B27","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1126\/science.aan0693","article-title":"Global analysis of protein folding using massively parallel design, synthesis, and testing","volume":"357","author":"Rocklin","year":"2017","journal-title":"Science"},{"volume-title":"The Analytic Hierarchy Process: Planning, Priority, Setting, Resource Allocation","year":"1990","author":"Saaty","key":"2024020806562627900_btae037-B28"},{"volume-title":"Global Sensitivity Analysis: The Primer","year":"2008","author":"Saltelli","key":"2024020806562627900_btae037-B29"},{"key":"2024020806562627900_btae037-B30","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1016\/j.cities.2014.06.006","article-title":"Application of a hybrid Entropy\u2013McKinsey Matrix method in evaluating sustainable urbanization: a China case study","volume":"42","author":"Shen","year":"2015","journal-title":"Cities"},{"key":"2024020806562627900_btae037-B31","doi-asserted-by":"crossref","first-page":"402","DOI":"10.1016\/j.cels.2020.08.016","article-title":"Fast and flexible protein design using deep graph neural networks","volume":"11","author":"Strokach","year":"2020","journal-title":"Cell Syst"},{"key":"2024020806562627900_btae037-B32","doi-asserted-by":"crossref","DOI":"10.1201\/b11032","volume-title":"Multiple Attribute Decision Making: Methods and Applications","author":"Tzeng","year":"2011"},{"key":"2024020806562627900_btae037-B8","doi-asserted-by":"crossref","first-page":"D506","DOI":"10.1093\/nar\/gky1049","article-title":"UniProt: a worldwide hub of protein knowledge","volume":"47","author":"UniProt Consortium","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"2024020806562627900_btae037-B33","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1126\/science.abn2100","article-title":"Scaffolding protein functional sites using deep learning","volume":"377","author":"Wang","year":"2022","journal-title":"Science"},{"key":"2024020806562627900_btae037-B34","doi-asserted-by":"crossref","first-page":"167160","DOI":"10.1016\/j.jmb.2021.167160","article-title":"A brief history of de novo protein design: minimal, rational, and computational","volume":"433","author":"Woolfson","year":"2021","journal-title":"J Mol Biol"},{"key":"2024020806562627900_btae037-B35","doi-asserted-by":"crossref","first-page":"05017020","DOI":"10.1061\/(ASCE)UP.1943-5444.0000423","article-title":"Evaluating sustainable urbanization of resource-based cities based on the Mckinsey matrix: case study in China","volume":"144","author":"Yang","year":"2018","journal-title":"J Urban Plann Dev"},{"year":"2021","author":"Ying","key":"2024020806562627900_btae037-B36"},{"key":"2024020806562627900_btae037-B37","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1016\/j.knosys.2010.07.014","article-title":"An extended TOPSIS for determining weights of decision makers with interval numbers","volume":"24","author":"Yue","year":"2011","journal-title":"Knowl Based Syst"},{"key":"2024020806562627900_btae037-B38","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1016\/S0019-9958(65)90241-X","article-title":"Fuzzy sets","volume":"8","author":"Zadeh","year":"1965","journal-title":"Inf Control"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btae037\/56336089\/btae037.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/40\/2\/btae037\/56618444\/btae037.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/40\/2\/btae037\/56618444\/btae037.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,8]],"date-time":"2024-02-08T06:56:49Z","timestamp":1707375409000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/doi\/10.1093\/bioinformatics\/btae037\/7585533"}},"subtitle":[],"editor":[{"given":"Arne","family":"Elofsson","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2024,1,23]]},"references-count":38,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,2,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btae037","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"type":"print","value":"1367-4803"},{"type":"electronic","value":"1367-4811"}],"subject":[],"published-other":{"date-parts":[[2024,2,1]]},"published":{"date-parts":[[2024,1,23]]},"article-number":"btae037"}}