{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,2]],"date-time":"2026-03-02T12:09:35Z","timestamp":1772453375884,"version":"3.50.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1010219","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,7,6]],"date-time":"2022-07-06T00:00:00Z","timestamp":1657065600000}}],"reference-count":37,"publisher":"Public Library of Science (PLoS)","issue":"6","license":[{"start":{"date-parts":[[2022,6,23]],"date-time":"2022-06-23T00:00:00Z","timestamp":1655942400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Many different types of generative models for protein sequences have been proposed in literature. Their uses include the prediction of mutational effects, protein design and the prediction of structural properties. Neural network (NN) architectures have shown great performances, commonly attributed to the capacity to extract non-trivial higher-order interactions from the data. In this work, we analyze two different NN models and assess how close they are to simple pairwise distributions, which have been used in the past for similar problems. We present an approach for extracting pairwise models from more complex ones using an energy-based modeling framework. We show that for the tested models the extracted pairwise models can replicate the energies of the original models and are also close in performance in tasks like mutational effect prediction. In addition, we show that even simpler, factorized models often come close in performance to the original models.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1010219","type":"journal-article","created":{"date-parts":[[2022,6,23]],"date-time":"2022-06-23T13:46:03Z","timestamp":1655991963000},"page":"e1010219","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":7,"title":["Interpretable pairwise distillations for generative protein sequence models"],"prefix":"10.1371","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8941-7333","authenticated-orcid":true,"given":"Christoph","family":"Feinauer","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7153-8735","authenticated-orcid":true,"given":"Barthelemy","family":"Meynard-Piganeau","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0837-9783","authenticated-orcid":true,"given":"Carlo","family":"Lucibello","sequence":"additional","affiliation":[]}],"member":"340","published-online":{"date-parts":[[2022,6,23]]},"reference":[{"issue":"4","key":"pcbi.1010219.ref001","doi-asserted-by":"crossref","first-page":"1061","DOI":"10.1002\/prot.22934","article-title":"Learning generative models for protein fold families","volume":"79","author":"S Balakrishnan","year":"2011","journal-title":"Proteins: Structure, Function, and Bioinformatics"},{"key":"pcbi.1010219.ref002","doi-asserted-by":"crossref","unstructured":"Feinauer C, Weigt M. Context-aware prediction of pathogenicity of missense mutations involved in human disease. arXiv preprint arXiv:170107246. 2017;.","DOI":"10.1101\/103051"},{"issue":"2","key":"pcbi.1010219.ref003","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1038\/nbt.3769","article-title":"Mutation effects predicted from sequence co-variation","volume":"35","author":"TA Hopf","year":"2017","journal-title":"Nature biotechnology"},{"issue":"6502","key":"pcbi.1010219.ref004","doi-asserted-by":"crossref","first-page":"440","DOI":"10.1126\/science.aba3304","article-title":"An evolution-based model for designing chorismate mutase enzymes","volume":"369","author":"WP Russ","year":"2020","journal-title":"Science"},{"issue":"1","key":"pcbi.1010219.ref005","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41467-019-13633-0","article-title":"Deciphering protein evolution and fitness landscapes with latent space models","volume":"10","author":"X Ding","year":"2019","journal-title":"Nature communications"},{"issue":"10","key":"pcbi.1010219.ref006","doi-asserted-by":"crossref","first-page":"816","DOI":"10.1038\/s41592-018-0138-4","article-title":"Deep generative models of genetic variation capture the effects of mutations","volume":"15","author":"AJ Riesselman","year":"2018","journal-title":"Nature methods"},{"issue":"2","key":"pcbi.1010219.ref007","doi-asserted-by":"crossref","first-page":"e1008736","DOI":"10.1371\/journal.pcbi.1008736","article-title":"Generating functional protein variants with variational autoencoders","volume":"17","author":"A Hawkins-Hooker","year":"2021","journal-title":"PLoS computational biology"},{"issue":"4","key":"pcbi.1010219.ref008","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1038\/s42256-021-00310-5","article-title":"Expanding functional protein sequence spaces using generative adversarial networks","volume":"3","author":"D Repecka","year":"2021","journal-title":"Nature Machine Intelligence"},{"issue":"1","key":"pcbi.1010219.ref009","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41467-021-25756-4","article-title":"Efficient generative modeling of protein sequences using simple autoregressive models","volume":"12","author":"J Trinquier","year":"2021","journal-title":"Nature communications"},{"issue":"1","key":"pcbi.1010219.ref010","first-page":"1","article-title":"Protein design and variant prediction using autoregressive generative models","volume":"12","author":"JE Shin","year":"2021","journal-title":"Nature communications"},{"key":"pcbi.1010219.ref011","doi-asserted-by":"crossref","unstructured":"Madani A, McCann B, Naik N, Keskar NS, Anand N, Eguchi RR, et al. Progen: Language modeling for protein generation. arXiv preprint arXiv:200403497. 2020;.","DOI":"10.1101\/2020.03.07.982272"},{"key":"pcbi.1010219.ref012","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1016\/j.cbpa.2021.04.004","article-title":"Protein sequence design with deep generative models","volume":"65","author":"Z Wu","year":"2021","journal-title":"Current Opinion in Chemical Biology"},{"key":"pcbi.1010219.ref013","article-title":"The structure-fitness landscape of pairwise relations in generative sequence models","author":"D Marshall","year":"2020","journal-title":"bioRxiv"},{"key":"pcbi.1010219.ref014","unstructured":"Zamuner S, Rios PDL. Interpretable Neural Networks based classifiers for categorical inputs. arXiv preprint arXiv:210203202. 2021;."},{"issue":"0","key":"pcbi.1010219.ref015","article-title":"A tutorial on energy-based learning","volume":"1(","author":"Y LeCun","year":"2006","journal-title":"Predicting structured data"},{"issue":"7","key":"pcbi.1010219.ref016","article-title":"Distilling the knowledge in a neural network","volume":"2","author":"G Hinton","year":"2015","journal-title":"arXiv preprint arXiv:150302531"},{"key":"pcbi.1010219.ref017","doi-asserted-by":"crossref","unstructured":"Liu X, Wang X, Matwin S. Improving the interpretability of deep neural networks with knowledge distillation. In: 2018 IEEE International Conference on Data Mining Workshops (ICDMW). IEEE; 2018. p. 905\u2013912.","DOI":"10.1109\/ICDMW.2018.00132"},{"key":"pcbi.1010219.ref018","doi-asserted-by":"crossref","first-page":"e39397","DOI":"10.7554\/eLife.39397","article-title":"Learning protein constitutive motifs from sequence data","volume":"8","author":"J Tubiana","year":"2019","journal-title":"Elife"},{"issue":"7883","key":"pcbi.1010219.ref019","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1038\/s41586-021-04043-8","article-title":"Disease variant prediction with deep generative models of evolutionary data","volume":"599","author":"J Frazer","year":"2021","journal-title":"Nature"},{"key":"pcbi.1010219.ref020","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511790492","volume-title":"Biological sequence analysis: probabilistic models of proteins and nucleic acids","author":"R Durbin","year":"1998"},{"issue":"2","key":"pcbi.1010219.ref021","doi-asserted-by":"crossref","first-page":"413","DOI":"10.1534\/genetics.115.175802","article-title":"Massively parallel functional analysis of BRCA1 RING domain variants","volume":"200","author":"LM Starita","year":"2015","journal-title":"Genetics"},{"issue":"3","key":"pcbi.1010219.ref022","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1038\/nmeth.3223","article-title":"Massively parallel single-amino-acid mutagenesis","volume":"12","author":"JO Kitzman","year":"2015","journal-title":"Nature methods"},{"issue":"12","key":"pcbi.1010219.ref023","doi-asserted-by":"crossref","first-page":"957","DOI":"10.15252\/msb.20177908","article-title":"A framework for exhaustively mapping functional missense variants","volume":"13","author":"J Weile","year":"2017","journal-title":"Molecular systems biology"},{"issue":"14","key":"pcbi.1010219.ref024","doi-asserted-by":"crossref","first-page":"E1263","DOI":"10.1073\/pnas.1303309110","article-title":"Activity-enhancing mutations in an E3 ubiquitin ligase identified by high-throughput mutagenesis","volume":"110","author":"LM Starita","year":"2013","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"42","key":"pcbi.1010219.ref025","doi-asserted-by":"crossref","first-page":"16858","DOI":"10.1073\/pnas.1209751109","article-title":"A fundamental protein property, thermodynamic stability, revealed solely from large-scale measurements of protein function","volume":"109","author":"CL Araya","year":"2012","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"1","key":"pcbi.1010219.ref026","first-page":"503","article-title":"On the limited memory BFGS method for large scale optimization","volume":"45","author":"DC Liu","year":"1989","journal-title":"Mathematical programming"},{"issue":"49","key":"pcbi.1010219.ref027","doi-asserted-by":"crossref","first-page":"E1293","DOI":"10.1073\/pnas.1111471108","article-title":"Direct-coupling analysis of residue coevolution captures native contacts across many protein families","volume":"108","author":"F Morcos","year":"2011","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"7","key":"pcbi.1010219.ref028","doi-asserted-by":"crossref","first-page":"e1004182","DOI":"10.1371\/journal.pcbi.1004182","article-title":"Inferring pairwise interactions from biological data using maximum-entropy probability models","volume":"11","author":"RR Stein","year":"2015","journal-title":"PLoS computational biology"},{"key":"pcbi.1010219.ref029","first-page":"229","article-title":"Information-type measures of difference of probability distributions and indirect observation","volume":"2","author":"I Csisz\u00e1r","year":"1967","journal-title":"Studia Scientiarum Mathematicarum Hungarica"},{"key":"pcbi.1010219.ref030","unstructured":"Kingma DP, Ba J. Adam: A method for stochastic optimization. arXiv preprint arXiv:14126980. 2014;."},{"key":"pcbi.1010219.ref031","unstructured":"Burda Y, Grosse R, Salakhutdinov R. Importance weighted autoencoders. arXiv preprint arXiv:150900519. 2015;."},{"issue":"1","key":"pcbi.1010219.ref032","doi-asserted-by":"crossref","first-page":"012707","DOI":"10.1103\/PhysRevE.87.012707","article-title":"Improved contact prediction in proteins: using pseudolikelihoods to infer Potts models","volume":"87","author":"M Ekeberg","year":"2013","journal-title":"Physical Review E"},{"issue":"12","key":"pcbi.1010219.ref033","doi-asserted-by":"crossref","first-page":"124007","DOI":"10.1088\/1742-5468\/ac3a7f","article-title":"Reconstruction of pairwise interactions using energy-based models","volume":"2021","author":"C Feinauer","year":"2021","journal-title":"Journal of Statistical Mechanics: Theory and Experiment"},{"issue":"4","key":"pcbi.1010219.ref034","doi-asserted-by":"crossref","first-page":"1018","DOI":"10.1093\/molbev\/msy007","article-title":"How pairwise coevolutionary models capture the collective residue variability in proteins?","volume":"35","author":"M Figliuzzi","year":"2018","journal-title":"Molecular biology and evolution"},{"issue":"10","key":"pcbi.1010219.ref035","doi-asserted-by":"crossref","first-page":"e1003847","DOI":"10.1371\/journal.pcbi.1003847","article-title":"Improving contact prediction along three dimensions","volume":"10","author":"C Feinauer","year":"2014","journal-title":"PLoS computational biology"},{"key":"pcbi.1010219.ref036","article-title":"Language models enable zero-shot prediction of the effects of mutations on protein function","author":"J Meier","year":"2021","journal-title":"bioRxiv"},{"key":"pcbi.1010219.ref037","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781139814782","volume-title":"Analysis of boolean functions","author":"R O\u2019Donnell","year":"2014"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1010219","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2022,7,6]],"date-time":"2022-07-06T00:00:00Z","timestamp":1657065600000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010219","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,7,6]],"date-time":"2022-07-06T13:54:47Z","timestamp":1657115687000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1010219"}},"subtitle":[],"editor":[{"given":"Joanna","family":"Slusky","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2022,6,23]]},"references-count":37,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2022,6,23]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1010219","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.10.14.464358","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,23]]}}}