{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T11:18:47Z","timestamp":1790767127496,"version":"4.1.0"},"reference-count":61,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2020,5,19]],"date-time":"2020-05-19T00:00:00Z","timestamp":1589846400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DMS-1406984"],"award-info":[{"award-number":["DMS-1406984"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Viruses"],"abstract":"<jats:p>The fast replication rate and lack of repair mechanisms of human immunodeficiency virus (HIV) contribute to its high mutation frequency, with some mutations resulting in the evolution of resistance to antiretroviral therapies (ART). As such, studying HIV drug resistance allows for real-time evaluation of evolutionary mechanisms. Characterizing the biological process of drug resistance is also critically important for sustained effectiveness of ART. Investigating the link between \u201cblack box\u201d deep learning methods applied to this problem and evolutionary principles governing drug resistance has been overlooked to date. Here, we utilized publicly available HIV-1 sequence data and drug resistance assay results for 18 ART drugs to evaluate the performance of three architectures (multilayer perceptron, bidirectional recurrent neural network, and convolutional neural network) for drug resistance prediction, jointly with biological analysis. We identified convolutional neural networks as the best performing architecture and displayed a correspondence between the importance of biologically relevant features in the classifier and overall performance. Our results suggest that the high classification performance of deep learning models is indeed dependent on drug resistance mutations (DRMs). These models heavily weighted several features that are not known DRM locations, indicating the utility of model interpretability to address causal relationships in viral genotype-phenotype data.<\/jats:p>","DOI":"10.3390\/v12050560","type":"journal-article","created":{"date-parts":[[2020,5,20]],"date-time":"2020-05-20T02:48:24Z","timestamp":1589942904000},"page":"560","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":79,"title":["Drug Resistance Prediction Using Deep Learning Techniques on HIV-1 Sequence Data"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1062-1228","authenticated-orcid":false,"given":"Margaret C.","family":"Steiner","sequence":"first","affiliation":[{"name":"Computational Biology Institute, Milken Institute School of Public Health, The George Washington University, Washington, DC 20052, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1909-3596","authenticated-orcid":false,"given":"Keylie M.","family":"Gibson","sequence":"additional","affiliation":[{"name":"Computational Biology Institute, Milken Institute School of Public Health, The George Washington University, Washington, DC 20052, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0836-3389","authenticated-orcid":false,"given":"Keith A.","family":"Crandall","sequence":"additional","affiliation":[{"name":"Computational Biology Institute, Milken Institute School of Public Health, The George Washington University, Washington, DC 20052, USA"},{"name":"Department of Biostatistics and Bioinformatics, Milken Institute School of Public Health, The George Washington University, Washington, DC 20052, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,5,19]]},"reference":[{"key":"ref_1","unstructured":"Centers for Disease Control and Prevention (2018). HIV Surveillance Report, Centers for Disease Control and Prevention."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"492","DOI":"10.1097\/COH.0000000000000298","article-title":"Trends in life expectancy of HIV-positive adults on ART across the globe: Comparisons with general population HHS Public Access","volume":"11","author":"Wandeler","year":"2016","journal-title":"Curr. Opin. HIV AIDS"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Das, M., Chu, P.L., Santos, G.M., Scheer, S., Vittinghoff, E., McFarland, W., and Colfax, G.N. (2010). Decreases in community viral load are accompanied by reductions in new HIV infections in San Francisco. PLoS ONE, 5.","DOI":"10.1371\/journal.pone.0011068"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"921","DOI":"10.1056\/NEJM200003303421303","article-title":"Viral load and heterosexual transmission of human immunodeficiency virus type 1","volume":"342","author":"Quinn","year":"2000","journal-title":"N. Engl. J. Med."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1038\/nrg1246","article-title":"The causes and consequences of HIV evolution","volume":"5","author":"Rambaut","year":"2004","journal-title":"Nat. Rev. Genet."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1093\/oxfordjournals.molbev.a026118","article-title":"Parallel evolution of drug resistance in HIV: Failure of nonsynonymous\/synonymous substitution rate ratio to detect selection","volume":"16","author":"Crandall","year":"1999","journal-title":"Mol. Biol. Evol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1984","DOI":"10.1001\/jama.279.24.1984","article-title":"Antiretroviral drug resistance testing in adults with HIV infection: Implications for clinical management","volume":"279","author":"Hirsch","year":"1998","journal-title":"J. Am. Med. Assoc."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"439","DOI":"10.1097\/01.qai.0000147526.64863.53","article-title":"Comparison of the precision and sensitivity of the antivirogram and PhenoSense HIV drug susceptibility assays","volume":"38","author":"Zhang","year":"2005","journal-title":"J. Acquir. Immune Defic. Syndr."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"298","DOI":"10.1093\/nar\/gkg100","article-title":"Human immunodeficiency virus reverse transcriptase and protease sequence database","volume":"31","author":"Rhee","year":"2003","journal-title":"Nucleic Acids Res."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Bonet, I., Garc\u00eda, M.M., Saeys, Y., Van De Peer, Y., and Grau, R. (2007, January 18\u201321). Predicting human immunodeficiency virus (HIV) drug resistance using recurrent neural networks. Proceedings of the IWINAC 2007, La Manga del Mar Menor, Spain.","DOI":"10.1007\/978-3-540-73053-8_23"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1608","DOI":"10.1086\/503914","article-title":"Web Resources for HIV type 1 Genotypic-Resistance Test Interpretation","volume":"42","author":"Liu","year":"2006","journal-title":"Clin. Infect. Dis."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"4698","DOI":"10.1128\/JVI.80.10.4698-4704.2006","article-title":"A Reliable Phenotype Predictor for Human Immunodeficiency Virus Type 1 Subtype C Based on Envelope V3 Sequences","volume":"80","author":"Jensen","year":"2006","journal-title":"J. Virol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"3850","DOI":"10.1093\/nar\/gkg575","article-title":"Geno2pheno: Estimating phenotypic drug resistance from HIV-1 genotypes","volume":"31","author":"Beerenwinkel","year":"2003","journal-title":"Nucleic Acids Res."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1186\/s12859-016-1179-2","article-title":"SHIVA\u2014A web application for drug resistance and tropism testing in HIV","volume":"17","author":"Riemenschneider","year":"2016","journal-title":"BMC Bioinform."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/srep24883","article-title":"Genotypic Prediction of Co-receptor Tropism of HIV-1 Subtypes A and C","volume":"6","author":"Riemenschneider","year":"2016","journal-title":"Sci. Rep."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1946","DOI":"10.1093\/bioinformatics\/btt331","article-title":"Multilabel classification for exploiting cross-resistance information in HIV-1 drug resistance prediction","volume":"29","author":"Heider","year":"2013","journal-title":"Bioinformatics"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"8271","DOI":"10.1073\/pnas.112177799","article-title":"Diversity and complexity of HIV-1 drug resistance: A bioinformatics approach to predicting phenotype from genotype","volume":"99","author":"Beerenwinkel","year":"2002","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"653","DOI":"10.1086\/377453","article-title":"Networks Enhanced Prediction of Lopinavir Resistance from Genotype by Use of Artificial Neural Networks","volume":"188","author":"Wang","year":"2003","journal-title":"J. Infect. Dis."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"369","DOI":"10.1186\/s12859-017-1782-x","article-title":"Improving fold resistance prediction of HIV-1 against protease and reverse transcriptase inhibitors using artificial neural networks","volume":"18","author":"Bishop","year":"2017","journal-title":"BMC Bioinform."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"e02080","DOI":"10.1016\/j.heliyon.2019.e02080","article-title":"Fuzzy-multidimensional deep learning for efficient prediction of patient response to antiretroviral therapy","volume":"5","author":"Ekpenyong","year":"2019","journal-title":"Heliyon"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"878","DOI":"10.15252\/msb.20156651","article-title":"Deep learning for computational biology","volume":"12","author":"Angermueller","year":"2016","journal-title":"Mol. Syst. Biol."},{"key":"ref_22","first-page":"851","article-title":"Deep learning in bioinformatics","volume":"18","author":"Min","year":"2017","journal-title":"Brief. Bioinform."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"786","DOI":"10.21105\/joss.00786","article-title":"iml: An R package for Interpretable Machine Learning","volume":"3","author":"Molnar","year":"2018","journal-title":"J. Open Source Softw."},{"key":"ref_24","unstructured":"R Core Team (2017). R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing."},{"key":"ref_25","unstructured":"RStudio Team (2015). RStudio: Integrated Development Environment for R, R Studio, Inc."},{"key":"ref_26","unstructured":"Chollet, F., and Allaire, J.J. (2020, April 21). Keras: R Interface to Keras; Keras Team. Available online: https:\/\/keras.rstudio.com\/index.html."},{"key":"ref_27","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2015). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems [White Paper], Google Research."},{"key":"ref_28","unstructured":"Van Rossum, G., and Drake, F.L. (2009). Python 3 Reference Manual, CreateSpace."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis, Springer.","DOI":"10.1007\/978-3-319-24277-4_9"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., and M\u00fcller, M. (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinform., 12.","DOI":"10.1186\/1471-2105-12-77"},{"key":"ref_31","first-page":"1","article-title":"All models are wrong, but many are useful: Learning a variable\u2019s importance by studying an entire class of prediction models simultaneously","volume":"20","author":"Fisher","year":"2019","journal-title":"J. Mach. Learn. Res."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1093\/molbev\/msz189","article-title":"ModelTest-NG: A New and Scalable Tool for the Selection of DNA and Protein Evolutionary Models","volume":"37","author":"Darriba","year":"2019","journal-title":"Mol. Biol. Evol."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Dang, C.C., Le, Q.S., Gascuel, O., and Le, V.S. (2010). FLU, an amino acid substitution model for influenza proteins. BMC Evol. Biol., 10.","DOI":"10.1186\/1471-2148-10-99"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1312","DOI":"10.1093\/bioinformatics\/btu033","article-title":"RAxML version 8: A tool for phylogenetic analysis and post-analysis of large phylogenies","volume":"30","author":"Stamatakis","year":"2014","journal-title":"Bioinformatics"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Miller, M.A., Pfeiffer, W., and Schwartz, T. (2010, January 14). Creating the CIPRES Science Gateway for inference of large phylogenetic trees. Proceedings of the 2010 Gateway Computing Environments Workshop (GCE), New Orleans, LA, USA.","DOI":"10.1109\/GCE.2010.5676129"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"W256","DOI":"10.1093\/nar\/gkz239","article-title":"Interactive Tree Of Life (iTOL) v4: Recent updates and new developments","volume":"47","author":"Letunic","year":"2019","journal-title":"Nucleic Acids Res."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"492","DOI":"10.1080\/10635150290069913","article-title":"An approximately unbiased test of phylogenetic tree selection","volume":"51","author":"Shimodaira","year":"2002","journal-title":"Syst. Biol."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1093\/molbev\/msu300","article-title":"IQ-TREE: A fast and effective stochastic algorithm for estimating maximum-likelihood phylogenies","volume":"32","author":"Nguyen","year":"2015","journal-title":"Mol. Biol. Evol."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1086\/284325","article-title":"Phylogenies and the comparative method","volume":"125","author":"Felsenstein","year":"1985","journal-title":"Am. Nat."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"e11282","DOI":"10.7554\/eLife.11282","article-title":"Population genomics of intrapatient HIV-1 evolution","volume":"4","author":"Zanini","year":"2015","journal-title":"Elife"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.virusres.2016.09.016","article-title":"Recent advances in inferring viral diversity from high-throughput sequencing data","volume":"239","author":"Seifert","year":"2017","journal-title":"Virus Res."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"630","DOI":"10.1097\/01.aids.0000163942.93563.fd","article-title":"Clonal analyses of HIV quasispecies in patients harbouring plasma genotype with K65R mutation associated with thymidine analogue mutations or L74V substitution","volume":"19","author":"Wirden","year":"2005","journal-title":"AIDS"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"e50524","DOI":"10.7554\/eLife.50524","article-title":"Epistasis and entrenchment of drug resistance in HIV-1 subtype B","volume":"8","author":"Biswas","year":"2019","journal-title":"Elife"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"777","DOI":"10.1016\/j.ympev.2011.11.019","article-title":"The evolution of HIV: Inferences using phylogenetics","volume":"62","author":"Burton","year":"2012","journal-title":"Mol. Phylogenet. Evol."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"20170387","DOI":"10.1098\/rsif.2017.0387","article-title":"Opportunities and Obstacles for Deep Learning in Biology and Medicine","volume":"15","author":"Ching","year":"2018","journal-title":"J. R. Soc. Interface"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1038\/s41588-018-0295-5","article-title":"A primer on deep learning in genomics","volume":"51","author":"Zou","year":"2019","journal-title":"Nat. Genet."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1111\/imr.12506","article-title":"HIV antibodies for treatment of HIV infection","volume":"275","author":"Margolis","year":"2017","journal-title":"Immunol. Rev."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.inffus.2018.09.012","article-title":"Machine learning for integrating data in biology and medicine: Principles, practice, and opportunities","volume":"50","author":"Zitnik","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_49","first-page":"1","article-title":"A cross-sectional study to characterize local HIV-1 dynamics in Washington, DC using next-generation sequencing","volume":"10","author":"Gibson","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"104277","DOI":"10.1016\/j.meegid.2020.104277","article-title":"Evaluation of haplotype callers for next-generation sequencing of viruses","volume":"82","author":"Eliseev","year":"2020","journal-title":"Infect. Genet. Evol."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1093\/protein\/7.3.349","article-title":"Can three-dimensional contacts in protein structures be predicted by analysis of correlated mutations?","volume":"7","author":"Shindyalov","year":"1994","journal-title":"Protein Eng. Des. Sel."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1196","DOI":"10.1093\/molbev\/mst030","article-title":"FUBAR: A fast, unconstrained bayesian AppRoximation for inferring selection","volume":"30","author":"Murrell","year":"2013","journal-title":"Mol. Biol. Evol."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"676","DOI":"10.1093\/bioinformatics\/bti079","article-title":"HyPhy: Hypothesis testing using phylogenies","volume":"21","author":"Frost","year":"2005","journal-title":"Bioinformatics"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1016\/j.meegid.2017.09.029","article-title":"Mutation and recombination in pathogen evolution: Relevance, methods and controversies","volume":"63","author":"Arenas","year":"2018","journal-title":"Infect. Genet. Evol."},{"key":"ref_55","first-page":"53","article-title":"HIV-1 infection alters intestinal expression of antiretroviral drug transporters and enzymes","volume":"22","author":"Kis","year":"2014","journal-title":"Top. Antivir. Med."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Balasubramaniam, M., Pandhare, J., and Dash, C. (2018). Are microRNAs important players in HIV-1 infection? An update. Viruses, 10.","DOI":"10.3390\/v10030110"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"1284","DOI":"10.1021\/acs.jctc.9b00781","article-title":"Characterizing Protein-Ligand Binding Using Atomistic Simulation and Machine Learning: Application to Drug Resistance in HIV-1 Protease","volume":"16","author":"Whitfield","year":"2020","journal-title":"J. Chem. Theory Comput."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"e1006954","DOI":"10.1371\/journal.pcbi.1006954","article-title":"Estimation of the breadth of CD4bs targeting HIV antibodies by molecular modeling and machine learning","volume":"15","author":"Conti","year":"2019","journal-title":"PLoS Comput. Biol."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"1246","DOI":"10.1002\/jmv.25078","article-title":"Naturally occurring antiviral drug resistance in HIV patients who are mono-infected or co-infected with HBV or HCV in China","volume":"90","author":"Deng","year":"2018","journal-title":"J. Med. Virol."},{"key":"ref_60","first-page":"4","article-title":"Effect of HIV \/ HCV Co-Infection on the Protease Evolution of HIV-1B: A Pilot Study in a Pediatric Population","volume":"8","author":"Rojas","year":"2018","journal-title":"Sci. Rep."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1038\/d41573-020-00016-0","article-title":"Therapeutic options for the 2019 novel coronavirus (2019-nCoV)","volume":"19","author":"Li","year":"2020","journal-title":"Nat. Rev. Drug Discov."}],"container-title":["Viruses"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4915\/12\/5\/560\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:30:11Z","timestamp":1760175011000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4915\/12\/5\/560"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,19]]},"references-count":61,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2020,5]]}},"alternative-id":["v12050560"],"URL":"https:\/\/doi.org\/10.3390\/v12050560","relation":{},"ISSN":["1999-4915"],"issn-type":[{"value":"1999-4915","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,19]]}}}