{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:54:32Z","timestamp":1760237672978,"version":"build-2065373602"},"reference-count":58,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2020,6,10]],"date-time":"2020-06-10T00:00:00Z","timestamp":1591747200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Viruses"],"abstract":"<jats:p>Next-generation sequencing (NGS)-based HIV drug resistance (HIVDR) assays outperform conventional Sanger sequencing in scalability, sensitivity, and quantitative detection of minority resistance variants. Thus far, HIVDR assays have been applied primarily in research but rarely in clinical settings. One main obstacle is the lack of standardized validation and performance evaluation systems that allow regulatory agencies to benchmark and accredit new assays for clinical use. By revisiting the existing principles for molecular assay validation, here we propose a new validation and performance evaluation system that helps to both qualitatively and quantitatively assess the performance of an NGS-based HIVDR assay. To accomplish this, we constructed a 70-specimen proficiency test panel that includes plasmid mixtures at known ratios, viral RNA from infectious clones, and anonymized clinical specimens. We developed assessment criteria and benchmarks for NGS-based HIVDR assays and used these to assess data from five separate MiSeq runs performed in two experienced HIVDR laboratories. This proposed platform may help to pave the way for the standardization of NGS HIVDR assay validation and performance evaluation strategies for accreditation and quality assurance purposes in both research and clinical settings.<\/jats:p>","DOI":"10.3390\/v12060627","type":"journal-article","created":{"date-parts":[[2020,6,12]],"date-time":"2020-06-12T05:02:24Z","timestamp":1591938144000},"page":"627","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Development and Application of Performance Assessment Criteria for Next-Generation Sequencing-Based HIV Drug Resistance Assays"],"prefix":"10.3390","volume":"12","author":[{"given":"Michael G.","family":"Becker","sequence":"first","affiliation":[{"name":"National HIV and Retrovirology Laboratories, National Microbiology Laboratory at JC Wilt Infectious Diseases Research Center, Public Health Agency of Canada, Winnipeg, MB R3E 3R2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dun","family":"Liang","sequence":"additional","affiliation":[{"name":"ViroDx Clinical Diagnostics Laboratory, St. Louis, MO 63017, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Breanna","family":"Cooper","sequence":"additional","affiliation":[{"name":"ViroDx Clinical Diagnostics Laboratory, St. Louis, MO 63017, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yan","family":"Le","sequence":"additional","affiliation":[{"name":"ViroDx Clinical Diagnostics Laboratory, St. Louis, MO 63017, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tracy","family":"Taylor","sequence":"additional","affiliation":[{"name":"National HIV and Retrovirology Laboratories, National Microbiology Laboratory at JC Wilt Infectious Diseases Research Center, Public Health Agency of Canada, Winnipeg, MB R3E 3R2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Emma R.","family":"Lee","sequence":"additional","affiliation":[{"name":"National HIV and Retrovirology Laboratories, National Microbiology Laboratory at JC Wilt Infectious Diseases Research Center, Public Health Agency of Canada, Winnipeg, MB R3E 3R2, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sutan","family":"Wu","sequence":"additional","affiliation":[{"name":"SutanStats, St. Louis, MO 63017, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paul","family":"Sandstrom","sequence":"additional","affiliation":[{"name":"National HIV and Retrovirology Laboratories, National Microbiology Laboratory at JC Wilt Infectious Diseases Research Center, Public Health Agency of Canada, Winnipeg, MB R3E 3R2, Canada"},{"name":"Department of Medical Microbiology and Infectious Diseases, University of Manitoba, Winnipeg, MB R3E 0J9, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5451-6294","authenticated-orcid":false,"given":"Hezhao","family":"Ji","sequence":"additional","affiliation":[{"name":"National HIV and Retrovirology Laboratories, National Microbiology Laboratory at JC Wilt Infectious Diseases Research Center, Public Health Agency of Canada, Winnipeg, MB R3E 3R2, Canada"},{"name":"Department of Medical Microbiology and Infectious Diseases, University of Manitoba, Winnipeg, MB R3E 0J9, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,6,10]]},"reference":[{"key":"ref_1","unstructured":"World Health Organization (WHO) (2020, March 10). WHO Global Action Plan on HIV Drug Resistance 2017\u20132021. Available online: https:\/\/www.who.int\/hiv\/pub\/drugresistance\/hivdr-action-plan-2017-2021\/en\/."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1093\/cid\/ciy463","article-title":"Human Immunodeficiency Virus Drug Resistance: 2018 Recommendations of the International Antiviral Society\u2013USA Panel","volume":"68","author":"Calvez","year":"2019","journal-title":"Clin. Infect. Dis."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1093\/cid\/ciy881","article-title":"Impact of next-generation sequencing defined human immunodeficiency virus pretreatment drug resistance on virological outcomes in the ANRS 12249 treatment-as-prevention trial","volume":"69","author":"Derache","year":"2019","journal-title":"Clin. Infect. Dis."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"e638","DOI":"10.1016\/S2352-3018(18)30177-2","article-title":"Clinically relevant thresholds for ultrasensitive HIV drug resistance testing: A multi-country nested case-control study","volume":"5","author":"Inzaule","year":"2018","journal-title":"Lancet HIV"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"693","DOI":"10.1086\/596736","article-title":"Low-Abundance Drug-Resistant Viral Variants in Chronically HIV-Infected, Antiretroviral Treatment-Naive Patients Significantly Impact Treatment Outcomes","volume":"199","author":"Simen","year":"2009","journal-title":"J. Infect. Dis."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e579","DOI":"10.1016\/S2352-3018(16)30119-9","article-title":"Pretreatment HIV-drug resistance in Mexico and its impact on the effectiveness of first-line antiretroviral therapy: A nationally representative 2015 WHO survey","volume":"3","author":"Ji","year":"2016","journal-title":"Lancet HIV"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s11908-014-0401-5","article-title":"Next-generation sequencing to help monitor patients infected with HIV: Ready for clinical use?","volume":"16","author":"Gibson","year":"2014","journal-title":"Curr. Infect. Dis. Rep."},{"key":"ref_8","unstructured":"WHO (2009). HIV Drug Resistance Laboratory Training Package, WHO."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1128\/CMR.00074-09","article-title":"Validation of laboratory-developed molecular assays for infectious diseases","volume":"23","author":"Burd","year":"2010","journal-title":"Clin. Microbiol. Rev."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"e00261-19","DOI":"10.1128\/JCM.00261-19","article-title":"Practical Guidance to Implementing Quality Management Systems in Public Health Laboratories Performing Next-Generation Sequencing: Personnel, Equipment, and Process Management (Phase 1)","volume":"57","author":"Hutchins","year":"2019","journal-title":"J. Clin. Microbiol."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Lee, E.R., Gao, F., Sandstrom, P., and Ji, H. (2020). External quality assessment for next-generation sequencing-based HIV drug resistance testing: Unique requirements and challenges. Viruses, 12.","DOI":"10.3390\/v12050550"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"8970","DOI":"10.1038\/s41598-019-45328-3","article-title":"A MiSeq-HyDRA platform for enhanced HIV drug resistance genotyping and surveillance","volume":"9","author":"Taylor","year":"2019","journal-title":"Sci. Rep."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"733","DOI":"10.1038\/gim.2013.92","article-title":"ACMG clinical laboratory standards for next-generation sequencing","volume":"15","author":"Rehm","year":"2013","journal-title":"Genet. Med."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1033","DOI":"10.1038\/nbt.2403","article-title":"Assuring the quality of NGS in clinical lab practice","volume":"30","author":"Gargis","year":"2012","journal-title":"Nat. Biotechnol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"743","DOI":"10.5858\/133.5.743","article-title":"Recommended principles and practices for validating clinical molecular pathology tests","volume":"133","author":"Jennings","year":"2009","journal-title":"Arch. Pathol. Lab. Med."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.jmoldx.2013.10.002","article-title":"Validation of a next-generation sequencing assay for clinical molecular oncology","volume":"16","author":"Cottrell","year":"2014","journal-title":"J. Mol. Diagn."},{"key":"ref_17","unstructured":"World Health Organization (WHO) (2020, April 20). WHO Manual for Organizing a National External Quality Assessment Programme for Health Laboratories and Other Testing Sites. Available online: https:\/\/www.who.int\/hiv\/pub\/toolkits\/manual-external-quality-assessment-testing\/en\/."},{"key":"ref_18","unstructured":"Persing, D.H. (2016). Practices of Sequencing Quality Assurance. Molecular Microbiology: Diagnostic Principles and Practice, John Wiley & Sons."},{"key":"ref_19","unstructured":"(2014). CLSI document MM09A2: Nucleic Acid Sequencing Methods in Diagnostic Laboratory Medicine. Approved Guideline, Clinical and Laboratory Standards Institute. [2nd ed.]."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1670","DOI":"10.1373\/clinchem.2011.168641","article-title":"Proficiency testing\/external quality assessment: Current challenges and future directions","volume":"57","author":"Miller","year":"2011","journal-title":"Clin. Chem."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"305","DOI":"10.4155\/bio.11.311","article-title":"Validation and verification of measurement methods in clinical chemistry","volume":"4","author":"Theodorsson","year":"2012","journal-title":"Bioanalysis"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"e25193","DOI":"10.1002\/jia2.25193","article-title":"Bioinformatic data processing pipelines in support of next-generation sequencing-based HIV drug resistance testing: The Winnipeg Consensus","volume":"21","author":"Ji","year":"2018","journal-title":"J. Int. AIDS Soc."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1097\/COH.0000000000000534","article-title":"HIV subtype diversity worldwide","volume":"14","author":"Bbosa","year":"2019","journal-title":"Curr. Opin. HIV AIDS"},{"key":"ref_24","unstructured":"(2012). CLSI Document EP17-A2: Evaluation of Detection Capability for Clinical Laboratory Measurement Procedures, Clinical and Laboratory Standards Institute. [2nd ed.]."},{"key":"ref_25","first-page":"662","article-title":"Pre-existing Minority Drug-Resistant HIV-1 Variants, Adherence, and Risk of Antiretroviral Treatment Failure","volume":"201","author":"Paredes","year":"2010","journal-title":"J. Infect. Dis."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1093\/biomet\/35.1-2.191","article-title":"A table for the calculation of working probits and weights in probit analysis","volume":"35","author":"Finney","year":"1948","journal-title":"Biometrika"},{"key":"ref_27","unstructured":"(2003). CLSI Document EP06-A: Evaluation of the Linearity of Quantitative Measurement Procedures: A Statistical Approach, Clinical and Laboratory Standards Institute. [1st ed.]."},{"key":"ref_28","unstructured":"(2005). CLSI document EP05-A2: Evaluation of Precision Performance of Quantitative Measurement Methods. Approved Guideline, Clinical and Laboratory Standards Institute. [2nd ed.]."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1186\/s12985-016-0627-y","article-title":"Analytical characteristics and comparative evaluation of Aptima HIV-1 Quant Dx assay with Ampliprep\/COBAS TaqMan HIV-1 test v2.0","volume":"13","author":"Hatzakis","year":"2016","journal-title":"Virol. J."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1016\/j.jcv.2016.02.002","article-title":"Aptima HIV-1 Quant Dx-A fully automated assay for both diagnosis and quantification of HIV-1","volume":"77","author":"Nair","year":"2016","journal-title":"J. Clin. Virol."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Ji, H., Sandstrom, P., Paredes, R., Harrigan, P.R., Brumme, C.J., Avila-Rios, S., and Kantor, R. (2020). Are we ready for NGS HIV drug resistance testing? The second Winnipeg symposium. Viruses, 12.","DOI":"10.3390\/v12060586"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1016\/j.jviromet.2014.04.007","article-title":"An international multicenter study on HIV-1 drug resistance testing by 454 ultra-deep pyrosequencing","volume":"204","author":"Simen","year":"2014","journal-title":"J. Virol. Methods"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"St John, E.P., Simen, B.B., Turenchalk, G.S., Braverman, M.S., Abbate, I., Aerssens, J., Bouchez, O., Gabriel, C., Izopet, J., and Meixenberger, K. (2016). A follow-up of the multicenter collaborative study on HIV-1 drug resistance and tropism testing using 454 ultra deep pyrosequencing. PLoS ONE, 11.","DOI":"10.1371\/journal.pone.0146687"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1634","DOI":"10.1038\/s41598-020-58544-z","article-title":"Performance comparison of next generation sequencing analysis pipelines for HIV-1 drug resistance testing","volume":"10","author":"Lee","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1038\/nature24621","article-title":"A communal catalogue reveals Earth\u2019s multiscale microbial diversity","volume":"551","author":"Thompson","year":"2017","journal-title":"Nature"},{"key":"ref_36","unstructured":"WHO (2020, May 27). World Health Organization Global Strategy for the Surveillance and Monitoring or HIV Drug Resistance. Available online: https:\/\/www.who.int\/hiv\/pub\/drugresistance\/drug_resistance_strategy\/en\/."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1001\/jama.2018.21167","article-title":"HIV viral load and transmissibility of HIV infection undetectable equals untransmittable","volume":"321","author":"Eisinger","year":"2019","journal-title":"J. Am. Med. Assoc."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1038\/nm0696-625","article-title":"HIV viral load markers in clinical practice","volume":"2","author":"Saag","year":"1996","journal-title":"Nat. Med."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"S829","DOI":"10.1093\/infdis\/jix397","article-title":"Next-Generation Human Immunodeficiency Virus Sequencing for Patient Management and Drug Resistance Surveillance","volume":"216","author":"Edgil","year":"2017","journal-title":"J. Infect. Dis."},{"key":"ref_40","unstructured":"(2006). CLSI document EP15-A2: User Verification of Performance for Precision and Trueness. Approved Guideline, Clinical and Laboratory Standards Institute. [2nd ed.]."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1038\/gim.2015.30","article-title":"Standards and guidelines for the interpretation of sequence variants: A joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology","volume":"17","author":"Richards","year":"2015","journal-title":"Genet. Med."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Gonzalez, C., Gondola, J., Ortiz, A.Y., Castillo, J.A., Pascale, J.M., and Martinez, A.A. (2018). Barcoding analysis of HIV drug resistance mutations using Oxford Nanopore MinION (ONT) sequencing. bioRxiv, 240077.","DOI":"10.1101\/240077"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Dessilly, G., Goeminne, L., Vandenbroucke, A.T., Dufrasne, F.E., Martin, A., and Kabamba-Mukabi, B. (2018). First evaluation of the next-generation sequencing platform for the detection of HIV-1 drug resistance mutations in Belgium. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0209561"},{"key":"ref_44","first-page":"S292","article-title":"Next Generation Sequencing and Bioinformatics Methodologies for Infectious Disease Research and Public Health: Approaches, Applications, and Considerations for Development of Laboratory Capacity","volume":"221","author":"Melendrez","year":"2019","journal-title":"J. Infect. Dis."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"675","DOI":"10.2116\/analsci.25.675","article-title":"Absolute quantification of plasmid DNA by real-time PCR with genomic DNA as external standard and its application to a biodistribution study of an HIV DNA vaccine","volume":"25","author":"Fu","year":"2009","journal-title":"Anal. Sci."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1186\/s12985-015-0280-x","article-title":"A general method to eliminate laboratory induced recombinants during massive, parallel sequencing of cDNA library","volume":"12","author":"Waugh","year":"2015","journal-title":"Virol. J."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"7171","DOI":"10.1128\/JVI.74.15.7171-7178.2000","article-title":"Structural Determinants of Murine Leukemia Virus Reverse Transcriptase That Affect the Frequency of Template Switching","volume":"74","author":"Svarovskaia","year":"2000","journal-title":"J. Virol."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Martin, S., and Leggett, R.M. (2019). Alvis: A tool for contig and read ALignment VISualisation and chimera detection. bioRxiv, 663401.","DOI":"10.1101\/663401"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"581","DOI":"10.1038\/nmeth.3869","article-title":"DADA2: High-resolution sample inference from Illumina amplicon data","volume":"13","author":"Callahan","year":"2016","journal-title":"Nat. Methods"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1322","DOI":"10.1093\/jac\/dkq139","article-title":"Low-frequency HIV-1 drug resistance mutations can be clinically significant but must be interpreted with caution","volume":"65","author":"Johnson","year":"2010","journal-title":"J. Antimicrob. Chemother."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Kugelman, J.R., Wiley, M.R., Nagle, E.R., Reyes, D., Pfeffer, B.P., Kuhn, J.H., Sanchez-Lockhart, M., and Palacios, G.F. (2017). Error baseline rates of five sample preparation methods used to characterize RNA virus populations. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0171333"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1371\/annotation\/ae27d48b-025f-47ce-8427-4af59f821ad7","article-title":"Analytical and Clinical Performance of the CDC Real Time RT-PCR Assay for Detection and Typing of Dengue Virus","volume":"7","author":"Santiago","year":"2013","journal-title":"PLoS Negl. Trop. Dis."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"2868","DOI":"10.1128\/JCM.06752-11","article-title":"Critical Analysis of Rhinovirus RNA Load Quantification by Real-Time Reverse Transcription-PCR","volume":"50","author":"Schibler","year":"2012","journal-title":"J. Clin. Microbiol."},{"key":"ref_54","unstructured":"WHO (2020, May 27). WHO Information for the Molecular Detection of Influenza Viruses. Available online: https:\/\/www.who.int\/influenza\/gisrs_laboratory\/molecular_diagnosis\/en\/."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"1667","DOI":"10.1093\/cid\/cit125","article-title":"Clinical Implications of HIV-1 Minority Variants","volume":"56","author":"Li","year":"2013","journal-title":"Clin. Infect. Dis."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"1584","DOI":"10.1093\/infdis\/jiz650","article-title":"Low-Abundance Drug-Resistant HIV-1 Variants in Antiretroviral Drug-Naive Individuals: A Systematic Review of Detection Methods, Prevalence, and Clinical Impact","volume":"221","author":"Mbunkah","year":"2019","journal-title":"J. Infect. Dis."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Kou, R., Lam, H., Duan, H., Ye, L., Jongkam, N., Chen, W., Zhang, S., and Li, S. (2016). Benefits and Challenges with Applying Unique Molecular Identifiers in Next Generation Sequencing to Detect Low Frequency Mutations. PLoS ONE, 11.","DOI":"10.1371\/journal.pone.0146638"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"658","DOI":"10.1089\/aid.2014.0031","article-title":"Primer ID informs next-generation sequencing platforms and reveals preexisting drug resistance mutations in the HIV-1 reverse transcriptase coding domain","volume":"31","author":"Keys","year":"2015","journal-title":"AIDS Res. Hum. Retrovir."}],"container-title":["Viruses"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4915\/12\/6\/627\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:37:33Z","timestamp":1760175453000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4915\/12\/6\/627"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,10]]},"references-count":58,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2020,6]]}},"alternative-id":["v12060627"],"URL":"https:\/\/doi.org\/10.3390\/v12060627","relation":{},"ISSN":["1999-4915"],"issn-type":[{"type":"electronic","value":"1999-4915"}],"subject":[],"published":{"date-parts":[[2020,6,10]]}}}