{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T16:14:26Z","timestamp":1793376866319,"version":"4.5.0"},"reference-count":31,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2020,6,25]],"date-time":"2020-06-25T00:00:00Z","timestamp":1593043200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000054","name":"US National Cancer Institute","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000016","name":"Department of Health and Human Services","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000016","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,1,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>With the advent of primary human papillomavirus testing followed by cytology for cervical cancer screening, visual interpretation of cytology slides remains the last subjective analysis step and suffers from low sensitivity and reproducibility.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We developed a cloud-based whole-slide imaging platform with a deep-learning classifier for p16\/Ki-67 dual-stained (DS) slides trained on biopsy-based gold standards. We compared it with conventional Pap and manual DS in 3 epidemiological studies of cervical and anal precancers from Kaiser Permanente Northern California and the University of Oklahoma comprising 4253 patients. All statistical tests were 2-sided.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>In independent validation at Kaiser Permanente Northern California, artificial intelligence (AI)-based DS had lower positivity than cytology (P\u2009&amp;lt;\u2009.001) and manual DS (P\u2009&amp;lt;\u2009.001) with equal sensitivity and substantially higher specificity compared with both Pap (P\u2009&amp;lt;\u2009.001) and manual DS (P\u2009&amp;lt;\u2009.001), respectively. Compared with Pap, AI-based DS reduced referral to colposcopy by one-third (41.9% vs 60.1%, P\u2009&amp;lt;\u2009.001). At a higher cutoff, AI-based DS had similar performance to high-grade squamous intraepithelial lesions cytology, indicating a risk high enough to allow for immediate treatment. The classifier was robust, showing comparable performance in 2 cytology systems and in anal cytology.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>Automated DS evaluation removes the remaining subjective component from cervical cancer screening and delivers consistent quality for providers and patients. Moving from Pap to automated DS substantially reduces the number of colposcopies and also achieves excellent performance in a simulated fully vaccinated population. Through cloud-based implementation, this approach is globally accessible. Our results demonstrate that AI not only provides automation and objectivity but also delivers a substantial benefit for women by reduction of unnecessary colposcopies.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jnci\/djaa066","type":"journal-article","created":{"date-parts":[[2020,5,4]],"date-time":"2020-05-04T19:12:33Z","timestamp":1588619553000},"page":"72-79","source":"Crossref","is-referenced-by-count":168,"title":["Accuracy and Efficiency of Deep-Learning\u2013Based Automation of Dual Stain Cytology in Cervical Cancer Screening"],"prefix":"10.1093","volume":"113","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1251-0836","authenticated-orcid":false,"given":"Nicolas","family":"Wentzensen","sequence":"first","affiliation":[{"name":"Affiliations of authors: Division of Cancer Epidemiology and Genetics, National Cancer Institute, Bethesda, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bernd","family":"Lahrmann","sequence":"additional","affiliation":[{"name":"Steinbeis Transfer Center for Medical Systems Biology, Heidelberg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0435-7376","authenticated-orcid":false,"given":"Megan A","family":"Clarke","sequence":"additional","affiliation":[{"name":"Affiliations of authors: Division of Cancer Epidemiology and Genetics, National Cancer Institute, Bethesda, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5323-2249","authenticated-orcid":false,"given":"Walter","family":"Kinney","sequence":"additional","affiliation":[{"name":"Global Coalition Against Cervical Cancer, Arlington, VA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0628-2857","authenticated-orcid":false,"given":"Diane","family":"Tokugawa","sequence":"additional","affiliation":[{"name":"Kaiser Permanente TPMG Regional Laboratory, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nancy","family":"Poitras","sequence":"additional","affiliation":[{"name":"Kaiser Permanente TPMG Regional Laboratory, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alex","family":"Locke","sequence":"additional","affiliation":[{"name":"Kaiser Permanente TPMG Regional Laboratory, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liam","family":"Bartels","sequence":"additional","affiliation":[{"name":"Hamamatsu Tissue Imaging and Analysis Center (TIGA), BIOQUANT, University Heidelberg, Heidelberg, Germany"},{"name":"National Center of Tumor Diseases, Medical Oncology, University Hospital Heidelberg, Heidelberg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandra","family":"Krauthoff","sequence":"additional","affiliation":[{"name":"Hamamatsu Tissue Imaging and Analysis Center (TIGA), BIOQUANT, University Heidelberg, Heidelberg, Germany"},{"name":"National Center of Tumor Diseases, Medical Oncology, University Hospital Heidelberg, Heidelberg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joan","family":"Walker","sequence":"additional","affiliation":[{"name":"University of Oklahoma, Oklahoma City, OK, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rosemary","family":"Zuna","sequence":"additional","affiliation":[{"name":"University of Oklahoma, Oklahoma City, OK, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kiranjit K","family":"Grewal","sequence":"additional","affiliation":[{"name":"Kaiser Permanente TPMG Regional Laboratory, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patricia E","family":"Goldhoff","sequence":"additional","affiliation":[{"name":"Kaiser Permanente TPMG Regional Laboratory, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julie D","family":"Kingery","sequence":"additional","affiliation":[{"name":"Kaiser Permanente TPMG Regional Laboratory, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1082-6554","authenticated-orcid":false,"given":"Philip E","family":"Castle","sequence":"additional","affiliation":[{"name":"Albert Einstein College of Medicine, Bronx, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark","family":"Schiffman","sequence":"additional","affiliation":[{"name":"Affiliations of authors: Division of Cancer Epidemiology and Genetics, National Cancer Institute, Bethesda, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas S","family":"Lorey","sequence":"additional","affiliation":[{"name":"Kaiser Permanente TPMG Regional Laboratory, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Niels","family":"Grabe","sequence":"additional","affiliation":[{"name":"Steinbeis Transfer Center for Medical Systems Biology, Heidelberg, Germany"},{"name":"Hamamatsu Tissue Imaging and Analysis Center (TIGA), BIOQUANT, University Heidelberg, Heidelberg, Germany"},{"name":"National Center of Tumor Diseases, Medical Oncology, University Hospital Heidelberg, Heidelberg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2020,6,25]]},"reference":[{"key":"2021050416162330900_djaa066-B1","doi-asserted-by":"crossref","first-page":"127","DOI":"10.3322\/caac.21552","article-title":"Artificial intelligence in cancer imaging: clinical challenges and applications","author":"Bi","year":"2019","journal-title":"CA Cancer J Clin"},{"issue":"11","key":"2021050416162330900_djaa066-B2","doi-asserted-by":"crossref","first-page":"1101","DOI":"10.1001\/jama.2018.11100","article-title":"Deep learning-a technology with the potential to transform health care","volume":"320","author":"Hinton","year":"2018","journal-title":"JAMA"},{"issue":"28","key":"2021050416162330900_djaa066-B3","doi-asserted-by":"crossref","first-page":"3144","DOI":"10.1200\/JCO.2014.59.1339","article-title":"Prediction of allogeneic hematopoietic stem-cell transplantation mortality 100 days after transplantation using a machine learning algorithm: a European Group for blood and marrow transplantation acute leukemia working party retrospective data mining study","volume":"33","author":"Shouval","year":"2015","journal-title":"J Clin Oncol"},{"issue":"11","key":"2021050416162330900_djaa066-B4","doi-asserted-by":"crossref","first-page":"1107","DOI":"10.1001\/jama.2018.11029","article-title":"Clinical implications and challenges of artificial intelligence and deep learning","volume":"320","author":"Stead","year":"2018","journal-title":"JAMA"},{"issue":"10","key":"2021050416162330900_djaa066-B5","doi-asserted-by":"crossref","first-page":"2192","DOI":"10.1002\/ijc.30579","article-title":"Eurogin 2016 roadmap: how HPV knowledge is changing screening practice","volume":"140","author":"Wentzensen","year":"2017","journal-title":"Int J Cancer"},{"issue":"5","key":"2021050416162330900_djaa066-B6","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1093\/jnci\/djq562","article-title":"Human papillomavirus testing in the prevention of cervical cancer","volume":"103","author":"Schiffman","year":"2011","journal-title":"J Natl Cancer Inst"},{"issue":"7","key":"2021050416162330900_djaa066-B7","doi-asserted-by":"crossref","first-page":"674","DOI":"10.1001\/jama.2018.10897","article-title":"Screening for cervical cancer: US Preventive Services Task Force recommendation statement","volume":"320","author":"Curry","year":"2018","journal-title":"JAMA"},{"issue":"8","key":"2021050416162330900_djaa066-B8","doi-asserted-by":"crossref","DOI":"10.1093\/jnci\/dju153","article-title":"Reassurance against future risk of precancer and cancer conferred by a negative human papillomavirus test","volume":"106","author":"Gage","year":"2014","journal-title":"J Natl Cancer Inst"},{"issue":"oct13 1","key":"2021050416162330900_djaa066-B9","doi-asserted-by":"crossref","first-page":"a1754","DOI":"10.1136\/bmj.a1754","article-title":"Long term predictive values of cytology and human papillomavirus testing in cervical cancer screening: joint European cohort study","volume":"337","author":"Dillner","year":"2008","journal-title":"BMJ"},{"issue":"7","key":"2021050416162330900_djaa066-B10","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1016\/S1470-2045(11)70145-0","article-title":"Cervical cancer risk for women undergoing concurrent testing for human papillomavirus and cervical cytology: a population-based study in routine clinical practice","volume":"12","author":"Katki","year":"2011","journal-title":"Lancet Oncol"},{"key":"2021050416162330900_djaa066-B11","doi-asserted-by":"crossref","first-page":"735","DOI":"10.1002\/ijc.31261","article-title":"Eurogin roadmap 2017: triage strategies for the management of HPV-positive women in cervical screening programs","author":"Cuschieri","year":"2018","journal-title":"Int J Cancer"},{"key":"2021050416162330900_djaa066-B12","doi-asserted-by":"crossref","first-page":"S49","DOI":"10.1016\/j.jcv.2015.11.015","article-title":"Triage of HPV positive women in cervical cancer screening","volume":"76(Suppl 1","author":"Wentzensen","year":"2016","journal-title":"J Clin Virol"},{"issue":"2","key":"2021050416162330900_djaa066-B13","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1097\/LGT.0000000000000103","article-title":"Use of primary high-risk human papillomavirus testing for cervical cancer screening: interim clinical guidance","volume":"19","author":"Huh","year":"2015","journal-title":"J Low Genit Tract Dis"},{"issue":"8","key":"2021050416162330900_djaa066-B14","doi-asserted-by":"crossref","first-page":"1835","DOI":"10.1002\/ijc.28514","article-title":"Interlaboratory variation in the performance of liquid-based cytology: insights from the ATHENA trial","volume":"134","author":"Wright","year":"2014","journal-title":"Int J Cancer"},{"issue":"11","key":"2021050416162330900_djaa066-B15","doi-asserted-by":"crossref","first-page":"1500","DOI":"10.1001\/jama.285.11.1500","article-title":"Interobserver reproducibility of cervical cytologic and histologic interpretations: realistic estimates from the ASCUS-LSIL Triage Study","volume":"285","author":"Stoler","year":"2001","journal-title":"JAMA"},{"issue":"7","key":"2021050416162330900_djaa066-B16","doi-asserted-by":"crossref","first-page":"881","DOI":"10.1001\/jamainternmed.2019.0306","article-title":"Clinical evaluation of HPV screening with p16\/Ki-67 dual stain triage in a large organized cervical cancer screening program","volume":"179","author":"Wentzensen","year":"2019","journal-title":"JAMA Intern Med"},{"issue":"2","key":"2021050416162330900_djaa066-B17","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1001\/jamaoncol.2018.4270","article-title":"Five-year risk of cervical precancer following p16\/Ki-67 dual-stain triage of HPV-positive women","volume":"5","author":"Clarke","year":"2019","journal-title":"JAMA Oncol."},{"issue":"12","key":"2021050416162330900_djaa066-B18","doi-asserted-by":"crossref","first-page":"djv257","DOI":"10.1093\/jnci\/djv257","article-title":"p16\/Ki-67 dual stain cytology for detection of cervical precancer in HPV-positive women","volume":"107","author":"Wentzensen","year":"2015","journal-title":"J Natl Cancer Inst"},{"issue":"15","key":"2021050416162330900_djaa066-B19","doi-asserted-by":"crossref","first-page":"4154","DOI":"10.1158\/1078-0432.CCR-12-0270","article-title":"Performance of p16\/Ki-67 immunostaining to detect cervical cancer precursors in a colposcopy referral population","volume":"18","author":"Wentzensen","year":"2012","journal-title":"Clin Cancer Res"},{"issue":"1","key":"2021050416162330900_djaa066-B20","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/j.ygyno.2016.10.031","article-title":"Triaging HPV-positive women with p16\/Ki-67 dual-stained cytology: results from a sub-study nested into the ATHENA trial","volume":"144","author":"Wright","year":"2017","journal-title":"Gynecol Oncol"},{"issue":"2","key":"2021050416162330900_djaa066-B21","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1016\/S1470-2045(12)70529-6","article-title":"Risk of high-grade cervical intraepithelial neoplasia during follow-up in HPV-positive women according to baseline p16-INK4A results: a prospective analysis of a nested substudy of the NTCC randomised controlled trial","volume":"14","author":"Carozzi","year":"2013","journal-title":"Lancet Oncol"},{"issue":"4","key":"2021050416162330900_djaa066-B22","doi-asserted-by":"crossref","first-page":"e61441","DOI":"10.1371\/journal.pone.0061441","article-title":"Semantic focusing allows fully automated single-layer slide scanning of cervical cytology slides","volume":"8","author":"Lahrmann","year":"2013","journal-title":"PLoS One"},{"issue":"1-2","key":"2021050416162330900_djaa066-B23","first-page":"109","article-title":"A virtual microscopy system to scan, evaluate and archive biomarker enhanced cervical cytology slides","volume":"32","author":"Grabe","year":"2010","journal-title":"Cell Oncol"},{"issue":"1","key":"2021050416162330900_djaa066-B24","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1200\/JCO.2014.55.9948","article-title":"Multiple biopsies and detection of cervical cancer precursors at colposcopy","volume":"33","author":"Wentzensen","year":"2015","journal-title":"J Clin Oncol"},{"issue":"17","key":"2021050416162330900_djaa066-B25","doi-asserted-by":"crossref","first-page":"2185","DOI":"10.1097\/QAD.0b013e328359f255","article-title":"Human papillomavirus genotyping, human papillomavirus mRNA expression, and p16\/Ki-67 cytology to detect anal cancer precursors in HIV-infected MSM","volume":"26","author":"Wentzensen","year":"2012","journal-title":"Aids"},{"issue":"16","key":"2021050416162330900_djaa066-B26","doi-asserted-by":"crossref","first-page":"2114","DOI":"10.1001\/jama.287.16.2114","article-title":"The 2001 Bethesda system: terminology for reporting results of cervical cytology","volume":"287","author":"Solomon","year":"2002","journal-title":"JAMA"},{"issue":"2","key":"2021050416162330900_djaa066-B27","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1111\/j.0006-341X.2000.00345.x","article-title":"Comparisons of predictive values of binary medical diagnostic tests for paired designs","volume":"56","author":"Leisenring","year":"2000","journal-title":"Biometrics"},{"issue":"1","key":"2021050416162330900_djaa066-B28","doi-asserted-by":"crossref","first-page":"16086","DOI":"10.1038\/nrdp.2016.86","article-title":"Carcinogenic human papillomavirus infection","volume":"2","author":"Schiffman","year":"2016","journal-title":"Nat Rev Dis Primers"},{"key":"2021050416162330900_djaa066-B29","doi-asserted-by":"crossref","first-page":"A16","DOI":"10.1016\/j.vaccine.2007.11.069","article-title":"Cervical cancer screening following prophylactic human papillomavirus vaccination","volume":"26(Suppl 1","author":"Franco","year":"2008","journal-title":"Vaccine"},{"key":"2021050416162330900_djaa066-B30","doi-asserted-by":"crossref","first-page":"S1","DOI":"10.1097\/LGT.0b013e318287d329","article-title":"2012 updated consensus guidelines for the management of abnormal cervical cancer screening tests and cancer precursors","volume":"17","author":"Massad","year":"2013","journal-title":"J Low Genit Tract Dis"},{"issue":"11","key":"2021050416162330900_djaa066-B31","doi-asserted-by":"crossref","first-page":"950","DOI":"10.1002\/cncy.22055","article-title":"Cytologic patterns of cervical adenocarcinomas with emphasis on factors associated with underdiagnosis","volume":"126","author":"Conrad","year":"2018","journal-title":"Cancer Cytopathol"}],"container-title":["JNCI: Journal of the National Cancer Institute"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jnci\/advance-article-pdf\/doi\/10.1093\/jnci\/djaa066\/33424415\/djaa066.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jnci\/article-pdf\/113\/1\/72\/37792038\/djaa066.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jnci\/article-pdf\/113\/1\/72\/37792038\/djaa066.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,30]],"date-time":"2023-09-30T17:33:15Z","timestamp":1696095195000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jnci\/article\/113\/1\/72\/5862008"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,25]]},"references-count":31,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,6,25]]},"published-print":{"date-parts":[[2021,1,4]]}},"URL":"https:\/\/doi.org\/10.1093\/jnci\/djaa066","relation":{},"ISSN":["0027-8874","1460-2105"],"issn-type":[{"value":"0027-8874","type":"print"},{"value":"1460-2105","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,1]]},"published":{"date-parts":[[2020,6,25]]}}}