{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T14:30:29Z","timestamp":1780497029288,"version":"3.54.1"},"reference-count":24,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T00:00:00Z","timestamp":1689120000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T00:00:00Z","timestamp":1689120000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000054","name":"U.S. Department of Health & Human Services | NIH | National Cancer Institute","doi-asserted-by":"publisher","award":["P30 CA008748"],"award-info":[{"award-number":["P30 CA008748"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000054","name":"U.S. Department of Health & Human Services | NIH | National Cancer Institute","doi-asserted-by":"publisher","award":["P30 CA008748"],"award-info":[{"award-number":["P30 CA008748"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100005976","name":"Harry J. Lloyd Charitable Trust","doi-asserted-by":"publisher","award":["n\/a"],"award-info":[{"award-number":["n\/a"]}],"id":[{"id":"10.13039\/100005976","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100005976","name":"Harry J. Lloyd Charitable Trust","doi-asserted-by":"publisher","award":["n\/a"],"award-info":[{"award-number":["n\/a"]}],"id":[{"id":"10.13039\/100005976","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006108","name":"U.S. Department of Health & Human Services | NIH | National Center for Advancing Translational Sciences","doi-asserted-by":"publisher","award":["2KL2-TR-2385"],"award-info":[{"award-number":["2KL2-TR-2385"]}],"id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]},{"name":"U.S. Department of Health & Human Services | NIH | National Cancer Institute"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The use of artificial intelligence (AI) has the potential to improve the assessment of lesions suspicious of melanoma, but few clinical studies have been conducted. We validated the accuracy of an open-source, non-commercial AI algorithm for melanoma diagnosis and assessed its potential impact on dermatologist decision-making. We conducted a prospective, observational clinical study to assess the diagnostic accuracy of the AI algorithm (ADAE) in predicting melanoma from dermoscopy skin lesion images. The primary aim was to assess the reliability of ADAE\u2019s sensitivity at a predefined threshold of 95%. Patients who had consented for a skin biopsy to exclude melanoma were eligible. Dermatologists also estimated the probability of melanoma and indicated management choices before and after real-time exposure to ADAE scores. All lesions underwent biopsy. Four hundred thirty-five participants were enrolled and contributed 603 lesions (95 melanomas). Participants had a mean age of 59 years, 54% were female, and 96% were White individuals. At the predetermined 95% sensitivity threshold, ADAE had a sensitivity of 96.8% (95% CI: 91.1\u201398.9%) and specificity of 37.4% (95% CI: 33.3\u201341.7%). The dermatologists\u2019 ability to assess melanoma risk significantly improved after ADAE exposure (AUC 0.7798 vs. 0.8161, <jats:italic>p<\/jats:italic>\u2009=\u20090.042). Post-ADAE dermatologist decisions also had equivalent or higher net benefit compared to biopsying all lesions. We validated the accuracy of an open-source melanoma AI algorithm and showed its theoretical potential for improving dermatology experts\u2019 ability to evaluate lesions suspicious of melanoma. Larger randomized trials are needed to fully evaluate the potential of adopting this AI algorithm into clinical workflows.<\/jats:p>","DOI":"10.1038\/s41746-023-00872-1","type":"journal-article","created":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T04:02:11Z","timestamp":1689134531000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":63,"title":["Prospective validation of dermoscopy-based open-source artificial intelligence for melanoma diagnosis (PROVE-AI study)"],"prefix":"10.1038","volume":"6","author":[{"given":"Michael A.","family":"Marchetti","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8616-3542","authenticated-orcid":false,"given":"Emily A.","family":"Cowen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6745-0386","authenticated-orcid":false,"given":"Nicholas R.","family":"Kurtansky","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2397-4954","authenticated-orcid":false,"given":"Jochen","family":"Weber","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Megan","family":"Dauscher","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"DeFazio","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liang","family":"Deng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen W.","family":"Dusza","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Helen","family":"Haliasos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Allan C.","family":"Halpern","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sharif","family":"Hosein","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7652-8512","authenticated-orcid":false,"given":"Zaeem H.","family":"Nazir","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6068-0114","authenticated-orcid":false,"given":"Ashfaq A.","family":"Marghoob","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elizabeth A.","family":"Quigley","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Trina","family":"Salvador","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0639-2677","authenticated-orcid":false,"given":"Veronica M.","family":"Rotemberg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,7,12]]},"reference":[{"key":"872_CR1","doi-asserted-by":"publisher","first-page":"1167","DOI":"10.1001\/jamadermatol.2019.1514","volume":"155","author":"KC Nelson","year":"2019","unstructured":"Nelson, K. C., Swetter, S. M., Saboda, K., Chen, S. C. & Curiel-Lewandrowski, C. Evaluation of the number-needed-to-biopsy metric for the diagnosis of cutaneous melanoma: a systematic review and meta-analysis. JAMA Dermatol. 155, 1167\u20131174 (2019).","journal-title":"JAMA Dermatol."},{"key":"872_CR2","doi-asserted-by":"publisher","first-page":"983","DOI":"10.1016\/j.jaad.2020.03.121","volume":"83","author":"L Fried","year":"2020","unstructured":"Fried, L. et al. Technological advances for the detection of melanoma: Advances in diagnostic techniques. J. Am. Acad. Dermatol. 83, 983\u2013992 (2020).","journal-title":"J. Am. Acad. Dermatol."},{"key":"872_CR3","doi-asserted-by":"publisher","first-page":"996","DOI":"10.1016\/j.jaad.2020.03.122","volume":"83","author":"L Fried","year":"2020","unstructured":"Fried, L. et al. Technological advances for the detection of melanoma: Advances in molecular techniques. J. Am. Acad. Dermatol. 83, 996\u20131004 (2020).","journal-title":"J. Am. Acad. Dermatol."},{"key":"872_CR4","doi-asserted-by":"publisher","first-page":"202","DOI":"10.1016\/j.ejca.2021.06.049","volume":"156","author":"S Haggenm\u00fcller","year":"2021","unstructured":"Haggenm\u00fcller, S. et al. Skin cancer classification via convolutional neural networks: systematic review of studies involving human experts. Eur. J. Cancer 156, 202\u2013216 (2021).","journal-title":"Eur. J. Cancer"},{"key":"872_CR5","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1001\/jamadermatol.2021.4915","volume":"158","author":"R Daneshjou","year":"2022","unstructured":"Daneshjou, R. et al. Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology: CLEAR Derm Consensus Guidelines From the International Skin Imaging Collaboration Artificial Intelligence Working Group. JAMA Dermatol. 158, 90\u201396 (2022).","journal-title":"JAMA Dermatol."},{"key":"872_CR6","doi-asserted-by":"publisher","first-page":"1362","DOI":"10.1001\/jamadermatol.2021.3129","volume":"157","author":"R Daneshjou","year":"2021","unstructured":"Daneshjou, R., Smith, M. P., Sun, M. D., Rotemberg, V. & Zou, J. Lack of Transparency and Potential Bias in Artificial Intelligence Data Sets and Algorithms: A Scoping Review. JAMA Dermatol. 157, 1362\u20131369 (2021).","journal-title":"JAMA Dermatol."},{"key":"872_CR7","doi-asserted-by":"publisher","first-page":"e330","DOI":"10.1016\/S2589-7500(22)00021-8","volume":"4","author":"M Combalia","year":"2022","unstructured":"Combalia, M. et al. Validation of artificial intelligence prediction models for skin cancer diagnosis using dermoscopy images: the 2019 International Skin Imaging Collaboration Grand Challenge. Lancet Digit Health 4, e330\u2013e339 (2022).","journal-title":"Lancet Digit Health"},{"key":"872_CR8","doi-asserted-by":"publisher","first-page":"270","DOI":"10.1016\/j.jaad.2017.08.016","volume":"78","author":"MA Marchetti","year":"2018","unstructured":"Marchetti, M. A. et al. Results of the 2016 International Skin Imaging Collaboration International Symposium on Biomedical Imaging challenge: Comparison of the accuracy of computer algorithms to dermatologists for the diagnosis of melanoma from dermoscopic images. J. Am. Acad. Dermatol. 78, 270\u2013277.e1 (2018).","journal-title":"J. Am. Acad. Dermatol."},{"key":"872_CR9","doi-asserted-by":"publisher","first-page":"622","DOI":"10.1016\/j.jaad.2019.07.016","volume":"82","author":"MA Marchetti","year":"2020","unstructured":"Marchetti, M. A. et al. Computer algorithms show potential for improving dermatologists' accuracy to diagnose cutaneous melanoma: Results of the International Skin Imaging Collaboration 2017. J. Am. Acad. Dermatol. 82, 622\u2013627 (2020).","journal-title":"J. Am. Acad. Dermatol."},{"key":"872_CR10","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-021-00815-z","volume":"8","author":"V Rotemberg","year":"2021","unstructured":"Rotemberg, V. et al. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Sci. Data 8, 34 (2021).","journal-title":"Sci. Data"},{"key":"872_CR11","doi-asserted-by":"publisher","first-page":"938","DOI":"10.1016\/S1470-2045(19)30333-X","volume":"20","author":"P Tschandl","year":"2019","unstructured":"Tschandl, P. et al. Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study. Lancet Oncol. 20, 938\u2013947 (2019).","journal-title":"Lancet Oncol."},{"key":"872_CR12","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-019-1466-7","volume":"17","author":"B Van Calster","year":"2019","unstructured":"Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L. & Steyerberg, E. W. Calibration: the Achilles heel of predictive analytics. BMC Med. 17, 230 (2019).","journal-title":"BMC Med."},{"key":"872_CR13","doi-asserted-by":"publisher","first-page":"558","DOI":"10.5858\/arpa.2012-0198-RA","volume":"137","author":"RL Schmidt","year":"2013","unstructured":"Schmidt, R. L. & Factor, R. E. Understanding sources of bias in diagnostic accuracy studies. Arch. Pathol. Lab Med. 137, 558\u2013565 (2013).","journal-title":"Arch. Pathol. Lab Med"},{"key":"872_CR14","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-019-08987-4","volume":"10","author":"S Lapuschkin","year":"2019","unstructured":"Lapuschkin, S. et al. Unmasking Clever Hans predictors and assessing what machines really learn. Nat. Commun. 10, 1096 (2019).","journal-title":"Nat. Commun."},{"key":"872_CR15","doi-asserted-by":"publisher","first-page":"2353","DOI":"10.1016\/j.jid.2022.02.003","volume":"142","author":"SS Han","year":"2022","unstructured":"Han, S. S. et al. Evaluation of Artificial Intelligence-Assisted Diagnosis of Skin Neoplasms: A Single-Center, Paralleled, Unmasked, Randomized Controlled Trial. J. investigative Dermatol. 142, 2353\u20132362.e2 (2022).","journal-title":"J. investigative Dermatol."},{"key":"872_CR16","first-page":"i6","volume":"352","author":"AJ Vickers","year":"2016","unstructured":"Vickers, A. J., Van Calster, B. & Steyerberg, E. W. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ (Clin. Res. ed.) 352, i6 (2016).","journal-title":"BMJ (Clin. Res. ed.)"},{"key":"872_CR17","doi-asserted-by":"publisher","first-page":"2534","DOI":"10.1200\/JCO.2015.65.5654","volume":"34","author":"KF Kerr","year":"2016","unstructured":"Kerr, K. F., Brown, M. D., Zhu, K. & Janes, H. Assessing the Clinical Impact of Risk Prediction Models With Decision Curves: Guidance for Correct Interpretation and Appropriate Use. J. Clin. Oncol. : Off. J. Am. Soc. Clin. Oncol. 34, 2534\u20132540 (2016).","journal-title":"J. Clin. Oncol. : Off. J. Am. Soc. Clin. Oncol."},{"key":"872_CR18","doi-asserted-by":"publisher","first-page":"409","DOI":"10.1001\/jama.2015.37","volume":"313","author":"M Fitzgerald","year":"2015","unstructured":"Fitzgerald, M., Saville, B. R. & Lewis, R. J. Decision curve analysis. Jama 313, 409\u2013410 (2015).","journal-title":"Jama"},{"key":"872_CR19","unstructured":"SIIM-ISIC Melanoma Classification. Accessed November 9, 2022. https:\/\/www.kaggle.com\/c\/siim-isic-melanoma-classification."},{"key":"872_CR20","unstructured":"Grandmaster Series \u2013 How to Build a World-Class ML Model for Melanoma Detection. Accessed November 9, 2022. https:\/\/developer.nvidia.com\/blog\/grandmaster-series-how-to-build-a-world-class-ml-model-for-melanoma-detection\/."},{"key":"872_CR21","doi-asserted-by":"publisher","unstructured":"Szegedy C., Ioffe S., Vanhoucke V., Alemi A. Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. Proceedings of the AAAI Conference on Artificial Intelligence. 31 https:\/\/doi.org\/10.1609\/aaai.v31i1.11231 (2017)","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"872_CR22","doi-asserted-by":"publisher","first-page":"100864","DOI":"10.1016\/j.mex.2020.100864","volume":"7","author":"N Gessert","year":"2020","unstructured":"Gessert, N., Nielsen, M., Shaikh, M., Werner, R. & Schlaefer, A. Skin lesion classification using ensembles of multi-resolution EfficientNets with meta data. MethodsX 7, 100864 (2020).","journal-title":"MethodsX"},{"key":"872_CR23","doi-asserted-by":"publisher","unstructured":"Identifying Melanoma Images using EfficientNet Ensemble: Winning Solution to the SIIM-ISIC Melanoma Classification Challenge. Accessed November 9, 2022. https:\/\/doi.org\/10.48550\/arXiv.2010.05351.","DOI":"10.48550\/arXiv.2010.05351"},{"key":"872_CR24","unstructured":"Tan, M. & Le, Q. V. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ArXiv. 2019;abs\/1905.11946."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00872-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00872-1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00872-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T04:04:33Z","timestamp":1689134673000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-023-00872-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,12]]},"references-count":24,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["872"],"URL":"https:\/\/doi.org\/10.1038\/s41746-023-00872-1","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,12]]},"assertion":[{"value":"17 February 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 June 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 July 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"V.R. is an expert advisor for Inhabit Brands. A.H. consults for Canfield Scientific Inc. and Janssen Research and Development, has ownership\/equity interests in HCW Health LLC, SKIP Derm LLC, and has a fiduciary role\/position and intellectual rights in SKIP Derm LLC. A.M. reports receiving honorarium from Canfield Scientific Inc. All other authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"127"}}