{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T22:45:24Z","timestamp":1790635524980,"version":"4.1.0"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T00:00:00Z","timestamp":1765756800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,1,13]],"date-time":"2026-01-13T00:00:00Z","timestamp":1768262400000},"content-version":"vor","delay-in-days":29,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Robert C. Watzke Endowment"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Autonomous AI for diabetic eye examination is among the most validated and trusted medical AI systems, supported by extensive real-world evidence demonstrating safety, efficacy, improved outcomes, increased productivity, and cost savings. Yet its adoption remains limited. In contrast, commercially available off-the-shelf generative AI models (OTSAIs) are being rapidly tested in medical settings despite a lack of such real-world validation. These models have shown strong performance on medical reasoning tasks, prompting interest in their potential for clinical deployment. We evaluated four OTSAIs\u2014GPT-4o and GPT-4o-mini (OpenAI, San Francisco, CA), Grok (xAI, San Francisco, CA), and Gemini (Google, Mountain View, CA)\u2014on a specific diagnostic task: diabetic eye examination. The OTSAIs were\n                    <jats:italic>bundled<\/jats:italic>\n                    to ensure consistency, and performance was assessed using a level 3 reference standard, the publicly available Messidor-2 dataset. GPT-4o achieved the highest area under the receiver operator characteristic curve (AUC), 0.83. Grok achieved 0.63, and AUC was not calculable for Gemini. The AUC of retina specialists on the same task was estimated at 0.94, so the emergent performance of OTSAIs does not match that of clinical experts, nor does it approach FDA endpoints for consideration as a medical device. Nevertheless, as the performance of these OTSAIs approaches theoretical limits in the future, there might be a regulatory path through task-specific licensing by State Medical Boards for specific clinical tasks. This path may be modeled after licensing for physician assistants, where trust in the bundled OTSAI, to be used in an assistive fashion, is achieved through rigorous validation for safety and efficacy according to widely accepted regulatory considerations for both patient-facing AI, as well as for SaMD processes.\n                  <\/jats:p>","DOI":"10.1038\/s41746-025-02216-7","type":"journal-article","created":{"date-parts":[[2025,12,15]],"date-time":"2025-12-15T17:24:35Z","timestamp":1765819475000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Evaluating commercial multimodal AI for diabetic eye screening and implications for an alternative regulatory pathway"],"prefix":"10.1038","volume":"9","author":[{"given":"Matthew S.","family":"Hunt","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tinglong","family":"Dai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael D.","family":"Abr\u00e0moff","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,12,15]]},"reference":[{"key":"2216_CR1","doi-asserted-by":"publisher","first-page":"AIoa2300030","DOI":"10.1056\/AIoa2300030","volume":"1","author":"K Wu","year":"2024","unstructured":"Wu, K. et al. Characterizing the clinical adoption of medical AI devices through U.S. insurance claims. NEJM AI 1, AIoa2300030 (2024).","journal-title":"NEJM AI"},{"key":"2216_CR2","first-page":"212","volume":"64","author":"J Huang","year":"2023","unstructured":"Huang, J. et al. Autonomous artificial intelligence exams are associated with higher adherence to diabetic retinopathy testing in an integrated healthcare system. Invest. Ophthalmol. Vis. Sci. 64, 212 (2023).","journal-title":"Invest. Ophthalmol. Vis. Sci."},{"key":"2216_CR3","doi-asserted-by":"crossref","unstructured":"Wolf, R. M. Autonomous artificial intelligence increases screening and follow-up for diabetic retinopathy in youth: the ACCESS randomized control trial. Nat. Commun. 15, 421 (2024).","DOI":"10.1038\/s41467-023-44676-z"},{"key":"2216_CR4","doi-asserted-by":"publisher","first-page":"196","DOI":"10.1038\/s41746-024-01197-3","volume":"7","author":"JJ Huang","year":"2024","unstructured":"Huang, J. J. et al. Autonomous artificial intelligence for diabetic eye disease increases access and health equity in underserved populations. Npj Digit. Med. 7, 196 (2024).","journal-title":"Npj Digit. Med."},{"key":"2216_CR5","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1038\/s41746-023-00931-7","volume":"6","author":"MD Abramoff","year":"2023","unstructured":"Abramoff, M. D. et al. Autonomous artificial intelligence increases real-world specialist clinic productivity in a cluster-randomized trial. Npj Digit. Med. 6, 184 (2023).","journal-title":"Npj Digit. Med."},{"key":"2216_CR6","doi-asserted-by":"publisher","first-page":"1063","DOI":"10.1001\/jamaophthalmol.2020.3190","volume":"138","author":"RM Wolf","year":"2020","unstructured":"Wolf, R. M., Channa, R., Abramoff, M. D. & Lehmann, H. P. Cost-effectiveness of autonomous point-of-care diabetic retinopathy screening for pediatric patients with diabetes. JAMA Ophthalmol. 138, 1063\u20131069 (2020).","journal-title":"JAMA Ophthalmol."},{"key":"2216_CR7","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1038\/s41746-022-00621-w","volume":"5","author":"MD Abr\u00e0moff","year":"2022","unstructured":"Abr\u00e0moff, M. D. et al. A reimbursement framework for artificial intelligence in healthcare. NPJ Digit. Med. 5, 72 (2022).","journal-title":"NPJ Digit. Med."},{"key":"2216_CR8","doi-asserted-by":"publisher","first-page":"e14","DOI":"10.1016\/j.ophtha.2021.08.023","volume":"129","author":"MD Abr\u00e0moff","year":"2022","unstructured":"Abr\u00e0moff, M. D. et al. Foundational considerations for artificial intelligence using ophthalmic images. Ophthalmology 129, e14\u2013e32 (2022).","journal-title":"Ophthalmology"},{"key":"2216_CR9","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1038\/s41746-018-0040-6","volume":"1","author":"MD Abr\u00e0moff","year":"2018","unstructured":"Abr\u00e0moff, M. D., Lavin, P. T., Birch, M., Shah, N. & Folk, J. C. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit. Med. 1, 39 (2018).","journal-title":"NPJ Digit. Med."},{"key":"2216_CR10","doi-asserted-by":"publisher","first-page":"AIpc2400083","DOI":"10.1056\/AIpc2400083","volume":"1","author":"MD Abramoff","year":"2024","unstructured":"Abramoff, M. D., Dai, T. & Zou, J. Scaling adoption of medical AI\u2014reimbursement from value-based care and fee-for-service perspectives. NEJM AI 1, AIpc2400083 (2024).","journal-title":"NEJM AI"},{"key":"2216_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-024-01389-x","volume":"7","author":"MD Abr\u00e0moff","year":"2024","unstructured":"Abr\u00e0moff, M. D. et al. Mitigation of AI adoption bias through an improved autonomous AI system for diabetic retinal disease. Npj Digit. Med. 7, 1\u201310 (2024).","journal-title":"Npj Digit. Med."},{"key":"2216_CR12","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1038\/s41586-023-06555-x","volume":"622","author":"Y Zhou","year":"2023","unstructured":"Zhou, Y. et al. A foundation model for generalizable disease detection from retinal images. Nature 622, 156\u2013163 (2023).","journal-title":"Nature"},{"key":"2216_CR13","doi-asserted-by":"publisher","first-page":"e0000198","DOI":"10.1371\/journal.pdig.0000198","volume":"2","author":"TH Kung","year":"2023","unstructured":"Kung, T. H. et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit. Health 2, e0000198 (2023).","journal-title":"PLOS Digit. Health"},{"key":"2216_CR14","unstructured":"Staff, B. ChatGPT Moderately Accurate in Interpreting Ophthalmic Images. https:\/\/www.reviewofoptometry.com\/article\/chatgpt-moderately-accurate-in-interpreting-ophthalmic-images (2025)."},{"key":"2216_CR15","doi-asserted-by":"publisher","unstructured":"Bommineni, V. L. et al. Performance of ChatGPT on the MCAT: the road to personalized and equitable premedical learning. Preprint at https:\/\/doi.org\/10.1101\/2023.03.05.23286533 (2023).","DOI":"10.1101\/2023.03.05.23286533"},{"key":"2216_CR16","doi-asserted-by":"crossref","unstructured":"Jin, Q. et al. Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine. Preprint at https:\/\/arxiv.org\/abs\/2401.08396 (2024).","DOI":"10.1038\/s41746-024-01185-7"},{"key":"2216_CR17","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1162\/106454605774270642","volume":"11","author":"M Prokopenko","year":"2005","unstructured":"Prokopenko, M. et al. Self-organizing hierarchies in sensor and communication networks. Artif. Life 11, 407\u2013426 (2005).","journal-title":"Artif. Life"},{"key":"2216_CR18","unstructured":"MAFFRE et al. Messidor-2. ADCIS. https:\/\/www.adcis.net\/en\/third-party\/messidor2\/ (2025)."},{"key":"2216_CR19","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1109\/TMI.2009.2033909","volume":"29","author":"M Niemeijer","year":"2010","unstructured":"Niemeijer, M. et al. Retinopathy online challenge: automatic detection of microaneurysms in digital color fundus photographs. IEEE Trans. Med. Imaging 29, 185\u2013195 (2010).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"2216_CR20","doi-asserted-by":"publisher","first-page":"231","DOI":"10.5566\/ias.1155","volume":"33","author":"E Decenci\u00e8re","year":"2014","unstructured":"Decenci\u00e8re, E. et al. FEEDBACK ON A PUBLICLY DISTRIBUTED IMAGE DATABASE: THE MESSIDOR DATABASE. Image Anal. Stereo. 33, 231\u2013234 (2014).","journal-title":"Image Anal. Stereo."},{"key":"2216_CR21","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1001\/jamaophthalmol.2013.1743","volume":"131","author":"MD Abr\u00e0moff","year":"2013","unstructured":"Abr\u00e0moff, M. D. et al. Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol. 131, 351 (2013).","journal-title":"JAMA Ophthalmol."},{"key":"2216_CR22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-023-00913-9","volume":"6","author":"MD Abr\u00e0moff","year":"2023","unstructured":"Abr\u00e0moff, M. D. et al. Considerations for addressing bias in artificial intelligence for health equity. Npj Digit. Med. 6, 1\u20137 (2023).","journal-title":"Npj Digit. Med."},{"key":"2216_CR23","doi-asserted-by":"publisher","first-page":"1264","DOI":"10.1016\/j.ophtha.2018.01.034","volume":"125","author":"J Krause","year":"2018","unstructured":"Krause, J. et al. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125, 1264\u20131272 (2018).","journal-title":"Ophthalmology"},{"key":"2216_CR24","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1038\/s41746-023-00929-1","volume":"6","author":"AD Saenz","year":"2023","unstructured":"Saenz, A. D., Harned, Z., Banerjee, O., Abr\u00e0moff, M. D. & Rajpurkar, P. Autonomous AI systems in the face of liability, regulations and costs. Npj Digit. Med. 6, 185 (2023).","journal-title":"Npj Digit. Med."},{"key":"2216_CR25","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-43436-9","volume":"13","author":"D Brin","year":"2023","unstructured":"Brin, D. et al. Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments. Sci. Rep. 13, 16492 (2023).","journal-title":"Sci. Rep."},{"key":"2216_CR26","doi-asserted-by":"crossref","unstructured":"Pugh, J. A. et al. Screening for diabetic retinopathy. The wide-angle retinal camera. Diabetes Care 16, 889\u2013895 (1993).","DOI":"10.2337\/diacare.16.6.889"},{"key":"2216_CR27","doi-asserted-by":"publisher","first-page":"204","DOI":"10.1016\/S0002-9394(02)01522-2","volume":"134","author":"DY Lin","year":"2002","unstructured":"Lin, D. Y., Blumenkranz, M. S., Brothers, R. J. & Grosvenor, D. M. The sensitivity and specificity of single-field nonmydriatic monochromatic digital fundus photography with remote image interpretation for diabetic retinopathy screening: a comparison with ophthalmoscopy and standardized mydriatic color photography. Am. J. Ophthalmol. 134, 204\u2013213 (2002).","journal-title":"Am. J. Ophthalmol."},{"key":"2216_CR28","unstructured":"AMA. H-480.939 Augmented Intelligence in Health Care (AMA; 2025); https:\/\/policysearch.ama-assn.org\/policyfinder\/detail\/augmented?uri=%2FAMADoc%2FHOD.xml-H-480.939.xml."},{"key":"2216_CR29","unstructured":"American Medical Association (AMA) Augmented Intelligence Development, Deployment, and Use in Health Care 2024. https:\/\/www.ama-assn.org\/system\/files\/ama-ai-principles.pdf (2025)."},{"key":"2216_CR30","unstructured":"FDA. Artificial Intelligence and Machine Learning (AI\/ML)-Enabled Medical Devices (FDA, 2025); https:\/\/www.fda.gov\/medical-devices\/software-medical-device-samd\/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices."},{"key":"2216_CR31","doi-asserted-by":"crossref","unstructured":"Abramoff, M. D. & Char, D. What do we do with physicians when autonomous AI-enabled workflow is better for patient outcomes? Am. J. Bioeth. https:\/\/www.tandfonline.com\/doi\/abs\/10.1080\/15265161.2024.2377111 (2025).","DOI":"10.1080\/15265161.2024.2377111"},{"key":"2216_CR32","doi-asserted-by":"crossref","unstructured":"Photocoagulation for diabetic macular edema. Early treatment diabetic retinopathy study report number 1. Early treatment diabetic retinopathy study research group. Arch. Ophthalmol. 103,1796\u20131806 (1985).","DOI":"10.1001\/archopht.1985.01050120030015"},{"key":"2216_CR33","unstructured":"Davis, M. D. et al. Risk factors for high-risk proliferative diabetic retinopathy and severe visual loss: early treatment diabetic retinopathy study report #18. Invest. Ophthalmol. Vis. Sci. 39, 233\u201352 (1998)."},{"key":"2216_CR34","doi-asserted-by":"publisher","first-page":"1677","DOI":"10.1016\/S0161-6420(03)00475-5","volume":"110","author":"CP Wilkinson","year":"2003","unstructured":"Wilkinson, C. P. et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology 110, 1677\u20131682 (2003).","journal-title":"Ophthalmology"},{"key":"2216_CR35","unstructured":"ISO 13485:2016. Medical devices \u2013 Quality management systems \u2013 Requirements for regulatory purposes. International Organization for Standardization. https:\/\/www.iso.org\/standard\/59752.html (accessed 2025)."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-02216-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-02216-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-02216-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T05:06:50Z","timestamp":1768367210000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-025-02216-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,15]]},"references-count":35,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["2216"],"URL":"https:\/\/doi.org\/10.1038\/s41746-025-02216-7","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,15]]},"assertion":[{"value":"1 May 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 November 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 December 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"M.D.A. reports the following conflicts of interest: patents and patent applications assigned to the University of Iowa and Digital Diagnostics relevant to the subject; Director, Consultant, shareholder, Digital Diagnostics Inc; Executive Secretary, Healthcare AI Coalition, Washington DC; Treasurer, Collaborative Community on Ophthalmic Imaging, Washington DC; member, American Academy of Ophthalmology (AAO) AI Committee; member, AI Workgroup Digital Medicine Payment Advisory Group (DMPAG) of the American Medical Association. T.D. declares no competing financial interests and a nonfinancial interest as a member of multiple study teams using LumineticsCore from Digital Diagnostics and as co-lead of Johns Hopkins University\u2019s Bloomberg Distinguished Professorship Cluster on Global Advances in Medical Artificial Intelligence. T.D. is an Editor for\n                      npj Digital Medicine\n                      and played no role in the internal review or decision to publish this article. M.S.H. reports no financial or non-financial competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"42"}}