{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T16:11:02Z","timestamp":1773677462319,"version":"3.50.1"},"reference-count":53,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T00:00:00Z","timestamp":1770249600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T00:00:00Z","timestamp":1773619200000},"content-version":"vor","delay-in-days":39,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Advances in artificial intelligence (AI) and machine learning (ML) have led to a surge in AI\/ML-enabled medical devices, posing new challenges for regulators because best practices for developing, testing, and monitoring these devices are still emerging. Consequently, there is a critical need for up-to-date data analyses of the regulatory landscape to inform policy-making. However, such analyses have historically relied upon manual annotation efforts because regulatory documents are unstructured, complex, multi-modal, and filled with jargon. Efforts to automate annotation using simple natural language processing methods have achieved limited success, as they lack the reasoning needed to interpret regulatory materials. Recent progress in large language models (LLMs) presents an unprecedented opportunity to unlock information embedded in regulatory documents. This work conducts the first wide-ranging validation study of LLMs for scaling data analyses in the field of medical device regulatory science. Evaluating LLM outputs using expert manual annotations and \u201cLLM-as-a-judge,\u201d we find that LLMs can accurately extract attributes spanning pre- and post-market settings, with accuracy rates often reaching 80% or higher. We then demonstrate how LLMs can scale up analyses in three applications: (1) monitoring device validation practices, (2) coding medical device reports, and (3) identifying potential risk factors for post-market adverse events.<\/jats:p>","DOI":"10.1038\/s41746-026-02353-7","type":"journal-article","created":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T17:53:59Z","timestamp":1770314039000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Scaling medical device regulatory science using large language models"],"prefix":"10.1038","volume":"9","author":[{"given":"Hanyang","family":"Li","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiao","family":"He","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Adarsh","family":"Subbaswamy","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Patrick","family":"Vossler","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexej","family":"Gossmann","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karandeep","family":"Singh","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jean","family":"Feng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2026,2,5]]},"reference":[{"key":"2353_CR1","unstructured":"U.S. Food and Drug Administration. Artificial intelligence-enabled medical devices. https:\/\/www.fda.gov\/medical-devices\/software-medical-device-samd\/artificial-intelligence-enabled-medical-devices (2025)."},{"key":"2353_CR2","unstructured":"U.S. Food and Drug Administration. Proposed regulatory framework for modifications to artificial intelligence\/machine learning (AI\/ML)-based software as a medical device (SaMD): discussion paper and request for feedback. (2019)."},{"key":"2353_CR3","unstructured":"U.S. Food and Drug Administration. Artificial intelligence\/machine learning-based software as a medical device action plan. (2021)."},{"key":"2353_CR4","unstructured":"U.S. Food and Drug Administration. Marketing submission recommendations for a predetermined change control plan for artificial intelligence\/machine learning (AI\/ML)-enabled device software functions. https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/marketing-submission-recommendations-predetermined-change-control-plan-artificial (2024)."},{"key":"2353_CR5","unstructured":"U.S. Food and Drug Administration. Artificial intelligence-enabled device software functions: lifecycle management and marketing submission recommendations. https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/artificial-intelligence-enabled-device-software-functions-lifecycle-management-and-marketing (2025)."},{"key":"2353_CR6","doi-asserted-by":"crossref","unstructured":"Wu, E. et al. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nat. Med. 27, 582\u2013584 (2021).","DOI":"10.1038\/s41591-021-01312-x"},{"key":"2353_CR7","doi-asserted-by":"publisher","first-page":"328","DOI":"10.1038\/s41746-025-01717-9","volume":"8","author":"B Babic","year":"2025","unstructured":"Babic, B., Glenn Cohen, I., Stern, A. D., Li, Y. & Ouellet, M. A general framework for governing marketed AI\/ML medical devices. NPJ Digit. Med. 8, 328 (2025).","journal-title":"NPJ Digit. Med."},{"key":"2353_CR8","doi-asserted-by":"publisher","first-page":"1227","DOI":"10.1093\/jamia\/ocad065","volume":"30","author":"D Lyell","year":"2023","unstructured":"Lyell, D., Wang, Y., Coiera, E. & Magrabi, F. \u201cMore than algorithms: an analysis of safety events involving ML-enabled medical devices reported to the FDA. J. Am. Med. Inform. Assoc. 30, 1227\u20131236 (2023).","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"2353_CR9","doi-asserted-by":"publisher","first-page":"e618","DOI":"10.1016\/S2589-7500(23)00126-7","volume":"5","author":"UJ Muehlematter","year":"2023","unstructured":"Muehlematter, U. J., Bluethgen, C. & Vokinger, K. N. FDA-cleared artificial intelligence and machine learning-based medical devices and their 510(k) predicate networks. Lancet Digit. Health 5, e618\u2013e626 (2023).","journal-title":"Lancet Digit. Health"},{"key":"2353_CR10","doi-asserted-by":"publisher","first-page":"118","DOI":"10.1038\/s41746-020-00324-0","volume":"3","author":"S Benjamens","year":"2020","unstructured":"Benjamens, S., Dhunnoo, P. & Mesk\u00f3, B. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. NPJ Digit. Med. 3, 118 (2020).","journal-title":"NPJ Digit. Med."},{"key":"2353_CR11","doi-asserted-by":"publisher","unstructured":"Lee, B. et al. Development and commercialization pathways of AI medical devices in the United States: implications for safety and regulatory oversight. NEJM AI 2 https:\/\/doi.org\/10.1056\/AIra2500061 (2025).","DOI":"10.1056\/AIra2500061"},{"key":"2353_CR12","doi-asserted-by":"publisher","first-page":"388","DOI":"10.1038\/s41746-025-01800-1","volume":"8","author":"R Singh","year":"2025","unstructured":"Singh, R., Bapna, M., Diab, A. R., Ruiz, E. S. & Lotter, W. How AI is used in FDA-authorized medical devices: a taxonomy across 1,016 authorizations. NPJ Digit. Med. 8, 388 (2025).","journal-title":"NPJ Digit. Med."},{"key":"2353_CR13","doi-asserted-by":"crossref","unstructured":"Warraich, H. J., Tazbaz, T. & Califf, R. FDA perspective on the regulation of artificial intelligence in health care and biomedicine. JAMA 333, 241\u2013247 (2024).","DOI":"10.1001\/jama.2024.21451"},{"key":"2353_CR14","unstructured":"U.S. Food and Drug Administration. Artificial intelligence and machine learning (AI\/ML)-enabled medical devices. https:\/\/www.fda.gov\/medical-devices\/software-medical-device-samd\/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices (2025)."},{"key":"2353_CR15","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1001\/jama.2022.22974","volume":"329","author":"AO Everhart","year":"2023","unstructured":"Everhart, A. O., Sen, S., Stern, A. D., Zhu, Y. & Karaca-Mandic, P. Association between regulatory submission characteristics and recalls of medical devices receiving 510(k) clearance. JAMA 329, 144\u2013156 (2023).","journal-title":"JAMA"},{"key":"2353_CR16","unstructured":"Zhu, Z., Everhart, A., Karaca-Mandic, P. & Sen, S. Using NLP to extract predicate history from medical device approvals. In International Conference on Information Systems (Association for Information Systems, 2020)."},{"key":"2353_CR17","doi-asserted-by":"publisher","first-page":"1217","DOI":"10.1001\/jamainternmed.2021.3942","volume":"181","author":"C Lalani","year":"2021","unstructured":"Lalani, C., Kunwar, E. M., Kinard, M., Dhruva, S. S. & Redberg, R. F. \u201cReporting of death in US Food and Drug Administration medical device adverse event reports in categories other than death. JAMA Intern. Med. 181, 1217\u20131223 (2021).","journal-title":"JAMA Intern. Med."},{"key":"2353_CR18","unstructured":"FDA. Design considerations for pivotal clinical investigations for medical devices. (2013)."},{"key":"2353_CR19","unstructured":"Center for Devices and Radiological Health. The 510(k) program: evaluating substantial equivalence in premarket notifications [510(k)]. U.S. Food and Drug Administration. https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/510k-program-evaluating-substantial-equivalence-premarket-notifications-510k (2025)."},{"key":"2353_CR20","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1016\/j.clinthera.2024.10.004","volume":"47","author":"M Mishali","year":"2025","unstructured":"Mishali, M., Sheffer, N., Mishali, O. & Negev, M. Understanding variation among medical device reporting sources: a study of the MAUDE database. Clin. Ther. 47, 76\u201381 (2025).","journal-title":"Clin. Ther."},{"key":"2353_CR21","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-024-11885-1","volume":"24","author":"SJ Choi","year":"2024","unstructured":"Choi, S. J. et al. The application study of harmonization code in medical device adverse event reporting. BMC Health Serv. Res. 24, 1402 (2024).","journal-title":"BMC Health Serv. Res."},{"key":"2353_CR22","doi-asserted-by":"crossref","unstructured":"Ryan, M., Myer, A. M. & Yustein, A. S. Reporting device-associated death events to the FDA. JAMA Intern. Med. 182, 462 (2022).","DOI":"10.1001\/jamainternmed.2021.8509"},{"key":"2353_CR23","doi-asserted-by":"publisher","first-page":"e195","DOI":"10.1016\/S2589-7500(20)30292-2","volume":"3","author":"UJ Muehlematter","year":"2021","unstructured":"Muehlematter, U. J., Daniore, P. & Vokinger, K. N. Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015-20): a comparative analysis. Lancet Digit. Health 3, e195\u2013e203 (2021).","journal-title":"Lancet Digit. Health"},{"key":"2353_CR24","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-023-01986-5","volume":"47","author":"BJ Miller","year":"2023","unstructured":"Miller, B. J., Blanks, W. & Yagi, B. The 510(k) third party review program: promise and potential. J. Med. Syst. 47, 93 (2023).","journal-title":"J. Med. Syst."},{"key":"2353_CR25","doi-asserted-by":"publisher","first-page":"e0283442","DOI":"10.1371\/journal.pone.0283442","volume":"18","author":"C Lefkovich","year":"2023","unstructured":"Lefkovich, C. & Rothenberg, S. Identification of predicate creep under the 510(k) process: a case study of a robotic surgical device. PLoS ONE 18, e0283442 (2023).","journal-title":"PLoS ONE"},{"key":"2353_CR26","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1001\/jamainternmed.2021.6626","volume":"182","author":"KT Kadakia","year":"2022","unstructured":"Kadakia, K. T., Beckman, A. L., Ross, J. S. & Krumholz, H. M. Renewing the call for reforms to medical device safety-the case of Penumbra. JAMA Intern. Med. 182, 59\u201365 (2022).","journal-title":"JAMA Intern. Med."},{"key":"2353_CR27","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1001\/jama.2022.23279","volume":"329","author":"KT Kadakia","year":"2023","unstructured":"Kadakia, K. T., Dhruva, S. S., Caraballo, C., Ross, J. S. & Krumholz, H. M. Use of recalled devices in new device authorizations under the US food and drug administration\u2019s 510(k) pathway and risk of subsequent recalls. JAMA 329, 136\u2013143 (2023).","journal-title":"JAMA"},{"key":"2353_CR28","doi-asserted-by":"crossref","unstructured":"Boussina, A. et al. Large language models for more efficient reporting of hospital quality measures. NEJM AI1, 10.1056\/aics2400420 (2024).","DOI":"10.1056\/AIcs2400420"},{"key":"2353_CR29","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-53081-z","volume":"15","author":"Q Jin","year":"2024","unstructured":"Jin, Q. et al. Matching patients to clinical trials with large language models. Nat. Commun. 15, 9074 (2024).","journal-title":"Nat. Commun."},{"key":"2353_CR30","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1038\/s41591-024-03434-4","volume":"31","author":"K Zhang","year":"2025","unstructured":"Zhang, K. et al. Artificial intelligence in drug development. Nat. Med. 31, 45\u201359 (2025).","journal-title":"Nat. Med."},{"key":"2353_CR31","doi-asserted-by":"publisher","first-page":"e2428276","DOI":"10.1001\/jamanetworkopen.2024.28276","volume":"7","author":"ME Matheny","year":"2024","unstructured":"Matheny, M. E. et al. Enhancing postmarketing surveillance of medical products with large language models. JAMA Netw. Open 7, e2428276 (2024).","journal-title":"JAMA Netw. Open"},{"key":"2353_CR32","unstructured":"Zhang, Y. et al. Siren\u2019s song in the AI ocean: a survey on hallucination in large language models. Comput. Linguist. Assoc. Comput. Linguist. 1\u201346 (2025)."},{"key":"2353_CR33","doi-asserted-by":"crossref","unstructured":"de Hond, A. et al. From text to treatment: the crucial role of validation for generative large language models in health care. Lancet. Digit. Health 6, e441\u2013e443 (2024).","DOI":"10.1016\/S2589-7500(24)00111-0"},{"key":"2353_CR34","doi-asserted-by":"crossref","unstructured":"Ensign, L. G. & Cohen, K. B. A primer to the structure, content and linkage of the FDA\u2019s manufacturer and user facility device experience (MAUDE) files. EGEMS 5, 12 (2017).","DOI":"10.13063\/egems.1253"},{"key":"2353_CR35","unstructured":"Center for Devices and Radiological Health. MDR adverse event codes. U.S. Food and Drug Administration. https:\/\/www.fda.gov\/medical-devices\/mandatory-reporting-requirements-manufacturers-importers-and-device-user-facilities\/mdr-adverse-event-codes (2025)."},{"key":"2353_CR36","unstructured":"Vossler, P. et al. LLMs Judging LLMs: A Simplex Perspective. International Conference on Artificial Intelligence and Statistics. In press. http:\/\/arxiv.org\/abs\/2505.21972 (2026)."},{"key":"2353_CR37","unstructured":"Zheng, L. et al. Judging LLM-as-a-judge with MT-bench and Chatbot arena. Neural Inf. Process. Syst. abs\/2306.05685 (2023)."},{"key":"2353_CR38","doi-asserted-by":"publisher","first-page":"e0258153","DOI":"10.1371\/journal.pone.0258153","volume":"16","author":"DB Pai","year":"2021","unstructured":"Pai, D. B. Mapping the genealogy of medical device predicates in the United States. PLoS ONE 16, e0258153 (2021).","journal-title":"PLoS ONE"},{"key":"2353_CR39","unstructured":"Office of the Commissioner. FDA launches agency-wide AI tool to optimize performance for the American People. U.S. Food and Drug Administration. https:\/\/www.fda.gov\/news-events\/press-announcements\/fda-launches-agency-wide-ai-tool-optimize-performance-american-people (2025)."},{"key":"2353_CR40","doi-asserted-by":"publisher","first-page":"30266","DOI":"10.1073\/pnas.2001238117","volume":"117","author":"S Wang","year":"2020","unstructured":"Wang, S., McCormick, T. H. & Leek, J. T. \u201cMethods for correcting inference based on outcomes predicted by machine learning. Proc. Natl. Acad. Sci. USA. 117, 30266\u201330275 (2020).","journal-title":"Proc. Natl. Acad. Sci. USA."},{"key":"2353_CR41","doi-asserted-by":"publisher","first-page":"669","DOI":"10.1126\/science.adi6000","volume":"382","author":"AN Angelopoulos","year":"2023","unstructured":"Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I. & Zrnic, T. Prediction-powered inference. Science 382, 669\u2013674 (2023).","journal-title":"Science"},{"key":"2353_CR42","doi-asserted-by":"publisher","first-page":"e2200174","DOI":"10.1200\/CCI.22.00174","volume":"7","author":"A Sondhi","year":"2023","unstructured":"Sondhi, A., Rich, A. S., Wang, S. & Leek, J. T. Postprediction inference for clinical characteristics extracted with machine learning on electronic health records. JCO Clin. Cancer Inform. 7, e2200174 (2023).","journal-title":"JCO Clin. Cancer Inform."},{"key":"2353_CR43","doi-asserted-by":"publisher","first-page":"334","DOI":"10.1038\/s41746-024-01275-6","volume":"7","author":"A Subbaswamy","year":"2024","unstructured":"Subbaswamy, A. et al. A data-driven framework for identifying patient subgroups on which an AI\/machine learning model may underperform. NPJ Digit. Med. 7, 334 (2024).","journal-title":"NPJ Digit. Med."},{"key":"2353_CR44","first-page":"181","volume":"238","author":"J Feng","year":"2024","unstructured":"Feng, J. et al. Is this model reliable for everyone? Testing for strong calibration. Int. Conf. Artif. Intell. Stat. 238, 181\u2013189 (2024).","journal-title":"Int. Conf. Artif. Intell. Stat."},{"key":"2353_CR45","unstructured":"Singh, H. et al. Who experiences large model decay and why?\u2019 A hierarchical framework for diagnosing heterogeneous performance drift. In Forty-second International Conference on Machine Learning (2025)."},{"key":"2353_CR46","unstructured":"Singh, H., Xia, F., Subbaswamy, A., Gossmann, A. & Feng, J. A hierarchical decomposition for explaining ML performance discrepancies. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024)."},{"key":"2353_CR47","doi-asserted-by":"publisher","unstructured":"Feng, J., Xia, F., Singh, K. & Pirracchio, R. Not all clinical AI monitoring systems are created equal: review and recommendations. NEJM AI 2 https:\/\/doi.org\/10.1056\/AIra2400657 (2025).","DOI":"10.1056\/AIra2400657"},{"key":"2353_CR48","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41746-022-00611-y","volume":"5","author":"J Feng","year":"2022","unstructured":"Feng, J. et al. Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare. npj Digit. Med. 5, 1\u20139 (2022).","journal-title":"npj Digit. Med."},{"key":"2353_CR49","doi-asserted-by":"publisher","first-page":"e250369","DOI":"10.1148\/ryai.250369","volume":"7","author":"AR Diab","year":"2025","unstructured":"Diab, A. R. & Lotter, W. Distinguishing between rigor and transparency in FDA marketing authorization of AI-enabled medical devices. Radiol. Artif. Intell. 7, e250369 (2025).","journal-title":"Radiol. Artif. Intell."},{"key":"2353_CR50","doi-asserted-by":"publisher","first-page":"170","DOI":"10.1038\/s41746-021-00542-0","volume":"4","author":"RJ Desai","year":"2021","unstructured":"Desai, R. J. et al. Broadening the reach of the FDA sentinel system: a roadmap for integrating electronic health record data in a causal analysis framework. NPJ Digit. Med. 4, 170 (2021).","journal-title":"NPJ Digit. Med."},{"key":"2353_CR51","unstructured":"Tesseract. Linux J. (2007)."},{"key":"2353_CR52","doi-asserted-by":"crossref","unstructured":"Kojima, T., Gu, S., Reid, M., Matsuo, Y. & Iwasawa, Y. Large language models are zero-shot reasoners. In Proc. 36th International Conference on Neural Information Processing Systems 22199\u201322213 (2022).","DOI":"10.52202\/068431-1613"},{"key":"2353_CR53","doi-asserted-by":"crossref","unstructured":"Dong, Q. et al. A survey on in-context learning. In Proc. 2024 Conference on Empirical Methods in Natural Language Processing 1107\u20131128 (2024).","DOI":"10.18653\/v1\/2024.emnlp-main.64"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02353-7","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02353-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02353-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T14:39:26Z","timestamp":1773671966000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-026-02353-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,5]]},"references-count":53,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["2353"],"URL":"https:\/\/doi.org\/10.1038\/s41746-026-02353-7","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,5]]},"assertion":[{"value":"15 September 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 January 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 February 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"221"}}