{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,19]],"date-time":"2026-11-19T23:06:02Z","timestamp":1795129562454,"version":"4.10.3"},"reference-count":30,"publisher":"BMJ","issue":"1","license":[{"start":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T00:00:00Z","timestamp":1782259200000},"content-version":"unspecified","delay-in-days":23,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Ment Health"],"accepted":{"date-parts":[[2026,5,6]]},"published-print":{"date-parts":[[2026,6]]},"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language models (LLMs) are rapidly entering clinical and consumer use, yet their probabilistic outputs have delivered a variety of unsafe user responses. Difficulties in quantifying and mitigating risks posed by LLMs threaten to stall regulatory evaluation and clinical deployment of LLM-based software as a medical device (LLM-SaMD). Practical approaches are needed to extend existing medical-device regulations to LLM-SaMDs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To demonstrate how simulation can extend existing medical-device risk management frameworks for addressing LLM-SaMD-specific risks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>\n                      We implement a simulation-based methodology for estimating LLM-SaMD risk. Fourteen open-source models were evaluated on three safety-classification tasks: suicidal-ideation, therapy-request and therapy-like interaction detection. Synthetic datasets were generated by Gemini 2.5 Pro and evaluated by psychiatrists. Model false-negative rates informed estimates of P\n                      <jats:sub>1<\/jats:sub>\n                      , the likelihood that a hazard progresses to a hazardous situation, and P\n                      <jats:sub>2<\/jats:sub>\n                      , the likelihood that that situation results in harm.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Findings<\/jats:title>\n                    <jats:p>\n                      LLM success at generating synthetic datasets varied by task, with strong performance for neutral and non-therapeutic content but frequent errors in suicidal-ideation and therapy-like interactions. Performance generally improved with model size. Estimated P\n                      <jats:sub>1<\/jats:sub>\n                      values ranged from 1.1\u00d710\u207b\u2078 to 1.6\u00d710\u207b\u2074 and P\n                      <jats:sub>2<\/jats:sub>\n                      from 4.9\u00d710\u207b\u2075 to 5.1\u00d710\u207b\u00b3, spanning four orders of magnitude.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>By linking model failure modes to structured pathways to harm, simulation can extend existing medical-device risk frameworks to help address the probabilistic and context-dependent risks of LLM-SaMDs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Clinical implications<\/jats:title>\n                    <jats:p>Simulation-based risk estimation offers a practical way to characterise the risk landscape for specific LLM-SaMD, patient population and clinical context combinations.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bmjment-2026-302626","type":"journal-article","created":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T15:40:23Z","timestamp":1782315623000},"page":"e302626","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":3,"title":["Leveraging simulation to provide a practical framework for estimating the novel scope of risk of large language models in healthcare"],"prefix":"10.1136","volume":"29","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8794-1289","authenticated-orcid":false,"given":"Mark","family":"Kalinich","sequence":"first","affiliation":[{"name":"Department of Psychiatry, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1606-3917","authenticated-orcid":false,"given":"James","family":"Luccarelli","sequence":"additional","affiliation":[{"name":"Department of Psychiatry, Massachusetts General Hospital, Boston, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"suffix":"Jr","given":"John","family":"Santa Maria","sequence":"additional","affiliation":[{"name":"Parabilis Medicines, Cambridge, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Frank","family":"Moss","sequence":"additional","affiliation":[{"name":"Strategic Software Ventures, LLC, East Orleans, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5362-7937","authenticated-orcid":false,"given":"John","family":"Torous","sequence":"additional","affiliation":[{"name":"Psychiatry, Beth Israel Deaconess Medical Center, Boston, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2026,6,24]]},"reference":[{"key":"2026091806103473000_29.1.e302626.1","unstructured":"Le Jeune P , Liu J , Rossi L , et al . RealHarm: a collection of real-world language model application failures. In: Derczynski L , Novikova J , Chen M , eds. Proceedings of the The First Workshop on LLM Security (LLMSEC). Association for Computational Linguistics; 2025:87\u2013100. Available: https:\/\/aclanthology.org\/2025.llmsec-1.7\/"},{"key":"2026091806103473000_29.1.e302626.2","doi-asserted-by":"crossref","unstructured":"Weidinger L , Uesato J , Rauh M , et al . Taxonomy of risks posed by language models. FAccT \u201922: 2022 ACM Conference on Fairness, Accountability, and Transparency; Seoul Republic of Korea, 2022:214\u201329. Available: https:\/\/dl.acm.org\/doi\/proceedings\/10.1145\/3531146","DOI":"10.1145\/3531146.3533088"},{"key":"2026091806103473000_29.1.e302626.3","unstructured":"Frances A , Ramos L . Preliminary report on dangers of AI Chatbots. Psychiatr Times; 2025. Available: https:\/\/www.psychiatrictimes.com\/view\/preliminary-report-on-dangers-of-ai-chatbots [Accessed 30 Oct 2025]."},{"key":"2026091806103473000_29.1.e302626.4","unstructured":"Application of risk management to medical devices; ISO 14971. 2019."},{"key":"2026091806103473000_29.1.e302626.5","unstructured":"FDA . General controls for medical devices. 2023. Available: https:\/\/www.fda.gov\/medical-devices\/regulatory-controls\/general-controls-medical-devices [Accessed 30 Oct 2025]."},{"key":"2026091806103473000_29.1.e302626.6","unstructured":"Medical devices \u2014 Guidance on the application of ISO 14971 (ISO-24971:2020). 2020."},{"key":"2026091806103473000_29.1.e302626.7","unstructured":"Application of ISO 14971 to machine learning in artificial intelligence (TIR34971:2023). 2023. Available: https:\/\/array.aami.org\/doi\/book\/10.2345\/9781570208669"},{"key":"2026091806103473000_29.1.e302626.8","unstructured":"Software as a Medical Device (SAMD): Clinical evaluation - guidance for industry and food and drug administration staff. FDA; 2017. Available: https:\/\/www.fda.gov\/media\/100714\/download"},{"key":"2026091806103473000_29.1.e302626.9","unstructured":"GOV.UK . Regulating medical devices in the UK. 2026. Available: https:\/\/www.gov.uk\/guidance\/regulating-medical-devices-in-the-uk"},{"key":"2026091806103473000_29.1.e302626.10","unstructured":"Regulation (EU) 2017\/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001\/83\/EC, Regulation (EC) No 178\/2002 and Regulation (EC) No 1223\/2009 and repealing Council Directives 90\/385\/EEC and 93\/42\/EEC (text with EEA relevance). Vol 117. 2017. Available: http:\/\/data.europa.eu\/eli\/reg\/2017\/745\/oj [accessed 10 Apr 2026]"},{"key":"2026091806103473000_29.1.e302626.11","unstructured":"Marketing submission recommendations for a predetermined change control plan for artificial intelligence-enabled device software functions. 2025. Available: https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/marketing-submission-recommendations-predetermined-change-control-plan-artificial-intelligence [Accessed 02 Sep 2025]."},{"key":"2026091806103473000_29.1.e302626.12","unstructured":"Artificial Intelligence\/Machine Learning (AI\/ML) . Based software as a medical device (SaMD) action plan. 2021."},{"key":"2026091806103473000_29.1.e302626.13","unstructured":"Elliott A , Krishnan S , Sarich T , et al . Risks of generative artificial intelligence in health and medicine. In: Generative artificial intelligence in health and medicine: opportunities and responsibilities for transformative innovation. National Academies Press (US), 2025. Available: https:\/\/www.ncbi.nlm.nih.gov\/books\/NBK615587\/"},{"key":"2026091806103473000_29.1.e302626.14","doi-asserted-by":"crossref","DOI":"10.2196\/75849","article-title":"Harm Reduction Strategies for Thoughtful Use of Large Language Models in the Medical Domain: Perspectives for Patients and Clinicians","volume":"27","author":"Mo\u00ebll","year":"2025","journal-title":"J Med Internet Res"},{"key":"2026091806103473000_29.1.e302626.15","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-025-01670-7","article-title":"A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation","volume":"8","author":"Asgari","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2026091806103473000_29.1.e302626.16","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-03445-1"},{"key":"2026091806103473000_29.1.e302626.17","unstructured":"U.S. Food and Drug Administration . Assessing the credibility of computational modeling and simulation in medical device submissions - guidance for industry and food and drug administration staff. FDA; 2023."},{"key":"2026091806103473000_29.1.e302626.18","doi-asserted-by":"crossref","unstructured":"Morrison TM , Pathmanathan P , Adwan M , et al . Advancing Regulatory Science With Computational Modeling for Medical Devices at the FDA\u2019s Office of Science and Engineering Laboratories. Front Med 2018;5. doi:10.3389\/fmed.2018.00241","DOI":"10.3389\/fmed.2018.00241"},{"key":"2026091806103473000_29.1.e302626.19","doi-asserted-by":"crossref","unstructured":"Tabassi E . Artificial intelligence risk management framework (AI RMF 1.0). National Institute of Standards and Technology (U.S.); 2023.","DOI":"10.6028\/NIST.AI.100-1"},{"key":"2026091806103473000_29.1.e302626.20","unstructured":"Ethics and governance of artificial intelligence for health. Available: https:\/\/www.who.int\/publications\/i\/item\/9789240029200 [Accessed 06 Jan 2026]."},{"key":"2026091806103473000_29.1.e302626.21","unstructured":"National Institute of Standards and Technology (US) . Artificial intelligence risk management framework: generative artificial intelligence profile. 2024."},{"key":"2026091806103473000_29.1.e302626.22","doi-asserted-by":"crossref","first-page":"308","DOI":"10.32628\/CSEIT24106165","article-title":"Shadow Testing in Autonomous Vehicles: A Novel Approach to Validating Full Self-Driving AI Systems","volume":"10","author":"Pathuri","year":"2024","journal-title":"Int J Sci Res Comput Sci Eng Inf Technol"},{"key":"2026091806103473000_29.1.e302626.23","unstructured":"Center for Devices and Radiological Health . The least burdensome provisions: concept and principles. 2020. Available: https:\/\/www.fda.gov\/regulatory-information\/search-fda-guidance-documents\/least-burdensome-provisions-concept-and-principles [Accessed 06 Jan 2026]."},{"key":"2026091806103473000_29.1.e302626.24","doi-asserted-by":"crossref","first-page":"104854","DOI":"10.1016\/j.jbi.2025.104854","article-title":"Monitoring strategies for continuous evaluation of deployed clinical prediction models","volume":"168","author":"Kim","year":"2025","journal-title":"J Biomed Inform"},{"key":"2026091806103473000_29.1.e302626.25","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2025.13685","article-title":"Detecting and Remediating Harmful Data Shifts for the Responsible Deployment of Clinical AI Models","volume":"8","author":"Subasri","year":"2025","journal-title":"JAMA Netw Open"},{"key":"2026091806103473000_29.1.e302626.26","doi-asserted-by":"publisher","DOI":"10.1093\/biostatistics\/kxz041"},{"key":"2026091806103473000_29.1.e302626.27","doi-asserted-by":"publisher","DOI":"10.1016\/j.ymeth.2020.01.011"},{"key":"2026091806103473000_29.1.e302626.28","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1010541","article-title":"Credibility assessment of patient-specific computational modeling using patient-specific cardiac modeling as an exemplar","volume":"18","author":"Galappaththige","year":"2022","journal-title":"PLoS Comput Biol"},{"key":"2026091806103473000_29.1.e302626.29","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1007\/s38311-024-1948-x","article-title":"Tool Chain for the Virtual Validation of Autonomous Vehicles","volume":"126","author":"Alirezaei","year":"2024","journal-title":"ATZ Worldw"},{"key":"2026091806103473000_29.1.e302626.30","doi-asserted-by":"crossref","unstructured":"Moore J , Mehta A , Agnew W , et al . Characterizing Delusional Spirals through Human-LLM Chat Logs. arXiv 2026. doi:10.48550\/arXiv.2603.16567","DOI":"10.1145\/3805689.3806443"}],"container-title":["BMJ Mental Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjment-2026-302626","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,18]],"date-time":"2026-09-18T13:11:10Z","timestamp":1789737070000},"score":1,"resource":{"primary":{"URL":"https:\/\/mentalhealth.bmj.com\/lookup\/doi\/10.1136\/bmjment-2026-302626"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6]]},"references-count":30,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,6,24]]},"published-print":{"date-parts":[[2026,6]]}},"alternative-id":["10.1136\/bmjment-2026-302626"],"URL":"https:\/\/doi.org\/10.1136\/bmjment-2026-302626","relation":{},"ISSN":["2755-9734"],"issn-type":[{"value":"2755-9734","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6]]}}}