{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,23]],"date-time":"2026-02-23T22:36:54Z","timestamp":1771886214301,"version":"3.50.1"},"reference-count":35,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2023,4,18]],"date-time":"2023-04-18T00:00:00Z","timestamp":1681776000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/501100000925","name":"NHMRC","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Centre for Research Excellence","award":["APP1134919"],"award-info":[{"award-number":["APP1134919"]}]},{"DOI":"10.13039\/501100001230","name":"Macquarie University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100001230","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,6,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To examine the real-world safety problems involving machine learning (ML)-enabled medical devices.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>We analyzed 266 safety events involving approved ML medical devices reported to the US FDA\u2019s MAUDE program between 2015 and October 2021. Events were reviewed against an existing framework for safety problems with Health IT to identify whether a reported problem was due to the ML device (device problem) or its use, and key contributors to the problem. Consequences of events were also classified.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Events described hazards with potential to harm (66%), actual harm (16%), consequences for healthcare delivery (9%), near misses that would have led to harm if not for intervention (4%), no harm or consequences (3%), and complaints (2%). While most events involved device problems (93%), use problems (7%) were 4 times more likely to harm (relative risk 4.2; 95% CI 2.5\u20137). Problems with data input to ML devices were the top contributor to events (82%).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Much of what is known about ML safety comes from case studies and the theoretical limitations of ML. We contribute a systematic analysis of ML safety problems captured as part of the FDA\u2019s routine post-market surveillance. Most problems involved devices and concerned the acquisition of data for processing by algorithms. However, problems with the use of devices were more likely to harm.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Safety problems with ML devices involve more than algorithms, highlighting the need for a whole-of-system approach to safe implementation with a special focus on how users interact with devices.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocad065","type":"journal-article","created":{"date-parts":[[2023,4,18]],"date-time":"2023-04-18T19:10:48Z","timestamp":1681845048000},"page":"1227-1236","source":"Crossref","is-referenced-by-count":41,"title":["More than algorithms: an analysis of safety events involving ML-enabled medical devices reported to the FDA"],"prefix":"10.1093","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2695-0368","authenticated-orcid":false,"given":"David","family":"Lyell","sequence":"first","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University , NSW 2109, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8537-3954","authenticated-orcid":false,"given":"Ying","family":"Wang","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University , NSW 2109, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6444-6584","authenticated-orcid":false,"given":"Enrico","family":"Coiera","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University , NSW 2109, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8426-5588","authenticated-orcid":false,"given":"Farah","family":"Magrabi","sequence":"additional","affiliation":[{"name":"Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University , NSW 2109, Australia"}]}],"member":"286","published-online":{"date-parts":[[2023,4,18]]},"reference":[{"issue":"2","key":"2023062008372490600_ocad065-B1","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1518\/001872097778543886","article-title":"Humans and automation: use, misuse, disuse, abuse","volume":"39","author":"Parasuraman","year":"1997","journal-title":"Hum Factors"},{"issue":"1","key":"2023062008372490600_ocad065-B2","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: the convergence of human and artificial intelligence","volume":"25","author":"Topol","year":"2019","journal-title":"Nat Med"},{"issue":"1","key":"2023062008372490600_ocad065-B3","doi-asserted-by":"crossref","first-page":"e100301","DOI":"10.1136\/bmjhci-2020-100301","article-title":"How machine learning is embedded to support clinician decision making: an analysis of FDA-approved medical devices","volume":"28","author":"Lyell","year":"2021","journal-title":"BMJ Health Care Inform"},{"key":"2023062008372490600_ocad065-B4","author":")"},{"key":"2023062008372490600_ocad065-B5","doi-asserted-by":"publisher","author":"Amodei","DOI":"10.48550\/arXiv.1606.06565"},{"issue":"3","key":"2023062008372490600_ocad065-B6","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1136\/bmjqs-2018-008370","article-title":"Artificial intelligence, bias and clinical safety","volume":"28","author":"Challen","year":"2019","journal-title":"BMJ Qual Saf"},{"issue":"8","key":"2023062008372490600_ocad065-B7","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"Wong","year":"2021","journal-title":"JAMA Intern Med"},{"issue":"11","key":"2023062008372490600_ocad065-B8","doi-asserted-by":"crossref","first-page":"e2135286","DOI":"10.1001\/jamanetworkopen.2021.35286","article-title":"Quantification of sepsis model alerts in 24 US hospitals before and during the COVID-19 pandemic","volume":"4","author":"Wong","year":"2021","journal-title":"JAMA Netw Open"},{"issue":"1","key":"2023062008372490600_ocad065-B9","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1038\/s41746-020-00324-0","article-title":"The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database","volume":"3","author":"Benjamens","year":"2020","journal-title":"NPJ Digit Med"},{"issue":"1","key":"2023062008372490600_ocad065-B10","doi-asserted-by":"crossref","first-page":"e100081","DOI":"10.1136\/bmjhci-2019-100081","article-title":"Human factors challenges for the safe use of artificial intelligence in patient care","volume":"26","author":"Sujan","year":"2019","journal-title":"BMJ Health Care Inform"},{"issue":"1","key":"2023062008372490600_ocad065-B11","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1016\/j.acra.2019.09.009","article-title":"The algorithmic audit: working with vendors to validate radiology-AI algorithms\u2014how we do it","volume":"27","author":"Mahajan","year":"2020","journal-title":"Acad Radiol"},{"issue":"5","key":"2023062008372490600_ocad065-B12","doi-asserted-by":"crossref","first-page":"e384","DOI":"10.1016\/S2589-7500(22)00003-6","article-title":"The medical algorithmic audit","volume":"4","author":"Liu","year":"2022","journal-title":"Lancet Digit Health"},{"issue":"3","key":"2023062008372490600_ocad065-B13","doi-asserted-by":"crossref","first-page":"224","DOI":"10.1136\/qhc.11.3.224","article-title":"Setting priorities for patient safety","volume":"11","author":"Runciman","year":"2002","journal-title":"Qual Saf Health Care"},{"issue":"2","key":"2023062008372490600_ocad065-B14","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1093\/jamia\/ocw154","article-title":"Problems with health information technology and their effects on care delivery and patient outcomes: a systematic review","volume":"24","author":"Kim","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"2023062008372490600_ocad065-B15","author":"Medical Device Reporting, 21 C.F.R","year":"2014"},{"key":"2023062008372490600_ocad065-B16","author":"U.S. Food & Drug Administration","year":"2022"},{"issue":"1","key":"2023062008372490600_ocad065-B17","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1136\/amiajnl-2011-000369","article-title":"Using FDA reports to inform a classification for health information technology safety problems","volume":"19","author":"Magrabi","year":"2012","journal-title":"J Am Med Inform Assoc"},{"key":"2023062008372490600_ocad065-B18","author":"U.S. Food and Drug Administration"},{"key":"2023062008372490600_ocad065-B19","author":"U.S. Food and Drug Administration"},{"key":"2023062008372490600_ocad065-B20","author":"U.S. Food and Drug Administration"},{"issue":"12","key":"2023062008372490600_ocad065-B21","doi-asserted-by":"crossref","first-page":"3532","DOI":"10.3390\/cancers12123532","article-title":"Application of artificial intelligence technology in oncology: towards the establishment of precision medicine","volume":"12","author":"Hamamoto","year":"2020","journal-title":"Cancers"},{"issue":"7","key":"2023062008372490600_ocad065-B22","doi-asserted-by":"crossref","first-page":"916","DOI":"10.1016\/j.ccell.2021.04.002","article-title":"Artificial intelligence for clinical oncology","volume":"39","author":"Kann","year":"2021","journal-title":"Cancer Cell"},{"key":"2023062008372490600_ocad065-B23","doi-asserted-by":"crossref","first-page":"e195","DOI":"10.1016\/S2589-7500(20)30292-2","article-title":"Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015-20): a comparative analysis","author":"Muehlematter","year":"2021","journal-title":"Lancet Digit Health"},{"issue":"4","key":"2023062008372490600_ocad065-B24","doi-asserted-by":"crossref","first-page":"582","DOI":"10.1038\/s41591-021-01312-x","article-title":"How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals","volume":"27","author":"Wu","year":"2021","journal-title":"Nat Med"},{"issue":"6","key":"2023062008372490600_ocad065-B25","doi-asserted-by":"crossref","first-page":"3797","DOI":"10.1007\/s00330-021-07892-z","article-title":"Artificial intelligence in radiology: 100 commercially available products and their scientific evidence","volume":"31","author":"van Leeuwen","year":"2021","journal-title":"Eur Radiol"},{"key":"2023062008372490600_ocad065-B26","author":"American College of Radiology DSI","year":"2021"},{"key":"2023062008372490600_ocad065-B27"},{"key":"2023062008372490600_ocad065-B28","author":"U.S. Food and Drug Administration","year":"2021"},{"key":"2023062008372490600_ocad065-B29","author":"U.S. Food & Drug Administration","year":"2018"},{"key":"2023062008372490600_ocad065-B30","author":"U.S. Food & Drug Administration","year":"2022"},{"issue":"6","key":"2023062008372490600_ocad065-B31","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1136\/jamia.2009.002444","article-title":"An analysis of computer-related patient safety incidents to inform the development of a classification","volume":"17","author":"Magrabi","year":"2010","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2023062008372490600_ocad065-B32","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1016\/j.ijmedinf.2014.12.003","article-title":"Clinical safety of England\u2019s national programme for IT: A retrospective analysis of all reported safety events 2005 to 2011","volume":"84","author":"Magrabi","year":"2015","journal-title":"Int J Med Inform"},{"key":"2023062008372490600_ocad065-B33","author":"U.S. Food & Drug Administration","year":"2022"},{"issue":"3","key":"2023062008372490600_ocad065-B34","doi-asserted-by":"crossref","first-page":"e1","DOI":"10.1136\/qshc.2002.004101","article-title":"Crisis management during anaesthesia: the development of an anaesthetic crisis management manual","volume":"14","author":"Runciman","year":"2005","journal-title":"Qual Saf Health Care"},{"key":"2023062008372490600_ocad065-B35","author":"U.S. Food & Drug Administration"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/7\/1227\/50634351\/ocad065.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/7\/1227\/50634351\/ocad065.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T08:39:34Z","timestamp":1687250374000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/30\/7\/1227\/7128049"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,18]]},"references-count":35,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2023,4,18]]},"published-print":{"date-parts":[[2023,6,20]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocad065","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,7,1]]},"published":{"date-parts":[[2023,4,18]]}}}