{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T21:19:15Z","timestamp":1769548755898,"version":"3.49.0"},"reference-count":30,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2020,6,23]],"date-time":"2020-06-23T00:00:00Z","timestamp":1592870400000},"content-version":"vor","delay-in-days":5,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Australian National Health and Medical Research Council","award":["APP1022964"],"award-info":[{"award-number":["APP1022964"]}]},{"name":"Centre for Research Excellence in Digital Health","award":["1134919"],"award-info":[{"award-number":["1134919"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>The study sought to evaluate the feasibility of using Unified Medical Language System (UMLS) semantic features for automated identification of reports about patient safety incidents by type and severity.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and Methods<\/jats:title><jats:p>Binary support vector machine (SVM) classifier ensembles were trained and validated using balanced datasets of critical incident report texts (n_type\u2009=\u20092860, n_severity\u2009=\u20091160) collected from a state-wide reporting system. Generalizability was evaluated on different and independent hospital-level reporting system. Concepts were extracted from report narratives using the UMLS Metathesaurus, and their relevance and frequency were used as semantic features. Performance was evaluated by F-score, Hamming loss, and exact match score and was compared with SVM ensembles using bag-of-words (BOW) features on 3 testing datasets (type\/severity: n_benchmark\u2009=\u2009286\/116, n_original\u2009=\u2009444\/4837, n_independent =6000\/5950).<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>SVMs using semantic features met or outperformed those based on BOW features to identify 10 different incident types (F-score [semantics\/BOW]: benchmark\u2009=\u200982.6%\/69.4%; original\u2009=\u200977.9%\/68.8%; independent\u2009=\u200978.0%\/67.4%) and extreme-risk events (F-score [semantics\/BOW]: benchmark\u2009=\u200987.3%\/87.3%; original\u2009=\u200925.5%\/19.8%; independent\u2009=\u200949.6%\/52.7%). For incident type, the exact match score for semantic classifiers was consistently higher than BOW across all test datasets (exact match [semantics\/BOW]: benchmark\u2009=\u200948.9%\/39.9%; original\u2009=\u200957.9%\/44.4%; independent\u2009=\u200959.5%\/34.9%).<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>BOW representations are not ideal for the automated identification of incident reports because they do not account for text semantics. UMLS semantic representations are likely to better capture information in report narratives, and thus may explain their superior performance.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>UMLS-based semantic classifiers were effective in identifying incidents by type and extreme-risk events, providing better generalizability than classifiers using BOW.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocaa082","type":"journal-article","created":{"date-parts":[[2020,4,27]],"date-time":"2020-04-27T11:08:34Z","timestamp":1587985714000},"page":"1502-1509","source":"Crossref","is-referenced-by-count":11,"title":["Can Unified Medical Language System\u2013based semantic representation improve automated identification of patient safety incident reports by type and severity?"],"prefix":"10.1093","volume":"27","author":[{"given":"Ying","family":"Wang","sequence":"first","affiliation":[]},{"given":"Enrico","family":"Coiera","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8426-5588","authenticated-orcid":false,"given":"Farah","family":"Magrabi","sequence":"additional","affiliation":[]}],"member":"286","published-online":{"date-parts":[[2020,6,18]]},"reference":[{"issue":"38","key":"2020110613094578000_ocaa082-B1","doi-asserted-by":"crossref","first-page":"e12509","DOI":"10.1097\/MD.0000000000012509","article-title":"Effectiveness and limitations of an incident-reporting system analyzed by local clinical safety leaders in a tertiary hospital: prospective evaluation through real-time observations of patient safety incidents","volume":"97","author":"Ramirez","year":"2018","journal-title":"Medicine (Baltimore)"},{"issue":"1","key":"2020110613094578000_ocaa082-B2","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1093\/intqhc\/mzn057","article-title":"Towards an international classification for patient safety: key concepts and terms","volume":"21","author":"Runciman","year":"2009","journal-title":"Int J Qual Health Care"},{"issue":"5","key":"2020110613094578000_ocaa082-B3","doi-asserted-by":"crossref","first-page":"698","DOI":"10.1093\/bja\/aeq287","article-title":"Critical incident reporting and learning","volume":"105","author":"Bolsin","year":"2010","journal-title":"Br J Anaesth"},{"key":"2020110613094578000_ocaa082-B4","author":"Clinical Incident Management in the NSW Public Health System"},{"key":"2020110613094578000_ocaa082-B5","doi-asserted-by":"crossref","first-page":"i82","DOI":"10.1136\/qshc.2005.017467","article-title":"An integrated framework for safety, quality and risk management: an information and incident management system based on a universal patient safety classification","volume":"15 (suppl 1","author":"Runciman","year":"2006","journal-title":"Qual Saf Health Care"},{"key":"2020110613094578000_ocaa082-B6","author":"Agency for Healthcare Research and Quality","year":"2016"},{"issue":"5","key":"2020110613094578000_ocaa082-B7","doi-asserted-by":"crossref","first-page":"316","DOI":"10.1093\/intqhc\/mzp034","article-title":"Medication errors: how reliable are the severity ratings reported to the national reporting and learning system?","volume":"21","author":"Williams","year":"2009","journal-title":"Int J Qual Health Care"},{"issue":"4","key":"2020110613094578000_ocaa082-B8","doi-asserted-by":"crossref","first-page":"609","DOI":"10.1007\/s10916-009-9398-y","article-title":"Data consistency in a voluntary medical incident reporting system","volume":"35","author":"Gong","year":"2011","journal-title":"J Med Syst"},{"issue":"3","key":"2020110613094578000_ocaa082-B9","doi-asserted-by":"crossref","first-page":"517","DOI":"10.1111\/j.1532-5415.2008.02142.x","article-title":"Inconsistency in classification and reporting of in-hospital falls","volume":"57","author":"Haines","year":"2009","journal-title":"J Am Geriatr Soc"},{"key":"2020110613094578000_ocaa082-B10","doi-asserted-by":"crossref","first-page":"103971","DOI":"10.1016\/j.ijmedinf.2019.103971","article-title":"A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysis","volume":"132","author":"Young","year":"2019","journal-title":"Int J Med Inform"},{"issue":"12","key":"2020110613094578000_ocaa082-B11","doi-asserted-by":"crossref","first-page":"1600","DOI":"10.1093\/jamia\/ocz146","article-title":"Using convolutional neural networks to identify patient safety incident reports by type and severity","volume":"26","author":"Wang","year":"2019","journal-title":"J Am Med Inform Assn"},{"key":"2020110613094578000_ocaa082-B12","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1186\/s12911-017-0483-8","article-title":"Using multiclass classification to automate the identification of patient safety incident reports by type and severity","volume":"17","author":"Wang","year":"2017","journal-title":"BMC Med Inform Decis Mak"},{"key":"2020110613094578000_ocaa082-B13","doi-asserted-by":"crossref","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The Unified Medical Language System (UMLS): integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res"},{"key":"2020110613094578000_ocaa082-B14","first-page":"1070","article-title":"Automated classification of multi-labeled patient safety reports: a shift from quantity to quality measure","volume":"245","author":"Liang","year":"2017","journal-title":"Stud Health Technol Inform"},{"key":"2020110613094578000_ocaa082-B15","first-page":"609","article-title":"Automating the identification of patient safety incident reports using multi-label classification","volume":"245","author":"Wang","year":"2017","journal-title":"Stud Health Technol Inform"},{"key":"2020110613094578000_ocaa082-B16"},{"key":"2020110613094578000_ocaa082-B17","volume-title":"Safety and Ethics in Healthcare: A Guide to Getting it Right","author":"Runciman","year":"2007"},{"issue":"10","key":"2020110613094578000_ocaa082-B18","first-page":"522","article-title":"Developing and deploying a patient safety program in a large health care delivery system: you can\u2019t fix what you don\u2019t know about","volume":"27","author":"Bagian","year":"2001","journal-title":"Jt Comm J Qual Improv"},{"issue":"3","key":"2020110613094578000_ocaa082-B19","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1136\/jamia.2009.002733","article-title":"An overview of MetaMap: historical perspective and recent advances","volume":"17","author":"Aronson","year":"2010","journal-title":"J Am Med Inform Assoc"},{"issue":"1\u20134","key":"2020110613094578000_ocaa082-B20","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1007\/s13042-010-0001-0","article-title":"Understanding bag-of-words model: a statistical framework","volume":"1","author":"Zhang","year":"2010","journal-title":"Int J Mach Learn Cybern"},{"key":"2020110613094578000_ocaa082-B21","first-page":"1","volume-title":"Lecture Notes in Computer Science: Multiple Classifier Systems","author":"Dietterich","year":"2000"},{"issue":"8","key":"2020110613094578000_ocaa082-B22","doi-asserted-by":"crossref","first-page":"1761","DOI":"10.1016\/j.patcog.2011.01.017","article-title":"An overview of ensemble methods for binary classifiers in multi-class problems: experimental study on one-vs-one and one-vs-all schemes","volume":"44","author":"Galar","year":"2011","journal-title":"Pattern Recogn"},{"issue":"9","key":"2020110613094578000_ocaa082-B23","doi-asserted-by":"crossref","first-page":"3084","DOI":"10.1016\/j.patcog.2012.03.004","article-title":"An extensive experimental comparison of methods for multi-label learning","volume":"45","author":"Madjarov","year":"2012","journal-title":"Pattern Recogn"},{"issue":"2","key":"2020110613094578000_ocaa082-B24","first-page":"265","article-title":"On the algorithmic implementation of multiclass kernel-based vector machines","volume":"2","author":"Crammer","year":"2002","journal-title":"J Mach Learn Res"},{"issue":"06","key":"2020110613094578000_ocaa082-B25","doi-asserted-by":"crossref","first-page":"1175","DOI":"10.1142\/S0219622013500338","article-title":"A multi-class support vector machine: theory and model","volume":"12","author":"Sun","year":"2013","journal-title":"Int J Inf Technol Decis Mak"},{"key":"2020110613094578000_ocaa082-B26","first-page":"254","article-title":"Classifier chains for multi-label classification. machine learning and knowledge discovery in databases","volume":"5782","author":"Read","year":"2009","journal-title":"Mach Learn"},{"issue":"1","key":"2020110613094578000_ocaa082-B27","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1086\/256633","article-title":"On the rationale of group decision-making","volume":"56","author":"Black","year":"1948","journal-title":"J Polit Econ"},{"issue":"4","key":"2020110613094578000_ocaa082-B28","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1016\/j.ipm.2009.03.002","article-title":"A systematic analysis of performance measures for classification tasks","volume":"45","author":"Sokolova","year":"2009","journal-title":"Inform Process Manag"},{"issue":"11","key":"2020110613094578000_ocaa082-B29","doi-asserted-by":"crossref","first-page":"e16323","DOI":"10.2196\/16323","article-title":"The last mile: where artificial intelligence meets reality","volume":"21","author":"Coiera","year":"2019","journal-title":"J Med Internet Res"},{"issue":"4","key":"2020110613094578000_ocaa082-B30","doi-asserted-by":"crossref","first-page":"e10203","DOI":"10.1002\/lrh2.10203","article-title":"Computable knowledge: an imperative for learning health systems","volume":"3","author":"Friedman","year":"2019","journal-title":"Learn Health Syst"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/10\/1502\/34153220\/ocaa082.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/27\/10\/1502\/34153220\/ocaa082.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,30]],"date-time":"2023-09-30T10:35:56Z","timestamp":1696070156000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/27\/10\/1502\/5859729"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,18]]},"references-count":30,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2020,6,18]]},"published-print":{"date-parts":[[2020,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaa082","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,10]]},"published":{"date-parts":[[2020,6,18]]}}}