{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T16:39:03Z","timestamp":1782491943309,"version":"3.54.5"},"reference-count":19,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T00:00:00Z","timestamp":1620259200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,7,30]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>The study sought to assess the clinical performance of a machine learning model aiming to identify unusual medication orders.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>This prospective study was conducted at CHU Sainte-Justine, Canada, from April to August 2020. An unsupervised machine learning model based on GANomaly and 2 baselines were trained to learn medication order patterns from 10 years of data. Clinical pharmacists dichotomously (typical or atypical) labeled orders and pharmacological profiles (patients\u2019 medication lists). Confusion matrices, areas under the precision-recall curve (AUPRs), and F1 scores were calculated.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>A total of 12\u00a0471 medication orders and 1356 profiles were labeled by 25 pharmacists. Medication order predictions showed a precision of 35%, recall (sensitivity) of 26%, and specificity of 97% as compared with pharmacist labels, with an AUPR of 0.25 and an F1 score of 0.30. Profile predictions showed a precision of 49%, recall of 75%, and specificity of 82%, with an AUPR of 0.60, and an F1 score of 0.59. The model performed better than the baselines. According to the pharmacists, the model was a useful screening tool, and 9 of 15 participants preferred predictions by medication, rather than by profile.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>Predictions for profiles had higher F1 scores and recall compared with medication order predictions. Although the performance was much better for profile predictions, pharmacists generally preferred medication order predictions.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Based on the AUPR, this model showed better performance for the identification of atypical pharmacological profiles than for medication orders. Pharmacists considered the model a useful screening tool. Improving these predictions should be prioritized in future research to maximize clinical impact.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocab071","type":"journal-article","created":{"date-parts":[[2021,4,6]],"date-time":"2021-04-06T19:13:19Z","timestamp":1617736399000},"page":"1712-1718","source":"Crossref","is-referenced-by-count":25,"title":["Pharmacists\u2019 perceptions of a machine learning model for the identification of atypical medication orders"],"prefix":"10.1093","volume":"28","author":[{"given":"Sophie-Camille","family":"Hogue","sequence":"first","affiliation":[{"name":"Pharmacy Practice Research Unit and Department of Pharmacy, CHU Sainte-Justine, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Flora","family":"Chen","sequence":"additional","affiliation":[{"name":"Pharmacy Practice Research Unit and Department of Pharmacy, CHU Sainte-Justine, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Genevi\u00e8ve","family":"Brassard","sequence":"additional","affiliation":[{"name":"Pharmacy Practice Research Unit and Department of Pharmacy, CHU Sainte-Justine, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Denis","family":"Lebel","sequence":"additional","affiliation":[{"name":"Pharmacy Practice Research Unit and Department of Pharmacy, CHU Sainte-Justine, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jean-Fran\u00e7ois","family":"Bussi\u00e8res","sequence":"additional","affiliation":[{"name":"Pharmacy Practice Research Unit and Department of Pharmacy, CHU Sainte-Justine, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Audrey","family":"Durand","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Software Engineering, Universit\u00e9 Laval, Quebec City, Quebec, Canada"},{"name":"Department of Electrical and Computer Engineering, Universit\u00e9 Laval, Quebec City, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maxime","family":"Thibault","sequence":"additional","affiliation":[{"name":"Pharmacy Practice Research Unit and Department of Pharmacy, CHU Sainte-Justine, Montreal, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2021,5,6]]},"reference":[{"issue":"19","key":"2021073020265108200_ocab071-B1","doi-asserted-by":"crossref","first-page":"1556","DOI":"10.1093\/ajhp\/zxaa218","article-title":"Demystifying artificial intelligence in pharmacy","volume":"77","author":"Nelson","year":"2020","journal-title":"Am J Health Syst Pharm"},{"issue":"9","key":"2021073020265108200_ocab071-B2","doi-asserted-by":"crossref","first-page":"622","DOI":"10.1093\/ajhp\/zxz018","article-title":"Using artificial intelligence in health-system pharmacy practice: Finding new patterns that matter","volume":"76","author":"Flynn","year":"2019","journal-title":"Am J Health Syst Pharm"},{"issue":"12","key":"2021073020265108200_ocab071-B3","doi-asserted-by":"crossref","first-page":"1078","DOI":"10.2146\/ajhp090095","article-title":"Automating order review is delegation, not abdication","volume":"66","author":"Tribble","year":"2009","journal-title":"Am J Health Syst Pharm"},{"issue":"8","key":"2021073020265108200_ocab071-B4","doi-asserted-by":"crossref","first-page":"704","DOI":"10.2146\/ajhp080410","article-title":"An informatics perspective on nearly universal prospective order review","volume":"66","author":"Poikonen","year":"2009","journal-title":"Am J Health Syst Pharm"},{"issue":"17","key":"2021073020265108200_ocab071-B5","doi-asserted-by":"crossref","first-page":"1471","DOI":"10.2146\/ajhp140673","article-title":"Frequency of and risk factors for medication errors by pharmacists during order verification in a tertiary care medical center","volume":"72","author":"Gorbach","year":"2015","journal-title":"Am J Health Syst Pharm"},{"issue":"4","key":"2021073020265108200_ocab071-B6","first-page":"288","article-title":"Hospital pharmacists' perceptions and decision-making related to drug-drug interactions","volume":"72","author":"Bagri","year":"2019","journal-title":"Can J Hosp Pharm"},{"issue":"8","key":"2021073020265108200_ocab071-B7","doi-asserted-by":"crossref","first-page":"530","DOI":"10.1093\/ajhp\/zxz012","article-title":"A systematic approach to optimize electronic health record medication alerts in a health system","volume":"76","author":"Bhakta","year":"2019","journal-title":"Am J Health Syst Pharm"},{"issue":"12","key":"2021073020265108200_ocab071-B8","doi-asserted-by":"crossref","first-page":"1560","DOI":"10.1093\/jamia\/ocz135","article-title":"Reducing drug prescription errors and adverse drug events by application of a probabilistic, machine-learning-based clinical decision support system in an inpatient setting","volume":"26","author":"Segal","year":"2019","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2021073020265108200_ocab071-B9","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1093\/jamia\/ocw171","article-title":"Screening for medication errors using an outlier detection system","volume":"24","author":"Schiff","year":"2017","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"2021073020265108200_ocab071-B10","doi-asserted-by":"crossref","first-page":"1688","DOI":"10.1093\/jamia\/ocaa154","article-title":"A machine learning-based clinical decision support system to identify prescriptions with a high risk of medication error","volume":"27","author":"Corny","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"2021073020265108200_ocab071-B11","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1136\/amiajnl-2013-002008","article-title":"Clinical decision support for atypical orders: detection and warning of atypical medication orders submitted to a computerized provider order entry system","volume":"21","author":"Woods","year":"2014","journal-title":"J Am Med Inform Assoc"},{"issue":"2","key":"2021073020265108200_ocab071-B12","doi-asserted-by":"crossref","first-page":"874","DOI":"10.1109\/JBHI.2018.2828028","article-title":"DDC-outlier: preventing medication errors using unsupervised learning","volume":"23","author":"Santos","year":"2019","journal-title":"IEEE J Biomed Health Inform"},{"key":"2021073020265108200_ocab071-B13","author":"Akcay","year":"2018"},{"issue":"1","key":"2021073020265108200_ocab071-B14","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1109\/TSMC.1979.4310076","article-title":"A threshold selection method from Gray-Level Histograms","volume":"9","author":"Otsu","year":"1979","journal-title":"IEEE Trans Syst Man Cybern"},{"key":"2021073020265108200_ocab071-B15","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"key":"2021073020265108200_ocab071-B16","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1186\/1471-2105-12-77","article-title":"pROC: an open-source package for R and S+ to analyze and compare ROC curves","volume":"12","author":"Robin","year":"2011","journal-title":"BMC Bioinformatics"},{"issue":"3","key":"2021073020265108200_ocab071-B17","doi-asserted-by":"crossref","first-page":"e0118432","DOI":"10.1371\/journal.pone.0118432","article-title":"The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets","volume":"10","author":"Saito","year":"2015","journal-title":"PLoS One"},{"issue":"7","key":"2021073020265108200_ocab071-B18","doi-asserted-by":"crossref","first-page":"1173","DOI":"10.1093\/jamia\/ocaa053","article-title":"Explainable artificial intelligence models using real-world electronic health record data: a systematic scoping review","volume":"27","author":"Payrovnaziri","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2021073020265108200_ocab071-B19","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.ijmedinf.2017.03.006","article-title":"Decaying relevance of clinical data towards future decisions in data-driven inpatient clinical order sets","volume":"102","author":"Chen","year":"2017","journal-title":"Int J Med Inform"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/8\/1712\/39502236\/ocab071.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/8\/1712\/39502236\/ocab071.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,30]],"date-time":"2021-07-30T23:05:42Z","timestamp":1627686342000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/8\/1712\/6270973"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,6]]},"references-count":19,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,5,6]]},"published-print":{"date-parts":[[2021,7,30]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab071","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,8,1]]},"published":{"date-parts":[[2021,5,6]]}}}