{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T13:30:04Z","timestamp":1791379804776,"version":"4.3.3"},"reference-count":45,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Artificial intelligence (AI)\u2013based clinical decision support systems are increasingly used in health care. Uncertainty-aware AI presents the model\u2019s confidence in its decision alongside its prediction, whereas black-box AI only provides a prediction. Little is known about how this type of AI affects health care providers\u2019 work performance and reaction time.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aimed to determine the effects of black-box and uncertainty-aware AI advice on pharmacist decision-making and reaction time.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Recruitment emails were sent to pharmacists through professional listservs describing a web-based, crossover, randomized controlled trial. Participants were randomized to the black-box AI or uncertainty-aware AI condition in a 1:1 manner. Participants completed 100 mock verification tasks with AI help and 100 without AI help. The order of no help and AI help was randomized. Participants were exposed to correct and incorrect prescription fills, where the correct decision was to \u201caccept\u201d or \u201creject,\u201d respectively. AI help provided correct (79%) or incorrect (21%) advice. Reaction times, participant decisions, AI advice, and AI help type were recorded for each verification. Likelihood ratio tests compared means across the three categories of AI type for each level of AI correctness.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>A total of 30 participants provided complete datasets. An equal number of participants were in each AI condition. Participants\u2019 decision-making performance and reaction times differed across the 3 conditions. Accurate AI recommendations resulted in the rejection of the incorrect drug 96.1% and 91.8% of the time for uncertainty-aware AI and black-box AI respectively, compared with 81.2% without AI help. Correctly dispensed medications were accepted at rates of 99.2% with black-box help, 94.1% with uncertainty-aware AI help, and 94.6% without AI help. Uncertainty-aware AI protected against bad AI advice to approve an incorrectly filled medication compared with black-box AI (83.3% vs 76.7%). When the AI recommended rejecting a correctly filled medication, pharmacists without AI help had a higher rate of correctly accepting the medication (94.6%) compared with uncertainty-aware AI help (86.2%) and black-box AI help (81.2%). Uncertainty-aware AI resulted in shorter reaction times than black-box AI and no AI help except in the scenario where \u201cAI rejects the correct drug.\u201d Black-box AI did not lead to reduced reaction times compared with pharmacists acting alone.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Pharmacists\u2019 performance and reaction times varied by AI type and AI accuracy. Overall, uncertainty-aware AI resulted in faster decision-making and acted as a safeguard against bad AI advice to approve a misfilled medication. Conversely, black-box AI had the longest reaction times, and user performance degraded in the presence of bad AI advice. However, uncertainty-aware AI could result in unnecessary double-checks, but it is preferred over false negative advice, where patients receive the wrong medication. These results highlight the importance of well-designed AI that addresses users\u2019 needs, enhances performance, and avoids overreliance on AI.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/64902","type":"journal-article","created":{"date-parts":[[2025,3,8]],"date-time":"2025-03-08T15:20:06Z","timestamp":1741447206000},"page":"e64902-e64902","source":"Crossref","is-referenced-by-count":15,"title":["Effect of Uncertainty-Aware AI Models on Pharmacists\u2019 Reaction Time and Decision-Making in a Web-Based Mock Medication Verification Task: Randomized Controlled Trial"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8774-793X","authenticated-orcid":false,"given":"Corey","family":"Lester","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8469-899X","authenticated-orcid":false,"given":"Brigid","family":"Rowell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6997-735X","authenticated-orcid":false,"given":"Yifan","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3656-0459","authenticated-orcid":false,"given":"Zoe","family":"Co","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2594-7407","authenticated-orcid":false,"given":"Vincent","family":"Marshall","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-1626-7204","authenticated-orcid":false,"given":"Jin Yong","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-4112-014X","authenticated-orcid":false,"given":"Qiyuan","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4546-324X","authenticated-orcid":false,"given":"Raed","family":"Kontar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6071-0387","authenticated-orcid":false,"given":"X Jessie","family":"Yang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,4,18]]},"reference":[{"issue":"11","key":"R1","doi-asserted-by":"publisher","first-page":"1818","DOI":"10.2174\/1386207325666211207153943","article-title":"Artificial intelligence (AI) in drugs and pharmaceuticals","volume":"25","author":"Sahu","journal-title":"Comb Chem High Throughput Screen"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1016\/j.semcancer.2020.06.002","article-title":"Artificial intelligence for breast cancer detection in mammography and digital breast tomosynthesis: state of the art","volume":"72","author":"Sechopoulos","journal-title":"Semin Cancer Biol"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1515\/cdbme-2019-0057","article-title":"Quantifying the uncertainty of deep learning-based computer-aided diagnosis for patient safety","volume":"5","author":"Laves","journal-title":"Curr Dir Biomed Eng"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1177\/23333928241234863","article-title":"Artificial intelligence and decision-making in healthcare: a thematic analysis of a systematic review of reviews","volume":"11","author":"Khosravi","journal-title":"Health Serv Res Manag Epidemiol"},{"issue":"4","key":"R5","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1111\/jcpt.12168","article-title":"Review of computerized clinical decision support in community pharmacy","volume":"39","author":"Curtain","journal-title":"J Clin Pharm Ther"},{"issue":"2","key":"R6","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1211\/ijpp.18.02.0002","volume":"18","author":"Robertson","journal-title":"Int J Pharm Pract"},{"issue":"4","key":"R7","doi-asserted-by":"publisher","first-page":"1607","DOI":"10.1007\/s13347-021-00477-0","article-title":"Transparency and the black box problem: why we do not trust AI","volume":"34","author":"von Eschenbach","journal-title":"Philos Technol"},{"issue":"CSCW","key":"R8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3359206","article-title":"\u201cHello AI\u201d: uncovering the onboarding needs of medical practitioners for human-AI collaborative decision-making","volume":"3","author":"Cai","journal-title":"Proc ACM Hum-Comput Interact"},{"issue":"3","key":"R9","doi-asserted-by":"publisher","first-page":"713","DOI":"10.1287\/isre.2020.0980","article-title":"Augmenting medical diagnosis decisions? An investigation into physicians\u2019 decision-making process with artificial intelligence","volume":"32","author":"Jussupow","journal-title":"Information Systems Research"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-00367-3","article-title":"Second opinion needed: communicating uncertainty in medical machine learning","volume":"4","author":"Kompa","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1038\/s42256-018-0004-1","article-title":"The need for uncertainty quantification in machine-assisted medical decision making","volume":"1","author":"Begoli","journal-title":"Nat Mach Intell"},{"issue":"4","key":"R12","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2020.100049","article-title":"Rapid trust calibration through interpretable and uncertainty-aware AI","volume":"1","author":"Tomsett","journal-title":"Patterns (N Y)"},{"key":"R13","doi-asserted-by":"crossref","unstructured":"Prabhudesai S Yang L Asthana S Huan X Liao QV Banovic N . Understanding uncertainty: how lay decision-makers perceive and interpret uncertainty in human-AI decision making. Presented at: 28th International Conference on Intelligent User Interfaces; Mar 27-31, 2023:379-396; Sydney, NSW, Australia. [doi: 10.1145\/3581641.3584033]","DOI":"10.1145\/3581641.3584033"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1136\/amiajnl-2011-000089","article-title":"Automation bias: a systematic review of frequency, effect mediators, and mitigators","volume":"19","author":"Goddard","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"R15","doi-asserted-by":"publisher","DOI":"10.1136\/bmjoq-2017-000193","article-title":"Systematic review and meta-analysis of community pharmacy error rates in the USA: 1993\u20132015","volume":"7","author":"Campbell","journal-title":"BMJ Open Qual"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","first-page":"17","DOI":"10.3758\/bf03208208","article-title":"Interaction of signal discriminability and task type in vigilance decrement","volume":"41","author":"Parasuraman","journal-title":"Percept Psychophys"},{"issue":"2","key":"R17","doi-asserted-by":"publisher","first-page":"328","DOI":"10.1016\/j.sapharm.2013.05.007","article-title":"The association of subjective workload dimensions on quality of care and pharmacist quality of work life","volume":"10","author":"Chui","journal-title":"Res Social Adm Pharm"},{"key":"R18","doi-asserted-by":"crossref","unstructured":"Warm J Dember WN Hancock P . Vigilance and workload in automated systems. In: Automation and Human Performance. Erlbaum; 1996. [doi: 10.1201\/9781315137957]","DOI":"10.1201\/9781315137957"},{"issue":"6","key":"R19","doi-asserted-by":"publisher","first-page":"661","DOI":"10.1016\/s0028-3932(98)00127-4","article-title":"The absent mind: further investigations of sustained attention to response","volume":"37","author":"Manly","journal-title":"Neuropsychologia"},{"key":"R20","unstructured":"NCPA releases 2023 digest report. National Community Pharmacists Association. 2023. URL: https:\/\/ncpa.org\/newsroom\/news-releases\/2023\/10\/15\/ncpa-releases-2023-digest-report [Accessed 26-03-2025]"},{"issue":"3","key":"R21","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2021-055597","article-title":"Pharmacist\u2019s time spent: Space for Pharmacy-based Interventions and Consultation TimE (SPICE)-an observational time and motion study","volume":"12","author":"Karia","journal-title":"BMJ Open"},{"issue":"6","key":"R22","doi-asserted-by":"publisher","first-page":"e215","DOI":"10.1016\/j.japh.2020.07.024","article-title":"Community pharmacist workflow and medication therapy management delegation: an assessment of preferences and barriers","volume":"60","author":"Hohmeier","journal-title":"J Am Pharm Assoc (2003)"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.2196\/51921","article-title":"Designing human-centered AI to prevent medication dispensing errors: focus group study with pharmacists","volume":"7","author":"Zheng","journal-title":"JMIR Form Res"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-021-00483-8","article-title":"Performance evaluation of a prescription medication image classification model: an observational cohort","volume":"4","author":"Lester","journal-title":"NPJ Digit Med"},{"key":"R25","unstructured":"Gal Y Ghahramani Z . Dropout as a Bayesian approximation: representing model uncertainty in deep learning. Presented at: 33rd International Conference on Machine Learning; Jun 19-24, 2016:1050-1059; New York City, USA. [doi: 10.48550\/arXiv.1506.02142]"},{"key":"R26","doi-asserted-by":"crossref","unstructured":"He K Zhang X Ren S Sun J . Deep residual learning for image recognition. Presented at: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Jun 26 to Jul 1, 2016:770-778; Las Vegas, NV, USA. 2016.[doi: 10.1109\/CVPR.2016.90]","DOI":"10.1109\/CVPR.2016.90"},{"key":"R27","unstructured":"R: a language and environment for statistical computing. R Foundation for Statistical Computing. 2017. URL: https:\/\/www.r-project.org\/ [Accessed 04-04-2025]"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.2196\/60273","article-title":"The effects of presenting AI uncertainty information on pharmacists\u2019 trust in automated pill recognition technology: exploratory mixed subjects study","volume":"12","author":"Kim","journal-title":"JMIR Hum Factors"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2022.103839","article-title":"Assessing the communication gap between AI models and healthcare professionals: explainability, utility and trust in AI-driven clinical decision-making","volume":"316","author":"Wysocki","journal-title":"Artif Intell"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2024.3437190","article-title":"Unpacking human-AI interaction in safety-critical industries: a systematic literature review","volume":"12","author":"Bach","journal-title":"IEEE Access"},{"key":"R31","doi-asserted-by":"crossref","unstructured":"Lindvall M Lundstr\u00f6m C L\u00f6wgren J . Rapid assisted visual search: supporting digital pathologists with imperfect AI. Presented at: 26th International Conference on Intelligent User Interfaces; Apr 14-17, 2021; College Station, TX, USA. [doi: 10.1145\/3397481.3450681]","DOI":"10.1145\/3397481.3450681"},{"issue":"2","key":"R32","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3577011","article-title":"Improving workflow integration with xPath: design and evaluation of a human-AI diagnosis system in pathology","volume":"30","author":"Gu","journal-title":"ACM Trans Comput-Hum Interact"},{"key":"R33","unstructured":"Pharmacists\u2019 impact on patient safety: a joint project of the American Pharmacists Association Academy of Pharmacy Practice and Management and Academy of Pharmaceutical Research and Science. American Pharmacists Association. URL: https:\/\/www.pharmacist.com\/Portals\/0\/PDFS\/Practice\/PharmacistsImpactonPatientSafety_Web.pdf?ver=dYeAzwlN3-PG9eSkMMsV-A%3D%3D [Accessed 26-03-2025]"},{"issue":"15","key":"R34","doi-asserted-by":"publisher","first-page":"1858","DOI":"10.1001\/jama.296.15.1858","article-title":"National surveillance of emergency department visits for outpatient adverse drug events","volume":"296","author":"Budnitz","journal-title":"JAMA"},{"issue":"21","key":"R35","doi-asserted-by":"publisher","first-page":"2002","DOI":"10.1056\/NEJMsa1103053","article-title":"Emergency hospitalizations for adverse drug events in older Americans","volume":"365","author":"Budnitz","journal-title":"N Engl J Med"},{"key":"R36","unstructured":"Tariq RA Vashisht R Sinha A Scherbak Y . Medication Dispensing Errors and Prevention. StatPearls; 2021. [Medline: 30085607]"},{"key":"R37","unstructured":"Pharmacist professional liability exposure claim report: 3rd edition. CNA, HSPO. URL: https:\/\/www.hpso.com\/getmedia\/261a3590-6368-41e6-857c-f2bf9dbdf6ea\/CNA_CLS_PHARM_072823_CF_PROD_SEC [Accessed 26-03-2025]"},{"key":"R38","doi-asserted-by":"crossref","unstructured":"Ilut C Schneider M . Modeling uncertainty as ambiguity: a review. National Bureau of Economic Research; 2022. Report No.: w29915. [doi: 10.3386\/w29915]","DOI":"10.3386\/w29915"},{"issue":"1","key":"R39","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1111\/j.1540-5915.1997.tb01303.x","article-title":"Risk, ambiguity, and decision choice: some additional evidence*","volume":"28","author":"Ghosh","journal-title":"Decision Sciences"},{"issue":"1","key":"R40","doi-asserted-by":"publisher","first-page":"37","DOI":"10.3821\/1913-701X-143.1.37","article-title":"Are pharmacists the ultimate barrier to pharmacy practice change?","volume":"143","author":"Rosenthal","journal-title":"Can Pharm J"},{"key":"R41","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2021.107018","article-title":"Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of AI advice","volume":"127","author":"Chong","journal-title":"Comput Human Behav"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.1145\/3449287","article-title":"To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making","volume":"5","author":"Bu\u00e7inca","journal-title":"Proc ACM Hum-Comput Interact"},{"key":"R43","doi-asserted-by":"crossref","unstructured":"Gajos KZ Mamykina L . Do people engage cognitively with AI? impact of AI assistance on incidental learning. Presented at: IUI \u201922; Mar 21-25, 2022; Helsinki, Finland. URL: https:\/\/dl.acm.org\/doi\/proceedings\/10.1145\/3490099 [Accessed 04-04-2025] [doi: 10.1145\/3490099.3511138]","DOI":"10.1145\/3490099.3511138"},{"key":"R44","doi-asserted-by":"crossref","unstructured":"Fogliato R Chappidi S Lungren M et al. Who goes first? Influences of human-ai workflow on decision making in clinical imaging. Presented at: FAccT \u201922; Jun 21-24, 2022; Seoul, Republic of Korea. URL: https:\/\/dl.acm.org\/doi\/proceedings\/10.1145\/3531146 [Accessed 04-04-2025] [doi: 10.1145\/3531146.3533193]","DOI":"10.1145\/3531146.3533193"},{"key":"R45","unstructured":"Chen Q Al Kontar R Nouiehed M Yang XJ Lester C . Cost-sensitive Adversarial Data Augmentation (CSADA). Code Ocean. 2024. URL: https:\/\/codeocean.com\/capsule\/0288821\/tree\/v1 [Accessed 04-04-2025]"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e64902","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:30:05Z","timestamp":1785537005000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e64902"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,18]]},"references-count":45,"URL":"https:\/\/doi.org\/10.2196\/64902","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,18]]},"article-number":"v13i11e64902"}}