{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T05:09:38Z","timestamp":1789016978211,"version":"build-2803163510"},"reference-count":53,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T00:00:00Z","timestamp":1788998400000},"content-version":"vor","delay-in-days":63,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000133","name":"Agency for Healthcare Research and Quality","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000133","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,7,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Despite the recognized need to leverage health information technology (IT) to facilitate error detection, there are few health IT safety measures in use and limited resources available to guide the conceptualization, development, and testing of new measures. Audit log data provide a rich source of clinician behaviors within the electronic health record (EHR), which can be used to develop health IT safety measures.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>With illustrative examples, we describe a framework for the development and evaluation of novel order error measures by employing the Retract-and-Reorder (RAR) methodology, which uses audit log data to identify actions indicative of potential order errors.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Topics covered include the following: (1) insight into the value of large data repositories generated during clinical care for creating automated health IT safety measures, (2) the principles and theoretical model that underly the ability of RAR measures to detect order errors, (3) an experience-based framework for the conceptualization, implementation, validation, and optimization of new RAR measures to detect order errors, and (4) applications of RAR measures to guide understanding of health IT safety events.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>This report is intended for informatics and health services researchers and healthcare system leadership who seek to utilize EHR audit log data to identify and quantify order errors in EHR systems.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The methodology described can be applied to develop automated measures to detect a range of order error types, examine the epidemiology, and investigate the root causes of order errors in near-real-time, as well as rigorously evaluate the impact of preventive interventions.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamiaopen\/ooag165","type":"journal-article","created":{"date-parts":[[2026,7,28]],"date-time":"2026-07-28T12:53:52Z","timestamp":1785243232000},"source":"Crossref","is-referenced-by-count":0,"title":["A framework for developing, validating, and utilizing automated measures of order errors using the retract-and-reorder methodology"],"prefix":"10.1093","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7002-0504","authenticated-orcid":false,"given":"Pooja Reddy","family":"Spector","sequence":"first","affiliation":[{"name":"Department of Pediatrics, Columbia University Irving Medical Center , New York, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anne","family":"Grauer","sequence":"additional","affiliation":[{"name":"Department of Medicine, Columbia University Irving Medical Center , New York, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jerard Z","family":"Kneifati-Hayek","sequence":"additional","affiliation":[{"name":"Department of Medicine, Columbia University Irving Medical Center , New York, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin L","family":"Ranard","sequence":"additional","affiliation":[{"name":"Department of Medicine, Columbia University Irving Medical Center , New York, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jo R","family":"Applebaum","sequence":"additional","affiliation":[{"name":"Department of Medicine, Columbia University Irving Medical Center , New York, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yelstin","family":"Fernandes","sequence":"additional","affiliation":[{"name":"Department of Medicine, Columbia University Irving Medical Center , New York, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hojjat","family":"Salmasian","sequence":"additional","affiliation":[{"name":"Department of Medicine, Brigham and Women\u2019s Hospital\/Harvard Medical School , Boston, MA 02115,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William N","family":"Southern","sequence":"additional","affiliation":[{"name":"Department of Medicine, Albert Einstein College of Medicine, Montefiore Medical Center , Bronx, NY 10467,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Clyde B","family":"Schechter","sequence":"additional","affiliation":[{"name":"Department of Family and Social Medicine, Albert Einstein College of Medicine , Bronx, NY 10467,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gregory","family":"Hruby","sequence":"additional","affiliation":[{"name":"Department of Quality and Patient Safety, NewYork-Presbyterian Hospital , New York, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joseph T","family":"Cooke","sequence":"additional","affiliation":[{"name":"Department of Medicine, NewYork-Presbyterian\/Queens Hospital , Queens, NY 11355,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deepa","family":"Kumaraiah","sequence":"additional","affiliation":[{"name":"Department of Medicine, Columbia University Irving Medical Center , New York, NY 10032,","place":["United States"]},{"name":"Department of Quality and Patient Safety, NewYork-Presbyterian Hospital , New York, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Baruch","family":"Fertel","sequence":"additional","affiliation":[{"name":"Department of Quality and Patient Safety, NewYork-Presbyterian Hospital , New York, NY 10032,","place":["United States"]},{"name":"Department of Emergency Medicine, Columbia University Irving Medical Center , New York, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason S","family":"Adelman","sequence":"additional","affiliation":[{"name":"Department of Medicine, Columbia University Irving Medical Center , New York, NY 10032,","place":["United States"]},{"name":"Department of Quality and Patient Safety, NewYork-Presbyterian Hospital , New York, NY 10032,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,9,9]]},"reference":[{"key":"2026091000252482600_ooag165-B1","doi-asserted-by":"publisher","DOI":"10.17226\/13269","volume-title":"Health IT and Patient Safety: Building Safer Systems for Better Care","author":"Institute of Medicine Committee on Patient Safety and Health Information Technology","year":"2011"},{"key":"2026091000252482600_ooag165-B2","volume-title":"Identification and Prioritization of Health IT Patient Safety Measures","author":"National Quality Forum","year":"2016"},{"key":"2026091000252482600_ooag165-B3","doi-asserted-by":"publisher","first-page":"461","DOI":"10.1197\/jamia.M2549","article-title":"Identifying and quantifying medication errors: evaluation of rapidly discontinued medication orders submitted to a computerized physician order entry system","volume":"15","author":"Koppel","year":"2008","journal-title":"J Am Med Inform Assoc"},{"key":"2026091000252482600_ooag165-B4","first-page":"1294","article-title":"\u201cI meant that med for Baylee not Bailey!\u201d: a mixed method study to identify incidence and risk factors for CPOE patient misidentification","volume":"2012","author":"Levin","year":"2012","journal-title":"AMIA Annu Symp Proc"},{"key":"2026091000252482600_ooag165-B5","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1136\/amiajnl-2012-001055","article-title":"Understanding and preventing wrong-patient electronic orders: a randomized controlled trial","volume":"20","author":"Adelman","year":"2013","journal-title":"J Am Med Inform Assoc"},{"key":"2026091000252482600_ooag165-B6","doi-asserted-by":"publisher","first-page":"908","DOI":"10.1136\/bmjqs-2019-009420","article-title":"Automated detection of wrong-drug prescribing errors","volume":"28","author":"Lambert","year":"2019","journal-title":"BMJ Qual Saf"},{"key":"2026091000252482600_ooag165-B7","doi-asserted-by":"publisher","first-page":"103343","DOI":"10.1016\/j.jbi.2019.103343","article-title":"EHR audit logs: a new goldmine for health services research?","volume":"101","author":"Adler-Milstein","year":"2020","journal-title":"J Biomed Inform"},{"key":"2026091000252482600_ooag165-B8","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1093\/jamia\/ocac173","article-title":"Using electronic health record audit log data for research: insights from early efforts","volume":"30","author":"Kannampallil","year":"2022","journal-title":"J Am Med Inform Assoc"},{"key":"2026091000252482600_ooag165-B9","author":"National Quality Forum","year":"2016"},{"key":"2026091000252482600_ooag165-B10","author":"Office of the National Coordinator for Health Information Technology (ONC). Patient Identification SAFER Guide","year":"2016"},{"key":"2026091000252482600_ooag165-B11","author":"The Joint Commission","year":"2018"},{"key":"2026091000252482600_ooag165-B12","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1542\/peds.2015-0007","article-title":"Use of temporary names for newborns and associated risks","volume":"136","author":"Adelman","year":"2015","journal-title":"Pediatrics"},{"key":"2026091000252482600_ooag165-B13","doi-asserted-by":"publisher","first-page":"e20162863","DOI":"10.1542\/peds.2016-2863","article-title":"Evaluating serial strategies for preventing wrong-patient orders in the NICU","volume":"139","author":"Adelman","year":"2017","journal-title":"Pediatrics"},{"key":"2026091000252482600_ooag165-B14","doi-asserted-by":"publisher","first-page":"979","DOI":"10.1001\/jamapediatrics.2019.2733","article-title":"Risk of wrong-patient orders among multiple vs singleton births in the neonatal intensive care units of 2 integrated health care systems","volume":"173","author":"Adelman","year":"2019","journal-title":"JAMA Pediatr"},{"key":"2026091000252482600_ooag165-B15","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1097\/AOG.0000000000004474","article-title":"Wrong-patient orders in obstetrics","volume":"138","author":"Kern-Goldberger","year":"2021","journal-title":"Obstet Gynecol"},{"key":"2026091000252482600_ooag165-B16","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1097\/AOG.0000000000003872","article-title":"Wrong-patient ordering errors in peripartum mother-newborn pairs: a unique patient-safety challenge in obstetrics","volume":"136","author":"Kern-Goldberger","year":"2020","journal-title":"Obstet Gynecol"},{"key":"2026091000252482600_ooag165-B17","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1016\/j.annemergmed.2014.11.017","article-title":"Intercepting wrong-patient orders in a computerized provider order entry system","volume":"65","author":"Green","year":"2015","journal-title":"Ann Emerg Med"},{"key":"2026091000252482600_ooag165-B18","doi-asserted-by":"publisher","first-page":"1780","DOI":"10.1001\/jama.2019.3698","article-title":"Effect of restriction of the number of concurrently open records in an electronic health record on wrong-patient order errors: a randomized clinical trial","volume":"321","author":"Adelman","year":"2019","journal-title":"JAMA"},{"key":"2026091000252482600_ooag165-B19","doi-asserted-by":"publisher","first-page":"739","DOI":"10.1093\/jamia\/ocx099","article-title":"Effect of number of open charts on intercepted wrong-patient medication orders in an emergency department","volume":"25","author":"Kannampallil","year":"2018","journal-title":"J Am Med Inform Assoc"},{"key":"2026091000252482600_ooag165-B20","doi-asserted-by":"publisher","first-page":"e2019652","DOI":"10.1001\/jamanetworkopen.2020.19652","article-title":"Association of display of patient photographs in the electronic health record with wrong-patient order entry errors","volume":"3","author":"Salmasian","year":"2020","journal-title":"JAMA Netw Open"},{"key":"2026091000252482600_ooag165-B21"},{"key":"2026091000252482600_ooag165-B22","doi-asserted-by":"publisher","first-page":"e167","DOI":"10.1097\/pq9.0000000000000167","article-title":"Encouraging resident adverse event reporting: a qualitative study of suggestions from the front lines","volume":"4","author":"Szymusiak","year":"2019","journal-title":"Pediatr Qual Saf"},{"key":"2026091000252482600_ooag165-B23","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1016\/j.jbi.2003.08.003","article-title":"Detecting adverse events for patient safety research: a review of current methodologies","volume":"36","author":"Murff","year":"2003","journal-title":"J Biomed Inform"},{"key":"2026091000252482600_ooag165-B24","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1136\/bmjqs-2017-006606","article-title":"Reasons for computerised provider order entry (CPOE)-based inpatient medication ordering errors: an observational study of voided orders","volume":"27","author":"Abraham","year":"2018","journal-title":"BMJ Qual Saf"},{"key":"2026091000252482600_ooag165-B25","doi-asserted-by":"publisher","first-page":"762","DOI":"10.1093\/jamia\/ocw187","article-title":"Learning from errors: analysis of medication order voiding in CPOE systems","volume":"24","author":"Kannampallil","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"2026091000252482600_ooag165-B26","volume-title":"Identifying and Characterizing CPOE-Related Order Errors in the Electronic Health Record","author":"Kneifati-Hayek","year":"2020"},{"key":"2026091000252482600_ooag165-B27","author":"Agency for Healthcare Research and Quality","year":"2019"},{"key":"2026091000252482600_ooag165-B28","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1093\/intqhc\/mzi021","article-title":"The JCAHO patient safety event taxonomy: a standardized terminology and classification schema for near misses and adverse events","volume":"17","author":"Chang","year":"2005","journal-title":"Int J Qual Health Care"},{"key":"2026091000252482600_ooag165-B29","doi-asserted-by":"publisher","first-page":"112","DOI":"10.1016\/j.ijmedinf.2017.12.022","article-title":"Prevalence of computerized physician order entry systems-related medication prescription errors: a systematic review","volume":"111","author":"Korb-Savoldelli","year":"2018","journal-title":"Int J Med Inform"},{"key":"2026091000252482600_ooag165-B30","doi-asserted-by":"publisher","first-page":"1281","DOI":"10.1001\/archinternmed.2011.327","article-title":"Defining health information technology-related errors: new developments since to err is human","volume":"171","author":"Sittig","year":"2011","journal-title":"Arch Intern Med"},{"key":"2026091000252482600_ooag165-B31","author":"National Coordinating Council for Medication Error Reporting and Prevention","year":"2022"},{"key":"2026091000252482600_ooag165-B32","volume-title":"Network of Patient Safety Databases Chartbook, 2021","author":"Agency for Healthcare Research and Quality","year":"2021"},{"key":"2026091000252482600_ooag165-B33","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1186\/s12916-020-01774-9","article-title":"Preventable medication harm across health care settings: a systematic review and meta-analysis","volume":"18","author":"Hodkinson","year":"2020","journal-title":"BMC Med"},{"key":"2026091000252482600_ooag165-B34","doi-asserted-by":"publisher","first-page":"838","DOI":"10.1093\/jamia\/ocad007","article-title":"Examining medication ordering errors using AHRQ network of patient safety databases","volume":"30","author":"Grauer","year":"2023","journal-title":"J Am Med Inform Assoc"},{"key":"2026091000252482600_ooag165-B35","doi-asserted-by":"publisher","first-page":"264","DOI":"10.1136\/bmjqs-2014-003555","article-title":"Computerised physician order entry-related medication errors: analysis of reported errors and vulnerability testing of current systems","volume":"24","author":"Schiff","year":"2015","journal-title":"BMJ Qual Saf"},{"key":"2026091000252482600_ooag165-B36","first-page":"39","article-title":"Defining medical error","volume":"48","author":"Grober","year":"2005","journal-title":"Can J Surg"},{"key":"2026091000252482600_ooag165-B37","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1136\/qshc.4.2.80","article-title":"Understanding adverse events: human factors","volume":"4","author":"Reason","year":"1995","journal-title":"Qual Health Care"},{"key":"2026091000252482600_ooag165-B38","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1016\/0376-6349(82)90041-4","article-title":"Human errors\u2013a taxonomy for describing human malfunction in industrial installations","volume":"4","author":"Rasmussen","year":"1982","journal-title":"J Occup Accid"},{"key":"2026091000252482600_ooag165-B39","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9781139062367","volume-title":"Human Error","author":"Reason","year":"1990"},{"key":"2026091000252482600_ooag165-B40","doi-asserted-by":"publisher","first-page":"1011","DOI":"10.2165\/00002018-200629110-00001","article-title":"Clarification of terminology in medication errors: definitions and classification","volume":"29","author":"Ferner","year":"2006","journal-title":"Drug Saf"},{"key":"2026091000252482600_ooag165-B41","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1214\/14-EJS909","article-title":"The cost of using exact confidence intervals for a binomial proportion","volume":"8","author":"Thulin","year":"2014","journal-title":"Electron J Statist"},{"key":"2026091000252482600_ooag165-B42","doi-asserted-by":"publisher","first-page":"1277","DOI":"10.1177\/1049732305276687","article-title":"Three approaches to qualitative content analysis","volume":"15","author":"Hsieh","year":"2005","journal-title":"Qual Health Res"},{"key":"2026091000252482600_ooag165-B43","doi-asserted-by":"publisher","first-page":"738","DOI":"10.7326\/m23-2063","article-title":"The safety of outpatient health care: review of electronic health records","volume":"177","author":"Levine","year":"2024","journal-title":"Ann Intern Med"},{"key":"2026091000252482600_ooag165-B44","doi-asserted-by":"publisher","first-page":"142","DOI":"10.1056\/NEJMsa2206117","article-title":"The safety of inpatient health care","volume":"388","author":"Bates","year":"2023","journal-title":"N Engl J Med"},{"key":"2026091000252482600_ooag165-B45","author":"Department of Health and Human Services (HHS)","year":"2018"},{"key":"2026091000252482600_ooag165-B46","author":"Executive Office of the President, President\u2019s Council of Advisors on Science and Technology","year":"2023"},{"key":"2026091000252482600_ooag165-B47","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1197\/jamia.M2042","article-title":"Types of unintended consequences related to computerized provider order entry","volume":"13","author":"Campbell","year":"2006","journal-title":"J Am Med Inform Assoc"},{"key":"2026091000252482600_ooag165-B48","doi-asserted-by":"publisher","first-page":"415","DOI":"10.1197\/jamia.M2373","article-title":"The extent and importance of unintended consequences related to computerized provider order entry","volume":"14","author":"Ash","year":"2007","journal-title":"J Am Med Inform Assoc"},{"key":"2026091000252482600_ooag165-B49","doi-asserted-by":"publisher","first-page":"S69","DOI":"10.1016\/j.ijmedinf.2008.07.015","article-title":"The unintended consequences of computerized provider order entry: findings from a mixed methods exploration","volume":"78 Suppl 1","author":"Ash","year":"2009","journal-title":"Int J Med Inform"},{"key":"2026091000252482600_ooag165-B50","doi-asserted-by":"publisher","DOI":"10.17226\/10863","volume-title":"Patient Safety: Achieving a New Standard for Care","author":"Institute of Medicine (US) Committee on Data Standards for Patient Safety","year":"2004"},{"key":"2026091000252482600_ooag165-B51","author":"Quality Interagency Coordination Task Force","year":"2000"},{"key":"2026091000252482600_ooag165-B52","author":"Leape","year":"2005"},{"key":"2026091000252482600_ooag165-B53","volume-title":"The Value of Close Calls in Improving Patient Safety","author":"Wu","year":"2011"}],"container-title":["JAMIA Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/5\/ooag165\/71013746\/ooag165.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/5\/ooag165\/71013746\/ooag165.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,10]],"date-time":"2026-09-10T04:25:29Z","timestamp":1789014329000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article\/doi\/10.1093\/jamiaopen\/ooag165\/8789715"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,9]]},"references-count":53,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2026,7,9]]}},"URL":"https:\/\/doi.org\/10.1093\/jamiaopen\/ooag165","relation":{},"ISSN":["2574-2531"],"issn-type":[{"value":"2574-2531","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,10]]},"published":{"date-parts":[[2026,7,9]]},"article-number":"ooag165"}}