{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,31]],"date-time":"2025-10-31T22:02:36Z","timestamp":1761948156442,"version":"3.37.3"},"reference-count":37,"publisher":"Oxford University Press (OUP)","issue":"1","funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"crossref","award":["R01LM011966"],"award-info":[{"award-number":["R01LM011966"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2017,1,1]]},"abstract":"<jats:p>Thorough and ongoing testing of electronic health records (EHRs) is key to ensuring their safety and effectiveness. Many health care organizations limit testing to test environments separate from, and often different than, the production environment used by clinicians. Because EHRs are complex hardware and software systems that often interact with other hardware and software systems, no test environment can exactly mimic how the production environment will behave. An effective testing process must integrate safely conducted testing in the production environment itself, using test patients. We propose recommendations for how to safely incorporate testing in production into current EHR testing practices, with suggestions regarding the incremental release of upgrades, test patients, tester accounts, downstream personnel, and reporting.<\/jats:p>","DOI":"10.1093\/jamia\/ocw039","type":"journal-article","created":{"date-parts":[[2016,4,24]],"date-time":"2016-04-24T00:18:19Z","timestamp":1461457099000},"page":"188-192","source":"Crossref","is-referenced-by-count":20,"title":["Testing electronic health records in the \u201cproduction\u201d environment: an essential step in the journey to a safe and effective health care system"],"prefix":"10.1093","volume":"24","author":[{"given":"Adam","family":"Wright","sequence":"first","affiliation":[{"name":"Brigham and Women\u2019s Hospital, Boston, MA, USA"},{"name":"Harvard Medical School, Harvard University, Boston, MA, USA"},{"name":"Partners HealthCare, Boston, MA, USA"}]},{"given":"Skye","family":"Aaron","sequence":"additional","affiliation":[{"name":"Brigham and Women\u2019s Hospital, Boston, MA, USA"}]},{"given":"Dean F","family":"Sittig","sequence":"additional","affiliation":[{"name":"University of Texas Health Science Center, University of Texas, Houston, TX, USA"}]}],"member":"286","published-online":{"date-parts":[[2016,4,23]]},"reference":[{"volume-title":"Health IT and Patient Safety: Building Safer Systems for Better Care","year":"2012","author":"Institute of Medicine","key":"2020110612370439800_ocw039-B1"},{"volume-title":"Health IT Safety Center Roadmap","year":"2015","author":"RTI International","key":"2020110612370439800_ocw039-B2"},{"volume-title":"Health Information Technology Patient Safety Action and Surveillance Plan","year":"2013","author":"Office of the National Coordinator for Health Information Technology","key":"2020110612370439800_ocw039-B3"},{"volume-title":"Recent Evidence that Health IT Improves Patient Safety","author":"Office of the National Coordinator for Health Information Technology","key":"2020110612370439800_ocw039-B4"},{"issue":"3","key":"2020110612370439800_ocw039-B5","doi-asserted-by":"crossref","first-page":"464","DOI":"10.1377\/hlthaff.2011.0178","article-title":"The benefits of health information technology: a review of the recent literature shows predominantly positive results","volume":"30","author":"Buntin","year":"2011","journal-title":"Health Aff (Millwood)."},{"issue":"4","key":"2020110612370439800_ocw039-B6","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1136\/jamia.2001.0080299","article-title":"Reducing the frequency of errors in medicine using information technology","volume":"8","author":"Bates","year":"2001","journal-title":"J Am Med Inform Assoc."},{"issue":"25","key":"2020110612370439800_ocw039-B7","doi-asserted-by":"crossref","first-page":"2526","DOI":"10.1056\/NEJMsa020847","article-title":"Improving safety with information technology","volume":"348","author":"Bates","year":"2003","journal-title":"N Engl J Med."},{"issue":"4","key":"2020110612370439800_ocw039-B8","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1197\/jamia.M2373","article-title":"The extent and importance of unintended consequences related to computerized provider order entry","volume":"14","author":"Ash","year":"2007","journal-title":"J Am Med Inform Assoc."},{"issue":"5","key":"2020110612370439800_ocw039-B9","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1197\/jamia.M2042","article-title":"Types of unintended consequences related to computerized provider order entry","volume":"13","author":"Campbell","year":"2006","journal-title":"J Am Med Inform Assoc."},{"issue":"1","key":"2020110612370439800_ocw039-B10","doi-asserted-by":"crossref","first-page":"\u00a0e187","DOI":"10.1136\/amiajnl-2012-001494","article-title":"The hazard of software updates to clinical workstations: a natural experiment","volume":"20","author":"Landman","year":"2013","journal-title":"J Am Med Inform Assoc."},{"issue":"5","key":"2020110612370439800_ocw039-B11","first-page":"418","article-title":"The SAFER guides: empowering organizations to improve the safety and effectiveness of electronic health records","volume":"20","author":"Sittig","year":"2014","journal-title":"Am J Manag Care."},{"article-title":"SAFER Guides","year":"2014","author":"Sittig","key":"2020110612370439800_ocw039-B12"},{"article-title":"High Priority Practices SAFER Guide","year":"2014","author":"Office of the National Coordinator for Health Information Technology","key":"2020110612370439800_ocw039-B13"},{"key":"2020110612370439800_ocw039-B14","volume-title":"Human Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications","author":"Jacko","year":"2007","edition":"2nd ed."},{"issue":"3","key":"2020110612370439800_ocw039-B15","article-title":"Critical conversations about optimal design column: thorough error testing a requirement for strong EHR usability","volume":"18","author":"Keenan","year":"2014","journal-title":"OJNI"},{"key":"2020110612370439800_ocw039-B16","first-page":"341","article-title":"Development of methods for usability evaluations of EHR systems","volume":"124","author":"Lilholt","year":"2006","journal-title":"Stud Health Technol Inform."},{"volume-title":"Go-Live: Smart Strategies from Davis Award-Winning EHR Implementations","year":"2011","author":"Schulte","key":"2020110612370439800_ocw039-B17"},{"volume-title":"Experiences of Test Automation: Case Studies of Software Test Automation","year":"2012","author":"Graham","key":"2020110612370439800_ocw039-B18"},{"issue":"3","key":"2020110612370439800_ocw039-B19","doi-asserted-by":"crossref","first-page":"284","DOI":"10.4338\/ACI-2011-02-RA-0012","article-title":"Comparison of computer-based clinical decision support systems and content for diabetes mellitus","volume":"2","author":"Kantor","year":"2011","journal-title":"Appl Clin Inform."},{"issue":"1","key":"2020110612370439800_ocw039-B20","doi-asserted-by":"crossref","first-page":"80","DOI":"10.4338\/ACI-2011-11-RA-0066","article-title":"Clinical summarization capabilities of commercially-available and internally-developed electronic health records","volume":"3","author":"Laxmisan","year":"2012","journal-title":"Appl Clin Inform."},{"issue":"5","key":"2020110612370439800_ocw039-B21","doi-asserted-by":"crossref","first-page":"1081","DOI":"10.1093\/jamia\/ocv073","article-title":"Cross-vendor evaluation of key user-defined clinical decision support capabilities: a scenario-based assessment of certified electronic health records with guidelines for future development","volume":"22","author":"McCoy","year":"2015","journal-title":"J Am Med Inform Assoc."},{"issue":"4","key":"2020110612370439800_ocw039-B22","doi-asserted-by":"crossref","first-page":"900","DOI":"10.1093\/jamia\/ocv013","article-title":"Graphical display of diagnostic test results in electronic health records: a comparison of 8 systems","volume":"22","author":"Sittig","year":"2015","journal-title":"J Am Med Inform Assoc."},{"issue":"2","key":"2020110612370439800_ocw039-B23","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1136\/jamia.2009.002030","article-title":"Governance for clinical decision support: case studies and recommended practices from leading institutions","volume":"18","author":"Wright","year":"2011","journal-title":"J Am Med Inform Assoc."},{"issue":"5","key":"2020110612370439800_ocw039-B24","doi-asserted-by":"crossref","first-page":"637","DOI":"10.1197\/jamia.M3111","article-title":"Clinical decision support capabilities of commercially-available clinical information systems","volume":"16","author":"Wright","year":"2009","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612370439800_ocw039-B25","first-page":"\u00a0181","article-title":"Integrating technology-centric and user-centric system testing methods: ensuring healthcare system usability and safety","volume":"157","author":"Kushniruk","year":"2010","journal-title":"Stud Health Technol Inform"},{"volume-title":"Lessons Learned in Software Testing: a Context-driven Approach","year":"2002","author":"Kaner","key":"2020110612370439800_ocw039-B26"},{"issue":"1","key":"2020110612370439800_ocw039-B27","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1197\/jamia.M3294","article-title":"Unintended errors with EHR-based result management: a case series","volume":"17","author":"Yackel","year":"2010","journal-title":"J Am Med Inform Assoc."},{"key":"2020110612370439800_ocw039-B28","doi-asserted-by":"crossref","DOI":"10.1093\/jamia\/ocw005","article-title":"Analysis of clinical decision support system malfunctions: a case series and survey","author":"Wright","year":"2016","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"2020110612370439800_ocw039-B29","doi-asserted-by":"crossref","first-page":"1053","DOI":"10.1136\/amiajnl-2013-002578","article-title":"An analysis of electronic health record-related patient safety concerns","volume":"21","author":"Meeks","year":"2014","journal-title":"J Am Med Inform Assoc."},{"author":"Beckwith","key":"2020110612370439800_ocw039-B30","article-title":"Laboratory Interoperability Best Practices: Ten Mistakes to Avoid. March 2013"},{"key":"2020110612370439800_ocw039-B31","article-title":"Usability testing and the relation of clinical information systems to patient safety","volume-title":"Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology)","author":"Rogers","year":"2005"},{"volume-title":"Frequently Occurring Surnames from the Census 2000","year":"2014","author":"United States Census Bureau","key":"2020110612370439800_ocw039-B32"},{"journal-title":"Patient Identification SAFER Guide","year":"2014","author":"Office of the National Coordinator for Health Information Technology","key":"2020110612370439800_ocw039-B33"},{"author":"Institute for Safe Medication Practices (ISMP)","key":"2020110612370439800_ocw039-B34"},{"journal-title":"SAIC","article-title":"Testing overview for a multi-application implementation","author":"Bishop","key":"2020110612370439800_ocw039-B35"},{"issue":"6","key":"2020110612370439800_ocw039-B36","doi-asserted-by":"crossref","first-page":"1188","DOI":"10.1007\/s11096-013-9848-y","article-title":"Detection and correct handling of prescribing errors in Dutch hospital pharmacies using test patients","volume":"35","author":"Beex-Oosterhuis","year":"2013","journal-title":"Int J Clin Pharm."},{"volume-title":"Test Procedure for \u00a7170.314(c) Clinical Quality Measures","year":"2013","author":"Office of the National Coordinator for Health Information Technology","key":"2020110612370439800_ocw039-B37"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/24\/1\/188\/34147973\/ocw039.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/24\/1\/188\/34147973\/ocw039.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,11,6]],"date-time":"2020-11-06T17:46:59Z","timestamp":1604684819000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/24\/1\/188\/2631439"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,4,23]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2016,4,23]]},"published-print":{"date-parts":[[2017,1,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocw039","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"type":"print","value":"1067-5027"},{"type":"electronic","value":"1527-974X"}],"subject":[],"published-other":{"date-parts":[[2017,1]]},"published":{"date-parts":[[2016,4,23]]}}}