{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,9]],"date-time":"2025-05-09T18:45:17Z","timestamp":1746816317757,"version":"3.28.0"},"reference-count":32,"publisher":"Georg Thieme Verlag KG","issue":"05","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl Clin Inform"],"published-print":{"date-parts":[[2022,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Usage log data are an important data source for characterizing the potential burden related to use of the electronic health record (EHR) system. However, the utility of this data source has been hindered by concerns related to the real-world validity and accuracy of the data. While time\u2013motion studies have historically been used to address this concern, the restrictions caused by the pandemic have made it difficult to carry out these studies in-person. In this regard, we introduce a practical approach for conducting validation studies for usage log data in a controlled environment. By developing test runs based on clinical workflows and conducting them within a test EHR environment, it allows for both comparison of the recorded timings and retrospective investigation of any discrepancies. In this case report, we describe the utility of this approach for validating our physician EHR usage logs at a large academic teaching mental health hospital in Canada. A total of 10 test runs were conducted across 3 days to validate 8 EHR usage log metrics, finding differences between recorded measurements and the usage analytics platform ranging from 9 to 60%.<\/jats:p>","DOI":"10.1055\/s-0042-1756424","type":"journal-article","created":{"date-parts":[[2022,10,5]],"date-time":"2022-10-05T22:38:38Z","timestamp":1665009518000},"page":"928-934","source":"Crossref","is-referenced-by-count":2,"title":["Accuracy of Physician Electronic Health Record Usage Analytics using Clinical Test Cases"],"prefix":"10.1055","volume":"13","author":[{"given":"Brian","family":"Lo","sequence":"additional","affiliation":[{"name":"Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada"},{"name":"Centre for Complex Interventions (Digital Interventions Unit), Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Canada"},{"name":"Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lydia","family":"Sequeira","sequence":"additional","affiliation":[{"name":"Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada"},{"name":"Centre for Complex Interventions (Digital Interventions Unit), Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Canada"},{"name":"Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gillian","family":"Strudwick","sequence":"additional","affiliation":[{"name":"Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada"},{"name":"Centre for Complex Interventions (Digital Interventions Unit), Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Canada"},{"name":"Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Damian","family":"Jankowicz","sequence":"additional","affiliation":[{"name":"Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Khaled","family":"Almilaji","sequence":"additional","affiliation":[{"name":"Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anjchuca","family":"Karunaithas","sequence":"additional","affiliation":[{"name":"Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada"},{"name":"Department of Health and Society, University of Toronto Scarborough, Scarborough, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dennis","family":"Hang","sequence":"additional","affiliation":[{"name":"Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada"},{"name":"Health Information Science, University of Victoria, Victoria, British Columbia, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tania","family":"Tajirian","sequence":"additional","affiliation":[{"name":"Information Management Group, Centre for Addiction and Mental Health, Toronto, Canada"},{"name":"Department of Family and Community Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"194","published-online":{"date-parts":[[2022,10,5]]},"reference":[{"issue":"05","key":"ref1","doi-asserted-by":"crossref","first-page":"998","DOI":"10.1093\/jamia\/ocaa325","article-title":"Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review","volume":"28","author":"A J Moy","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"01","key":"ref2","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1055\/s-0037-1621705","article-title":"Efficiency of emergency physicians: insights from an observational study using EHR log files","volume":"9","author":"T G Kannampallil","year":"2018","journal-title":"Appl Clin Inform"},{"issue":"04","key":"ref3","doi-asserted-by":"crossref","first-page":"639","DOI":"10.1093\/jamia\/ocz223","article-title":"Metrics for assessing physician activity using electronic health record log data","volume":"27","author":"C A Sinsky","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"03","key":"ref4","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1055\/s-0041-1731679","article-title":"Using log data to measure provider EHR activity at a cancer center during rapid telemedicine deployment","volume":"12","author":"C Moore","year":"2021","journal-title":"Appl Clin Inform"},{"issue":"04","key":"ref5","doi-asserted-by":"crossref","first-page":"598","DOI":"10.1055\/s-0040-1715829","article-title":"Registered nurse strain detection using ambient data: an exploratory study of underutilized operational data streams in the hospital workplace","volume":"11","author":"D M Womack","year":"2020","journal-title":"Appl Clin Inform"},{"issue":"08","key":"ref6","doi-asserted-by":"crossref","first-page":"1632","DOI":"10.1093\/jamia\/ocab059","article-title":"The association between perceived electronic health record usability and professional burnout among US nurses","volume":"28","author":"E R Melnick","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"03","key":"ref7","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1016\/j.mayocp.2019.09.024","article-title":"The association between perceived electronic health record usability and professional burnout among US physicians","volume":"95","author":"E R Melnick","year":"2020","journal-title":"Mayo Clin Proc"},{"issue":"07","key":"ref8","doi-asserted-by":"crossref","first-page":"e19274","DOI":"10.2196\/19274","article-title":"The influence of electronic health record use on physician burnout: cross-sectional survey","volume":"22","author":"T Tajirian","year":"2020","journal-title":"J Med Internet Res"},{"issue":"09","key":"ref9","doi-asserted-by":"crossref","first-page":"1004","DOI":"10.1007\/s11606-016-3720-4","article-title":"Worklife and wellness in academic general internal medicine: results from a national survey","volume":"31","author":"M Linzer","year":"2016","journal-title":"J Gen Intern Med"},{"issue":"05","key":"ref10","doi-asserted-by":"crossref","first-page":"960","DOI":"10.1093\/jamia\/ocab053","article-title":"Associations of physician burnout with organizational electronic health record support and after-hours charting","volume":"28","author":"H C Eschenroeder","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"03","key":"ref11","doi-asserted-by":"crossref","first-page":"169","DOI":"10.7326\/M18-3684","article-title":"Physician time spent using the electronic health record during outpatient encounters: a descriptive study","volume":"172","author":"J M Overhage","year":"2020","journal-title":"Ann Intern Med"},{"issue":"06","key":"ref12","doi-asserted-by":"crossref","first-page":"e20194017","DOI":"10.1542\/peds.2019-4017","article-title":"Pediatrician electronic health record time use for outpatient encounters","volume":"146","author":"J M Overhage","year":"2020","journal-title":"Pediatrics"},{"issue":"04","key":"ref13","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1097\/CEH.0000000000000268","article-title":"Examining associations between physician data utilization for practice improvement and lifelong learning","volume":"39","author":"S Sockalingam","year":"2019","journal-title":"J Contin Educ Health Prof"},{"issue":"02","key":"ref14","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1055\/s-0041-1728699","article-title":"Multidisciplinary sprint program achieved specialty-specific EHR optimization in 20 clinics","volume":"12","author":"A Sieja","year":"2021","journal-title":"Appl Clin Inform"},{"issue":"03","key":"ref15","doi-asserted-by":"crossref","first-page":"e25148","DOI":"10.2196\/25148","article-title":"Beyond Getting Rid of Stupid Stuff in the Electronic Health Record (Beyond-GROSS): protocol for a user-centered, mixed-method intervention to improve the electronic health record system","volume":"10","author":"A U Otokiti","year":"2021","journal-title":"JMIR Res Protoc"},{"issue":"02","key":"ref16","doi-asserted-by":"crossref","first-page":"ooab018","DOI":"10.1093\/jamiaopen\/ooab018","article-title":"EHR \u201cSWAT\u201d teams: a physician engagement initiative to improve electronic health record (EHR) experiences and mitigate possible causes of EHR-related burnout","volume":"4","author":"L Sequeira","year":"2021","journal-title":"JAMIA Open"},{"issue":"05","key":"ref17","doi-asserted-by":"crossref","first-page":"1032","DOI":"10.1093\/jamia\/ocaa305","article-title":"Conceptual considerations for using EHR-based activity logs to measure clinician burnout and its effects","volume":"28","author":"T Kannampallil","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"ref18","doi-asserted-by":"crossref","first-page":"103343","DOI":"10.1016\/j.jbi.2019.103343","article-title":"EHR audit logs: a new goldmine for health services research?","volume":"101","author":"J Adler-Milstein","year":"2020","journal-title":"J Biomed Inform"},{"issue":"06","key":"ref19","doi-asserted-by":"crossref","first-page":"e97","DOI":"10.1016\/j.jsurg.2018.10.010","article-title":"Documenting or operating: where is time spent in general surgery residency?","volume":"75","author":"M L Cox","year":"2018","journal-title":"J Surg Educ"},{"issue":"02","key":"ref20","doi-asserted-by":"crossref","first-page":"e0205379","DOI":"10.1371\/journal.pone.0205379","article-title":"Characterizing electronic health record usage patterns of inpatient medicine residents using event log data","volume":"14","author":"J K Wang","year":"2019","journal-title":"PLoS One"},{"issue":"11","key":"ref21","doi-asserted-by":"crossref","first-page":"1467","DOI":"10.1001\/jamainternmed.2018.3956","article-title":"Association of medical scribes in primary care with physician workflow and patient experience","volume":"178","author":"P Mishra","year":"2018","journal-title":"JAMA Intern Med"},{"key":"ref22","first-page":"136","article-title":"Burnout and EHR use among academic primary care physicians with varied clinical workloads","volume":"2019","author":"B Tran","year":"2019","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"issue":"03","key":"ref23","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1093\/intqhc\/mzs011","article-title":"New paradigms for measuring clinical performance using electronic health records","volume":"24","author":"J P Weiner","year":"2012","journal-title":"Int J Qual Health Care"},{"issue":"03","key":"ref24","first-page":"1102","article-title":"Evaluating the reliability of EHR-generated clinical outcomes reports: a case study","volume":"2","author":"C Kanger","year":"2014","journal-title":"EGEMS (Wash DC)"},{"issue":"03","key":"ref25","doi-asserted-by":"crossref","first-page":"480","DOI":"10.1093\/jamia\/ocz196","article-title":"Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods","volume":"27","author":"A Rule","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"11","key":"ref27","doi-asserted-by":"crossref","first-page":"1250","DOI":"10.1001\/jamaophthalmol.2017.4187","article-title":"Time requirements for electronic health record use in an academic ophthalmology center","volume":"135","author":"S Read-Brown","year":"2017","journal-title":"JAMA Ophthalmol"},{"issue":"04","key":"ref28","doi-asserted-by":"crossref","first-page":"655","DOI":"10.1377\/hlthaff.2016.0811","article-title":"Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine","volume":"36","author":"M Tai-Seale","year":"2017","journal-title":"Health Aff (Millwood)"},{"issue":"05","key":"ref29","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1370\/afm.2121","article-title":"Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations","volume":"15","author":"B G Arndt","year":"2017","journal-title":"Ann Fam Med"},{"issue":"04","key":"ref30","doi-asserted-by":"crossref","first-page":"721","DOI":"10.1016\/j.jbi.2013.05.007","article-title":"Privacy-preserving screen capture: towards closing the loop for health IT usability","volume":"46","author":"J Cooley","year":"2013","journal-title":"J Biomed Inform"},{"issue":"04","key":"ref32","doi-asserted-by":"crossref","first-page":"737","DOI":"10.1055\/s-0041-1733851","article-title":"Measuring electronic health record use in the pediatric ICU using audit-logs and screen recordings","volume":"12","author":"A Sinha","year":"2021","journal-title":"Appl Clin Inform"},{"key":"ref33","first-page":"1909","volume-title":"Secondary use of EHR timestamp data: validation and application for workflow optimization","author":"M R Hribar","year":"2015"},{"issue":"01","key":"ref34","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1186\/s12916-018-1079-6","article-title":"Using flawed, uncertain, proximate and sparse (FUPS) data in the context of complexity: learning from the case of child mental health","volume":"16","author":"M Wolpert","year":"2018","journal-title":"BMC Med"}],"container-title":["Applied Clinical Informatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.thieme-connect.de\/products\/ejournals\/pdf\/10.1055\/s-0042-1756424.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,5]],"date-time":"2024-11-05T13:59:31Z","timestamp":1730815171000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.thieme-connect.de\/DOI\/DOI?10.1055\/s-0042-1756424"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10]]},"references-count":32,"journal-issue":{"issue":"05","published-online":{"date-parts":[[2022,10,5]]},"published-print":{"date-parts":[[2022,10]]}},"URL":"https:\/\/doi.org\/10.1055\/s-0042-1756424","archive":["Portico","CLOCKSS"],"relation":{},"ISSN":["1869-0327"],"issn-type":[{"type":"electronic","value":"1869-0327"}],"subject":[],"published":{"date-parts":[[2022,10]]}}}