{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T01:39:53Z","timestamp":1772156393238,"version":"3.50.1"},"reference-count":35,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2022,12,8]],"date-time":"2022-12-08T00:00:00Z","timestamp":1670457600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100007812","name":"Washington University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100007812","id-type":"DOI","asserted-by":"crossref"}]},{"name":"BJC Healthcare Big Ideas Healthcare Innovation Competition"},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["5T32GM108539-07"],"award-info":[{"award-number":["5T32GM108539-07"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,2,16]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Raw audit logs provide a comprehensive record of clinicians\u2019 activities on an electronic health record (EHR) and have considerable potential for studying clinician behaviors. However, research using raw audit logs is limited because they lack context for clinical tasks, leading to difficulties in interpretation. We describe a novel unsupervised approach using the comparison and visualization of EHR action embeddings to learn context and structure from raw audit log activities. Using a dataset of 15\u00a0767\u00a0634 raw audit log actions performed by 88 intern physicians over 6 months of EHR use across inpatient and outpatient settings, we demonstrated that embeddings can be used to learn the situated context for EHR-based work activities, identify discrete clinical workflows, and discern activities typically performed across diverse contexts. Our approach represents an important methodological advance in raw audit log research, facilitating the future development of metrics and predictive models to measure clinician behaviors at the macroscale.<\/jats:p>","DOI":"10.1093\/jamia\/ocac239","type":"journal-article","created":{"date-parts":[[2022,12,8]],"date-time":"2022-12-08T14:50:48Z","timestamp":1670511048000},"page":"539-544","source":"Crossref","is-referenced-by-count":11,"title":["Characterizing the macrostructure of electronic health record work using raw audit logs: an unsupervised action embeddings approach"],"prefix":"10.1093","volume":"30","author":[{"given":"Sunny S","family":"Lou","sequence":"first","affiliation":[{"name":"Department of Anesthesiology, School of Medicine, Washington University in St Louis , St Louis, Missouri, USA"},{"name":"Institute for Informatics, School of Medicine, Washington University in St Louis , St Louis, Missouri, USA"}]},{"given":"Hanyang","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, McKelvey School of Engineering, Washington University in St Louis , St Louis, Missouri, USA"}]},{"given":"Derek","family":"Harford","sequence":"additional","affiliation":[{"name":"Department of Anesthesiology, School of Medicine, Washington University in St Louis , St Louis, Missouri, USA"}]},{"given":"Chenyang","family":"Lu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, McKelvey School of Engineering, Washington University in St Louis , St Louis, Missouri, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4119-4836","authenticated-orcid":false,"given":"Thomas","family":"Kannampallil","sequence":"additional","affiliation":[{"name":"Department of Anesthesiology, School of Medicine, Washington University in St Louis , St Louis, Missouri, USA"},{"name":"Institute for Informatics, School of Medicine, Washington University in St Louis , St Louis, Missouri, USA"},{"name":"Department of Computer Science and Engineering, McKelvey School of Engineering, Washington University in St Louis , St Louis, Missouri, USA"}]}],"member":"286","published-online":{"date-parts":[[2022,12,8]]},"reference":[{"key":"2023021611061046600_","author":"Office of the National Coordinator for Health Information Technology","year":"2019"},{"key":"2023021611061046600_","author":"Office of the National Coordinator for Health Information Technology"},{"issue":"2","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1001\/jamainternmed.2015.6831","article-title":"Internal medicine resident computer usage: an electronic audit of an inpatient service","volume":"176","author":"Ouyang","year":"2016","journal-title":"JAMA Intern Med"},{"issue":"4","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"655","DOI":"10.1377\/hlthaff.2016.0811","article-title":"Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine","volume":"36","author":"Tai-Seale","year":"2017","journal-title":"Health Aff (Millwood)"},{"issue":"5","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1370\/afm.2121","article-title":"Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations","volume":"15","author":"Arndt","year":"2017","journal-title":"Ann Fam Med"},{"key":"2023021611061046600_","author":"Rossetti","year":"2021"},{"issue":"5","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"974","DOI":"10.1093\/jamia\/ocaa339","article-title":"A systematic review of contributing factors of and solutions to electronic health record-related impacts on physician well-being","volume":"28","author":"Nguyen","year":"2021","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"998","DOI":"10.1093\/jamia\/ocaa325","article-title":"Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review","volume":"28","author":"Moy","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"103343","DOI":"10.1016\/j.jbi.2019.103343","article-title":"EHR audit logs: a new goldmine for health services research?","volume":"101","author":"Adler-Milstein","year":"2020","journal-title":"J Biomed Inform"},{"issue":"11 Suppl","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"S116","DOI":"10.7326\/M19-0871","article-title":"Studying workflow and workarounds in electronic health record-supported work to improve health system performance","volume":"172","author":"Zheng","year":"2020","journal-title":"Ann Intern Med"},{"key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"480","DOI":"10.1093\/jamia\/ocz196","article-title":"Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods","author":"Rule","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"639","DOI":"10.1093\/jamia\/ocz223","article-title":"Metrics for assessing physician activity using electronic health record log data","volume":"27","author":"Sinsky","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"644","DOI":"10.1093\/jamia\/ocz222","article-title":"Have you got the time? Challenges using vendor electronic health record metrics of provider efficiency","volume":"27","author":"Hron","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"9","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"2165","DOI":"10.1007\/s11606-022-07620-3","article-title":"Temporal associations between EHR-derived workload, burnout, and errors: a prospective cohort study","volume":"37","author":"Lou","year":"2022","journal-title":"J Gen Intern Med"},{"key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"104015","DOI":"10.1016\/j.jbi.2022.104015","article-title":"Predicting physician burnout using clinical activity logs: model performance and lessons learned","volume":"127","author":"Lou","year":"2022","journal-title":"J Biomed Inform"},{"key":"2023021611061046600_","doi-asserted-by":"publisher","author":"Mikolov","year":"2013","DOI":"10.48550\/arXiv.1301.3781"},{"issue":"2\u20133","key":"2023021611061046600_","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1080\/00437956.1954.11659520","article-title":"Distributional structure","volume":"10","author":"Harris","year":"1954","journal-title":"WORD"},{"key":"2023021611061046600_","doi-asserted-by":"publisher","author":"Mikolov","year":"2013","DOI":"10.48550\/arXiv.1310.4546"},{"key":"2023021611061046600_","doi-asserted-by":"publisher","DOI":"10.1016\/j.bja.2022.04.012","article-title":"Effect of clinician attention switching on workload and wrong-patient errors","author":"Lou","year":"2022","journal-title":"Br J Anaesth"},{"key":"2023021611061046600_","doi-asserted-by":"publisher","first-page":"280","DOI":"10.18653\/v1\/D15-1033","author":"Sergienya","year":"2015"},{"key":"2023021611061046600_","first-page":"746","author":"Mikolov","year":"2013"},{"issue":"86","key":"2023021611061046600_","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"L van der","year":"2008","journal-title":"J Mach Learn Res"},{"issue":"9","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"506","DOI":"10.5694\/j.1326-5377.2008.tb01762.x","article-title":"All in a day\u2019s work: an observational study to quantify how and with whom doctors on hospital wards spend their time","volume":"188","author":"Westbrook","year":"2008","journal-title":"Med J Aust"},{"issue":"9","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"1382","DOI":"10.1001\/jamainternmed.2017.2668","article-title":"Multitasking and silent electronic health record use in ambulatory visits","volume":"177","author":"Ratanawongsa","year":"2017","journal-title":"JAMA Intern Med"},{"issue":"6","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"e97","DOI":"10.1016\/j.jsurg.2018.10.010","article-title":"Documenting or operating: where is time spent in general surgery residency?","volume":"75","author":"Cox","year":"2018","journal-title":"J Surg Educ"},{"issue":"1","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1055\/s-0038-1676819","article-title":"Automatic detection of front-line clinician hospital shifts: a novel use of electronic health record timestamp data","volume":"10","author":"Dziorny","year":"2019","journal-title":"Appl Clin Inform"},{"issue":"2","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1197\/jamia.M2852","article-title":"An interface-driven analysis of user interactions with an electronic health records system","volume":"16","author":"Zheng","year":"2009","journal-title":"J Am Med Inform Assoc"},{"key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"103312","DOI":"10.1016\/j.jbi.2019.103312","article-title":"Using the PARAFAC2 tensor factorization on EHR audit data to understand PCP desktop work","volume":"101","author":"Perros","year":"2020","journal-title":"J Biomed Inform"},{"key":"2023021611061046600_","first-page":"612","article-title":"Learning tasks of pediatric providers from electronic health record audit logs","volume":"2020","author":"Jones","year":"2021","journal-title":"AMIA Annu Symp Proc"},{"issue":"6","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"1168","DOI":"10.1093\/jamia\/ocaa338","article-title":"Mining tasks and task characteristics from electronic health record audit logs with unsupervised machine learning","volume":"28","author":"Chen","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1093\/jamia\/ocab253","article-title":"Characterizing styles of clinical note production and relationship to clinical work hours among first-year residents","author":"Gong","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"104004","DOI":"10.1016\/j.jbi.2022.104004","article-title":"Signal from the noise: a mixed graphical and quantitative process mining approach to evaluate care pathways applied to emergency stroke care","volume":"127","author":"Noshad","year":"2022","journal-title":"J Biomed Inform"},{"issue":"9","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"e28998","DOI":"10.2196\/28998","article-title":"Measuring collaboration through concurrent electronic health record usage: network analysis study","volume":"9","author":"Li","year":"2021","journal-title":"JMIR Med Inform"},{"issue":"1","key":"2023021611061046600_","doi-asserted-by":"crossref","first-page":"e2031856","DOI":"10.1001\/jamanetworkopen.2020.31856","article-title":"Evaluation of attention switching and duration of electronic inbox work among primary care physicians","volume":"4","author":"Lieu","year":"2021","journal-title":"JAMA Netw Open"},{"key":"2023021611061046600_","doi-asserted-by":"publisher","author":"Liu","year":"2022","DOI":"10.1145\/3534678.3539056"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/3\/539\/49198803\/ocac239.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/3\/539\/49198803\/ocac239.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,14]],"date-time":"2023-03-14T19:50:11Z","timestamp":1678823411000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/30\/3\/539\/6883974"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,8]]},"references-count":35,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,12,8]]},"published-print":{"date-parts":[[2023,2,16]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocac239","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,3,1]]},"published":{"date-parts":[[2022,12,8]]}}}