{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T20:06:12Z","timestamp":1792785972117,"version":"4.5.0"},"reference-count":57,"publisher":"Oxford University Press (OUP)","license":[{"start":{"date-parts":[[2026,10,12]],"date-time":"2026-10-12T00:00:00Z","timestamp":1791763200000},"content-version":"am","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"name":"Agency for Healthcare Research and Quality","award":["#R01HS029020"],"award-info":[{"award-number":["#R01HS029020"]}],"id":[{"id":"https:\/\/ror.org\/03jmfdf59","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Electronic health record (EHR) audit logs capture clinician-EHR interaction patterns, but most audit log research relies on aggregated measures (eg, total time). We investigated how audit logs could be modeled using large language model (LLM) architectures to learn underlying workflow sequences during clinician-EHR interactions.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and methods<\/jats:title>\n                    <jats:p>Using &amp;gt;295 million EHR-based audit log actions from inpatient settings spanning 2019 to 2024, we fine-tuned Llama-3-8B under 2 encodings: (1) symbolic (field-based) tokens and (2) semantic (word-based) natural language-based audit log action descriptions. A first-order Markov model, using only the immediate prior action, served as a baseline minimal context comparator. Model representation was assessed using next-action prediction accuracy in an early-period test set and 2 temporally distinct out-of-sample (OOS) periods.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>In the early-period test set, the word-based LLM achieved the highest accuracy (0.7473; 95% CI, 0.7468-0.7478) compared to the field-based LLM (0.3464; 95% CI, 0.3456-0.3472) and Markov baseline (0.4230; 95% CI, 0.4223-0.4236). The word-based approach retained a substantial performance advantage over the Markov baseline across the OOS periods, although accuracy declined over time (word-based LLM vs Markov accuracy, OOS-1: 0.6555 vs 0.3139, OOS-2: 0.6286 vs 0.2720).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Word-based LLM relying on audit log action descriptions yielded the highest next-action prediction accuracy and demonstrated relative robustness to temporal drift, suggesting that longer sequential context and word-based action descriptions may improve audit log-based sequence modeling.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>These findings support the development of natural language-based LLM models for audit log research for task identification, workflow characterization, and safety-focused analyses of clinician-EHR interaction patterns.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocag163","type":"journal-article","created":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T11:14:28Z","timestamp":1789384468000},"source":"Crossref","is-referenced-by-count":0,"title":["A language model framework for sequence modeling of audit logs to characterize clinician\u2013electronic health record interactions"],"prefix":"10.1093","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9447-4888","authenticated-orcid":false,"given":"Seunghwan","family":"Kim","sequence":"first","affiliation":[{"name":"Institute for Informatics, Data Science, and Biostatistics, Washington University School of Medicine , St Louis, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4215-605X","authenticated-orcid":false,"given":"Sunny S","family":"Lou","sequence":"additional","affiliation":[{"name":"Department of Anesthesiology, Washington University School of Medicine , St Louis, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adam","family":"Cobb","sequence":"additional","affiliation":[{"name":"Computer Science Laboratory, SRI International , Menlo Park, CA 94027,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Susmit","family":"Jha","sequence":"additional","affiliation":[{"name":"Computer Science Laboratory, SRI International , Menlo Park, CA 94027,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4119-4836","authenticated-orcid":false,"given":"Thomas","family":"Kannampallil","sequence":"additional","affiliation":[{"name":"Institute for Informatics, Data Science, and Biostatistics, Washington University School of Medicine , St Louis, MO 63110,","place":["United States"]},{"name":"Department of Anesthesiology, Washington University School of Medicine , St Louis, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,10,12]]},"reference":[{"key":"2026101214320480700_ocag163-B1","doi-asserted-by":"publisher","first-page":"1709","DOI":"10.1001\/jama.2010.1497","article-title":"Meaningful use of electronic health records: the road ahead","volume":"304","author":"Jha","year":"2010","journal-title":"JAMA."},{"key":"2026101214320480700_ocag163-B2","doi-asserted-by":"publisher","first-page":"1628","DOI":"10.1056\/NEJMsa0900592","article-title":"Use of electronic health records in US hospitals","volume":"360","author":"Jha","year":"2009","journal-title":"N Engl J Med."},{"key":"2026101214320480700_ocag163-B3","doi-asserted-by":"publisher","first-page":"103343","DOI":"10.1016\/j.jbi.2019.103343","article-title":"EHR audit logs:a new goldmine for health services research?","volume":"101","author":"Adler-Milstein","year":"2020","journal-title":"J Biomed Inform."},{"key":"2026101214320480700_ocag163-B4","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1093\/jamia\/ocac173","article-title":"Using electronic health record audit log data for research: insights from early efforts","volume":"30","author":"Kannampallil","year":"2022","journal-title":"J Am Med Inform Assoc."},{"key":"2026101214320480700_ocag163-B5","doi-asserted-by":"publisher","first-page":"S116","DOI":"10.7326\/M19-0871","article-title":"Studying workflow and workarounds in electronic health record\u2014supported work to improve health system performance","volume":"172","author":"Zheng","year":"2020","journal-title":"Ann Intern Med."},{"key":"2026101214320480700_ocag163-B6","doi-asserted-by":"publisher","first-page":"396","DOI":"10.1145\/358886.358895","article-title":"The keystroke-level model for user performance time with interactive systems","volume":"23","author":"Card","year":"1980","journal-title":"Commun ACM."},{"key":"2026101214320480700_ocag163-B7","volume-title":"Understanding Computers and Cognition: A New Foundation for Design","author":"Winograd","year":"1986"},{"key":"2026101214320480700_ocag163-B8","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1207\/s15327051hci0903&4_6","article-title":"Characterizing the sequential structure of interactive behaviors through statistical and grammatical techniques","volume":"9","author":"Olson","year":"1994","journal-title":"Hum Comp Interact"},{"key":"2026101214320480700_ocag163-B9","doi-asserted-by":"publisher","first-page":"228","DOI":"10.1197\/jamia.M2852","article-title":"An interface-driven analysis of user interactions with an electronic health records system","volume":"16","author":"Zheng","year":"2009","journal-title":"J Am Med Inform Assoc"},{"key":"2026101214320480700_ocag163-B10","doi-asserted-by":"publisher","first-page":"480","DOI":"10.1093\/jamia\/ocz196","article-title":"Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods","volume":"27","author":"Rule","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2026101214320480700_ocag163-B11","doi-asserted-by":"publisher","first-page":"784","DOI":"10.1093\/jamia\/ocad254","article-title":"Guidance for reporting analyses of metadata on electronic health record use","volume":"31","author":"Rule","year":"2024","journal-title":"J Am Med Inform Assoc."},{"key":"2026101214320480700_ocag163-B12","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1093\/jamia\/ocac177","article-title":"Using event logs to observe interactions with electronic health records: an updated scoping review shows increasing use of vendor-derived measures","volume":"30","author":"Rule","year":"2022","journal-title":"J Am Med Inform Assoc."},{"key":"2026101214320480700_ocag163-B13","author":"Magon"},{"key":"2026101214320480700_ocag163-B14","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1370\/afm.2121","article-title":"Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations","volume":"15","author":"Arndt","year":"2017","journal-title":"Ann Fam Med."},{"key":"2026101214320480700_ocag163-B15","doi-asserted-by":"publisher","first-page":"2165","DOI":"10.1007\/s11606-022-07620-3","article-title":"Temporal associations between EHR-derived workload, burnout, and errors: a prospective cohort study","volume":"37","author":"Lou","year":"2022","journal-title":"J Gen Intern Med."},{"key":"2026101214320480700_ocag163-B16","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1093\/jamia\/ocab253","article-title":"Characterizing styles of clinical note production and relationship to clinical work hours among first-year residents","volume":"29","author":"Gong","year":"2021","journal-title":"J Am Med Inform Assoc."},{"key":"2026101214320480700_ocag163-B17","doi-asserted-by":"publisher","first-page":"e232","DOI":"10.1016\/j.jsurg.2021.08.011","article-title":"Estimation of surgical resident duty hours and workload in real time using electronic health record data","volume":"78","author":"Lin","year":"2021","journal-title":"J Surg Educ."},{"key":"2026101214320480700_ocag163-B18","doi-asserted-by":"publisher","first-page":"737","DOI":"10.1055\/s-0041-1733851","article-title":"Measuring electronic health record use in the pediatric ICU using audit-logs and screen recordings","volume":"12","author":"Sinha","year":"2021","journal-title":"Appl Clin Inform."},{"key":"2026101214320480700_ocag163-B19","doi-asserted-by":"publisher","first-page":"e143","DOI":"10.1055\/s-0040-1716411","article-title":"Electronic health record use among ophthalmology residents while on call","volume":"12","author":"Long","year":"2020","journal-title":"J Acad Ophthalmol (2017)."},{"key":"2026101214320480700_ocag163-B20","first-page":"573","article-title":"Methods for large-scale quantitative analysis of scribe impacts on clinical documentation","volume":"2020","author":"Hribar","year":"2020","journal-title":"AMIA Annu Symp Proc."},{"key":"2026101214320480700_ocag163-B21","doi-asserted-by":"publisher","first-page":"404","DOI":"10.12788\/jhm.3607","article-title":"Automating measurement of trainee work hours","volume":"16","author":"Soleimani","year":"2021","journal-title":"J Hosp Med."},{"key":"2026101214320480700_ocag163-B22","doi-asserted-by":"publisher","first-page":"e0226493","DOI":"10.1371\/journal.pone.0226493","article-title":"Pediatric trainees systematically under-report duty hour violations compared to electronic health record defined shifts","volume":"14","author":"Dziorny","year":"2019","journal-title":"PLoS One."},{"key":"2026101214320480700_ocag163-B23","first-page":"490","article-title":"Analysis of total time requirements of electronic health record use by ophthalmologists using secondary EHR Data","volume":"2018","author":"Goldstein","year":"2018","journal-title":"AMIA Annu Symp Proc."},{"key":"2026101214320480700_ocag163-B24","doi-asserted-by":"publisher","first-page":"e22","DOI":"10.1016\/j.bja.2022.04.012","article-title":"Effect of clinician attention switching on workload and wrong-patient errors","volume":"129","author":"Lou","year":"2022","journal-title":"Br J Anaesth."},{"key":"2026101214320480700_ocag163-B25","doi-asserted-by":"publisher","first-page":"e2031856","DOI":"10.1001\/jamanetworkopen.2020.31856","article-title":"Evaluation of attention switching and duration of electronic inbox work among primary care physicians","volume":"4","author":"Lieu","year":"2021","journal-title":"JAMA Netw Open."},{"key":"2026101214320480700_ocag163-B26","doi-asserted-by":"publisher","first-page":"637","DOI":"10.1055\/s-0041-1731678","article-title":"Novel nonproprietary measures of ambulatory electronic health record use associated with physician work exhaustion","volume":"12","author":"Sinha","year":"2021","journal-title":"Appl Clin Inform"},{"key":"2026101214320480700_ocag163-B27","doi-asserted-by":"publisher","first-page":"104015","DOI":"10.1016\/j.jbi.2022.104015","article-title":"Predicting physician burnout using clinical activity logs: model performance and lessons learned","volume":"127","author":"Lou","year":"2022","journal-title":"J Biomed Inform."},{"key":"2026101214320480700_ocag163-B28","first-page":"3377","author":"Liu"},{"key":"2026101214320480700_ocag163-B29","doi-asserted-by":"publisher","first-page":"960","DOI":"10.1093\/jamia\/ocab053","article-title":"Associations of physician burnout with organizational electronic health record support and after-hours charting","volume":"28","author":"Eschenroeder","year":"2021","journal-title":"J Am Med Inform Assoc."},{"key":"2026101214320480700_ocag163-B30","doi-asserted-by":"publisher","first-page":"398","DOI":"10.1016\/j.mayocp.2022.10.027","article-title":"Relationships between EHR-based audit log data and physician burnout and clinical practice process measures","volume":"98","author":"Dyrbye","year":"2023","journal-title":"Mayo Clin Proc."},{"key":"2026101214320480700_ocag163-B31","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1093\/jamia\/ocz220","article-title":"Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians","volume":"27","author":"Adler-Milstein","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2026101214320480700_ocag163-B32","doi-asserted-by":"publisher","first-page":"1780","DOI":"10.1001\/jama.2019.3698","article-title":"Effect of restriction of the number of concurrently open records in an electronic health record on wrong-patient order errors: a randomized clinical trial","volume":"321","author":"Adelman","year":"2019","journal-title":"JAMA."},{"key":"2026101214320480700_ocag163-B33","doi-asserted-by":"publisher","first-page":"739","DOI":"10.1093\/jamia\/ocx099","article-title":"Effect of number of open charts on intercepted wrong-patient medication orders in an emergency department","volume":"25","author":"Kannampallil","year":"2018","journal-title":"J Am Med Inform Assoc."},{"key":"2026101214320480700_ocag163-B34","doi-asserted-by":"publisher","first-page":"e2328514","DOI":"10.1001\/jamanetworkopen.2023.28514","article-title":"Anesthesia clinical workload estimated from electronic health record documentation vs billed relative value units","volume":"6","author":"Lou","year":"2023","journal-title":"JAMA Netw Open."},{"key":"2026101214320480700_ocag163-B35","doi-asserted-by":"publisher","first-page":"103312","DOI":"10.1016\/j.jbi.2019.103312","article-title":"Using the PARAFAC2 tensor factorization on EHR audit data to understand PCP desktop work","volume":"101","author":"Perros","year":"2020","journal-title":"J Biomed Inform."},{"key":"2026101214320480700_ocag163-B36","first-page":"612","article-title":"Learning tasks of pediatric providers from electronic health record audit logs","volume":"2020","author":"Jones","year":"2020","journal-title":"AMIA Annu Symp Proc"},{"key":"2026101214320480700_ocag163-B37","doi-asserted-by":"publisher","first-page":"1168","DOI":"10.1093\/jamia\/ocaa338","article-title":"Mining tasks and task characteristics from electronic health record audit logs with unsupervised machine learning","volume":"28","author":"Chen","year":"2021","journal-title":"J Am Med Inform Assoc."},{"key":"2026101214320480700_ocag163-B38","doi-asserted-by":"publisher","first-page":"539","DOI":"10.1093\/jamia\/ocac239","article-title":"Characterizing the macrostructure of electronic health record work using raw audit logs: an unsupervised action embeddings approach","volume":"30","author":"Lou","year":"2023","journal-title":"J Am Med Inform Assoc."},{"key":"2026101214320480700_ocag163-B39","doi-asserted-by":"publisher","first-page":"e24","DOI":"10.37765\/ajmc.2023.89310","article-title":"Classifying clinical work settings using EHR audit logs: a machine learning approach","volume":"29","author":"Kim","year":"2023","journal-title":"Am J Manag Care."},{"key":"2026101214320480700_ocag163-B40","author":"Kim"},{"key":"2026101214320480700_ocag163-B41","doi-asserted-by":"publisher","first-page":"2670","DOI":"10.1093\/jamia\/ocab211","article-title":"Predicting next-day discharge via electronic health record access logs","volume":"28","author":"Zhang","year":"2021","journal-title":"J Am Med Inform Assoc."},{"key":"2026101214320480700_ocag163-B42","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1016\/j.jbi.2006.06.002","article-title":"Workflow modeling in critical care: piecing together your own puzzle","volume":"40","author":"Malhotra","year":"2007","journal-title":"J Biomed Inform."},{"key":"2026101214320480700_ocag163-B43","author":"Warner","year":"2023"},{"key":"2026101214320480700_ocag163-B44","doi-asserted-by":"publisher","first-page":"2228","DOI":"10.1093\/jamia\/ocae171","article-title":"Measuring cognitive effort using tabular transformer-based language models of electronic health record-based audit log action sequences","volume":"31","author":"Kim","year":"2024","journal-title":"J Am Med Inform Assoc."},{"key":"2026101214320480700_ocag163-B45","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1136\/bmjqs-2017-006606","article-title":"Reasons for computerised provider order entry (CPOE)-based inpatient medication ordering errors: an observational study of voided orders","volume":"27","author":"Abraham","year":"2018","journal-title":"BMJ Qual Saf."},{"key":"2026101214320480700_ocag163-B46","doi-asserted-by":"publisher","first-page":"762","DOI":"10.1093\/jamia\/ocw187","article-title":"Learning from errors: analysis of medication order voiding in CPOE systems","volume":"24","author":"Kannampallil","year":"2017","journal-title":"J Am Med Inform Assoc."},{"key":"2026101214320480700_ocag163-B47","doi-asserted-by":"publisher","first-page":"1441","DOI":"10.1177\/1049732311411927","article-title":"The nurse\u2019s medication day","volume":"21","author":"Jennings","year":"2011","journal-title":"Qual Health Res."},{"key":"2026101214320480700_ocag163-B48","author":"Grattafiori","year":"2024"},{"key":"2026101214320480700_ocag163-B49","author":"Hu","year":"2021"},{"key":"2026101214320480700_ocag163-B50","doi-asserted-by":"publisher","first-page":"10088","DOI":"10.52202\/075280-0441","article-title":"Qlora: efficient finetuning of quantized llms","volume":"36","author":"Dettmers","year":"2024","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026101214320480700_ocag163-B51","author":"AI@Meta. Llama-3 Model Card. meta-llama [Computer software]"},{"key":"2026101214320480700_ocag163-B52"},{"key":"2026101214320480700_ocag163-B53","author":"Lukas","year":"2020"},{"key":"2026101214320480700_ocag163-B54","author":"Boer"},{"key":"2026101214320480700_ocag163-B55","doi-asserted-by":"publisher","first-page":"246","DOI":"10.1002\/asi.10367","article-title":"The effects of domain knowledge on search tactic formulation","volume":"55","author":"Wildemuth","year":"2004","journal-title":"J Am Soc Inf Sci."},{"key":"2026101214320480700_ocag163-B56","doi-asserted-by":"publisher","first-page":"260","DOI":"10.1037\/0033-295X.84.3.260","article-title":"Evolution of cognitive structures and processes","volume":"84","author":"Hayes-Roth","year":"1977","journal-title":"Psychol Rev."},{"key":"2026101214320480700_ocag163-B57","author":"Waxler","year":"2025"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag163\/71880820\/ocag163.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag163\/71880820\/ocag163.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,12]],"date-time":"2026-10-12T18:32:14Z","timestamp":1791829934000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article\/doi\/10.1093\/jamia\/ocag163\/8910679"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,12]]},"references-count":57,"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocag163","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,12]]},"article-number":"ocag163"}}