{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T18:11:58Z","timestamp":1783447918582,"version":"3.55.0"},"reference-count":24,"publisher":"Oxford University Press (OUP)","issue":"10","license":[{"start":{"date-parts":[[2024,7,17]],"date-time":"2024-07-17T00:00:00Z","timestamp":1721174400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>This work presents the development and evaluation of coordn8, a web-based application that streamlines fax processing in outpatient clinics using a \u201chuman-in-the-loop\u201d machine learning framework. We demonstrate the effectiveness of the platform at reducing fax processing time and producing accurate machine learning inferences across the tasks of patient identification, document classification, spam classification, and duplicate document detection.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>We deployed coordn8 in 11 outpatient clinics and conducted a time savings analysis by observing users and measuring fax processing event logs. We used statistical methods to evaluate the machine learning components across different datasets to show generalizability. We conducted a time series analysis to show variations in model performance as new clinics were onboarded and to demonstrate our approach to mitigating model drift.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Our observation analysis showed a mean reduction in individual fax processing time by 147.5 s, while our event log analysis of over 7000 faxes reinforced this finding. Document classification produced an accuracy of 81.6%, patient identification produced an accuracy of 83.7%, spam classification produced an accuracy of 98.4%, and duplicate document detection produced a precision of 81.0%. Retraining document classification increased accuracy by 10.2%.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>coordn8 significantly decreased fax-processing time and produced accurate machine learning inferences. Our human-in-the-loop framework facilitated the collection of high-quality data necessary for model training. Expanding to new clinics correlated with performance decline, which was mitigated through model retraining.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>Our framework for automating clinical tasks with machine learning offers a template for health systems looking to implement similar technologies.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae194","type":"journal-article","created":{"date-parts":[[2024,7,17]],"date-time":"2024-07-17T19:22:15Z","timestamp":1721244135000},"page":"2236-2245","source":"Crossref","is-referenced-by-count":3,"title":["The incremental design of a machine learning framework for medical records processing"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0684-4354","authenticated-orcid":false,"given":"Christopher","family":"Streiffer","sequence":"first","affiliation":[{"name":"Department of Medicine, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Divya","family":"Saini","sequence":"additional","affiliation":[{"name":"Department of Medicine, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gideon","family":"Whitehead","sequence":"additional","affiliation":[{"name":"Center for Health Care Transformation and Innovation, University of Pennsylvania , Philadelphia, PA 19104, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jency","family":"Daniel","sequence":"additional","affiliation":[{"name":"Center for Health Care Transformation and Innovation, University of Pennsylvania , Philadelphia, PA 19104, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carolina","family":"Garzon-Mrad","sequence":"additional","affiliation":[{"name":"Center for Health Care Transformation and Innovation, University of Pennsylvania , Philadelphia, PA 19104, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laura","family":"Kavanaugh","sequence":"additional","affiliation":[{"name":"Center for Health Care Transformation and Innovation, University of Pennsylvania , Philadelphia, PA 19104, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emeka","family":"Anyanwu","sequence":"additional","affiliation":[{"name":"Department of Medicine, Perelman School of Medicine, University of Pennsylvania , Philadelphia, PA 19104, United States"},{"name":"Center for Health Care Transformation and Innovation, University of Pennsylvania , Philadelphia, PA 19104, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,7,17]]},"reference":[{"issue":"5","key":"2024092007523771400_ocae194-B1","first-page":"CAT. 22.0438","article-title":"In-basket reduction: a multiyear pragmatic approach to lessen the work burden of primary care physicians","volume":"4","author":"Fogg","year":"2023","journal-title":"NEJM Catalyst Innovations in Care Delivery"},{"key":"2024092007523771400_ocae194-B2","first-page":"572","article-title":"EHRs: the challenge of making electronic data usable and interoperable","volume":"42","author":"Reisman","year":"2017","journal-title":"Pharm.Ther"},{"key":"2024092007523771400_ocae194-B3","author":"Minor","year":"2019"},{"key":"2024092007523771400_ocae194-B4","author":"Editorial Team","year":"2019"},{"issue":"1-2","key":"2024092007523771400_ocae194-B5","doi-asserted-by":"crossref","first-page":"50","DOI":"10.31128\/AFP-07-17-4285","article-title":"eReferrals: why are we still faxing?","volume":"47","author":"Hughes","year":"2018","journal-title":"Aust J Gen Pract"},{"key":"2024092007523771400_ocae194-B6","author":"Anjum","year":"2023"},{"issue":"5","key":"2024092007523771400_ocae194-B7","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1370\/afm.2121","article-title":"Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations","volume":"15","author":"Arndt","year":"2017","journal-title":"Ann Fam Med"},{"key":"2024092007523771400_ocae194-B8","author":"Christiano","year":"2023"},{"issue":"2","key":"2024092007523771400_ocae194-B9","doi-asserted-by":"crossref","first-page":"ooac045","DOI":"10.1093\/jamiaopen\/ooac045","article-title":"Deep learning-based NLP data pipeline for EHR-scanned document information extraction","volume":"5","author":"Hsu","year":"2022","journal-title":"JAMIA Open"},{"issue":"5","key":"2024092007523771400_ocae194-B10","doi-asserted-by":"crossref","first-page":"831","DOI":"10.1093\/jamia\/ocac007","article-title":"Closing the loop: automatically identifying abnormal imaging results in scanned documents","volume":"29","author":"Kumar","year":"2022","journal-title":"J Am Med Inform Assoc"},{"key":"2024092007523771400_ocae194-B11","doi-asserted-by":"crossref","first-page":"104302","DOI":"10.1016\/j.ijmedinf.2020.104302","article-title":"Automatic classification of scanned electronic health record documents","volume":"144","author":"Goodrum","year":"2020","journal-title":"Int J Med Inform"},{"issue":"2","key":"2024092007523771400_ocae194-B12","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1136\/amiajnl-2011-000456","article-title":"Importance of multi-modal approaches to effectively identify cataract cases from electronic health records","volume":"19","author":"Peissig","year":"2012","journal-title":"J Am Med Inform Assoc"},{"key":"2024092007523771400_ocae194-B13","author":"Eikvil","year":"1993"},{"issue":"1","key":"2024092007523771400_ocae194-B14","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1075\/li.30.1.03nad","article-title":"A survey of named entity recognition and classification","volume":"30","author":"Nadeau","year":"2007","journal-title":"LI"},{"issue":"1","key":"2024092007523771400_ocae194-B15","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1017\/S1351324916000334","article-title":"Word2Vec","volume":"23","author":"Church","year":"2017","journal-title":"Nat Lang Eng"},{"issue":"4","key":"2024092007523771400_ocae194-B16","doi-asserted-by":"crossref","first-page":"285","DOI":"10.21512\/comtech.v7i4.3746","article-title":"Single document automatic text summarization using term frequency-inverse document frequency (TF-IDF)","volume":"7","author":"Christian","year":"2016","journal-title":"ComTech"},{"issue":"3","key":"2024092007523771400_ocae194-B17","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1093\/jamiaopen\/ooaa036","article-title":"Design and development of referrals automation, a SMART on FHIR solution to improve patient access to specialty care","volume":"3","author":"Odisho","year":"2020","journal-title":"JAMIA Open"},{"issue":"9","key":"2024092007523771400_ocae194-B18","doi-asserted-by":"crossref","first-page":"1631","DOI":"10.1093\/jamia\/ocac078","article-title":"A framework for the oversight and local deployment of safe and high-quality prediction models","volume":"29","author":"Bedoya","year":"2022","journal-title":"J Am Med Inform Assoc"},{"key":"2024092007523771400_ocae194-B19","author":"Chen","year":"2016"},{"key":"2024092007523771400_ocae194-B20","author":"Xu","year":"2004"},{"issue":"3","key":"2024092007523771400_ocae194-B21","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1109\/TBDATA.2019.2921572","article-title":"Billion-scale similarity search with gpus","volume":"7","author":"Johnson","year":"2021","journal-title":"IEEE Trans Big Data"},{"issue":"1","key":"2024092007523771400_ocae194-B22","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1038\/s41746-022-00742-2","article-title":"A large language model for electronic health records","volume":"5","author":"Yang","year":"2022","journal-title":"NPJ Digit Med"},{"issue":"8","key":"2024092007523771400_ocae194-B23","doi-asserted-by":"crossref","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2024092007523771400_ocae194-B24","author":"Achiam","year":"2023"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/10\/2236\/59206305\/ocae194.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/10\/2236\/59206305\/ocae194.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T07:53:11Z","timestamp":1726818791000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/10\/2236\/7715993"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,17]]},"references-count":24,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2024,7,17]]},"published-print":{"date-parts":[[2024,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae194","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,10]]},"published":{"date-parts":[[2024,7,17]]}}}