{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T06:20:24Z","timestamp":1770186024869,"version":"3.49.0"},"posted":{"date-parts":[[2025,6,24]]},"group-title":"Health Informatics","reference-count":37,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T00:00:00Z","timestamp":1750723200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2025,9,26]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                <jats:sec>\n                  <jats:title>Background<\/jats:title>\n                  <jats:p>Natural language processing (NLP) allows efficient extraction of clinical variables and outcomes from electronic health records (EHR). However, measuring pragmatic clinical trial outcomes may demand accuracy that exceeds NLP performance. Combining NLP with human adjudication can address this gap, yet few software solutions support such workflows. We developed a modular, scalable system for NLP-screened human abstraction to measure the primary outcomes of two clinical trials.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>In two clinical trials of hospitalized patients with serious illness, a deep-learning NLP model screened EHR passages for documented goals-of-care discussions. Screen-positive passages were referred for human adjudication using a REDCap-based system to measure the trial outcomes. Dynamic pooling of passages using structured query language (SQL) within the REDCap database reduced unnecessary abstraction while ensuring data completeness.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>In the first trial (N=2,512), NLP identified 22,187 screen-positive passages (0.8%) from 2.6 million EHR passages. Human reviewers adjudicated 7,494 passages over 34.3 abstractor-hours to measure the cumulative incidence and time to first documented goals-of-care discussion for all patients with 92.6% patient-level sensitivity. In the second trial (N=617), NLP identified 8,952 screen-positive passages (1.6%) from 559,596 passages at a threshold with near-100% sensitivity. Human reviewers adjudicated 3,509 passages over 27.9 abstractor-hours to measure the same outcome for all patients.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>We present the design and source code for a scalable and efficient pipeline for measuring complex EHR-derived outcomes using NLP-screened human abstraction. This implementation is adaptable to diverse research needs, and its modular pipeline represents a practical middle ground between custom software and commercial platforms.<\/jats:p>\n                <\/jats:sec>","DOI":"10.1101\/2025.06.23.25330134","type":"posted-content","created":{"date-parts":[[2025,6,25]],"date-time":"2025-06-25T01:35:20Z","timestamp":1750815320000},"source":"Crossref","is-referenced-by-count":0,"title":["A modular pipeline for natural language processing-screened human abstraction of a pragmatic trial outcome from electronic health records"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5535-287X","authenticated-orcid":false,"given":"Robert Y.","family":"Lee","sequence":"first","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington, Seattle, USA"},{"name":"Cambia Palliative Care Center of Excellence at UW Medicine, University of Washington, Seattle, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kevin S.","family":"Li","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington, Seattle, USA"},{"name":"Division of Biomedical and Health Informatics, Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"James","family":"Sibley","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington, Seattle, USA"},{"name":"Department of Biobehavioral Nursing and Health Informatics, University of Washington, Seattle, WA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Trevor","family":"Cohen","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington, Seattle, USA"},{"name":"Division of Biomedical and Health Informatics, Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"William B.","family":"Lober","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington, Seattle, USA"},{"name":"Division of Biomedical and Health Informatics, Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA"},{"name":"Department of Biobehavioral Nursing and Health Informatics, University of Washington, Seattle, WA"},{"name":"Department of Global Health, University of Washington, Seattle, WA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Janaki","family":"O\u2019Brien","sequence":"additional","affiliation":[{"name":"Cambia Palliative Care Center of Excellence at UW Medicine, University of Washington, Seattle, USA"},{"name":"Norcliffe Foundation Center for Integrative Brain Research, Seattle Children\u2019s Research Institute, Seattle Children\u2019s Hospital, Seattle, WA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicole","family":"LeDuc","sequence":"additional","affiliation":[{"name":"Cambia Palliative Care Center of Excellence at UW Medicine, University of Washington, Seattle, USA"},{"name":"Project ECHO, University of New Mexico Health Sciences Center, Albuquerque, NM"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kasey","family":"Mallon Andrews","sequence":"additional","affiliation":[{"name":"Cambia Palliative Care Center of Excellence at UW Medicine, University of Washington, Seattle, USA"},{"name":"Department of Surgery, University of Washington, Seattle, WA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anna","family":"Ungar","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington, Seattle, USA"},{"name":"Cambia Palliative Care Center of Excellence at UW Medicine, University of Washington, Seattle, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jessica","family":"Walsh","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington, Seattle, USA"},{"name":"Cambia Palliative Care Center of Excellence at UW Medicine, University of Washington, Seattle, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elizabeth L.","family":"Nielsen","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington, Seattle, USA"},{"name":"Cambia Palliative Care Center of Excellence at UW Medicine, University of Washington, Seattle, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Danae G.","family":"Dotolo","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington, Seattle, USA"},{"name":"Cambia Palliative Care Center of Excellence at UW Medicine, University of Washington, Seattle, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Erin K.","family":"Kross","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington, Seattle, USA"},{"name":"Cambia Palliative Care Center of Excellence at UW Medicine, University of Washington, Seattle, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"54368","reference":[{"key":"2026020309263434000_2025.06.23.25330134v2.1","doi-asserted-by":"publisher","DOI":"10.2196\/12239"},{"key":"2026020309263434000_2025.06.23.25330134v2.2","doi-asserted-by":"publisher","DOI":"10.2196\/17984"},{"key":"2026020309263434000_2025.06.23.25330134v2.3","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2022.5088"},{"key":"2026020309263434000_2025.06.23.25330134v2.4","doi-asserted-by":"publisher","DOI":"10.1016\/j.cct.2022.106879"},{"key":"2026020309263434000_2025.06.23.25330134v2.5","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2023.8812"},{"key":"2026020309263434000_2025.06.23.25330134v2.6","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-019-05446-0"},{"key":"2026020309263434000_2025.06.23.25330134v2.7","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2019.0059"},{"key":"2026020309263434000_2025.06.23.25330134v2.8","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2021.12.020"},{"key":"2026020309263434000_2025.06.23.25330134v2.9","doi-asserted-by":"publisher","DOI":"10.1016\/J.JPAINSYMMAN.2020.08.024"},{"key":"2026020309263434000_2025.06.23.25330134v2.10","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2022.02.006"},{"key":"2026020309263434000_2025.06.23.25330134v2.11","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.1204"},{"key":"2026020309263434000_2025.06.23.25330134v2.12","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2024.0496"},{"key":"2026020309263434000_2025.06.23.25330134v2.13","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2008.08.010"},{"key":"2026020309263434000_2025.06.23.25330134v2.14","unstructured":"Institute of Medicine. Dying in America: Improving Quality and Honoring Individual Preferences Near the End of Life. National Academies Press; 2015."},{"key":"2026020309263434000_2025.06.23.25330134v2.15","doi-asserted-by":"publisher","DOI":"10.1046\/j.1532-5415.2002.50116.x"},{"key":"2026020309263434000_2025.06.23.25330134v2.16","doi-asserted-by":"publisher","DOI":"10.1200\/JCO.2009.25.4672"},{"key":"2026020309263434000_2025.06.23.25330134v2.17","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-007-0362-6"},{"key":"2026020309263434000_2025.06.23.25330134v2.18","doi-asserted-by":"publisher","DOI":"10.1016\/S2213-2600(19)30087-6"},{"key":"2026020309263434000_2025.06.23.25330134v2.19","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2014.5271"},{"key":"2026020309263434000_2025.06.23.25330134v2.20","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2020.5179"},{"key":"2026020309263434000_2025.06.23.25330134v2.21","doi-asserted-by":"publisher","DOI":"10.1097\/00003246-200102001-00006"},{"key":"2026020309263434000_2025.06.23.25330134v2.22","doi-asserted-by":"publisher","DOI":"10.1513\/AnnalsATS.201612-965OI"},{"key":"2026020309263434000_2025.06.23.25330134v2.23","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2012.0317"},{"key":"2026020309263434000_2025.06.23.25330134v2.24","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2014.12.007"},{"key":"2026020309263434000_2025.06.23.25330134v2.25","doi-asserted-by":"publisher","DOI":"10.1503\/cmaj.121274"},{"key":"2026020309263434000_2025.06.23.25330134v2.26","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2021.23695"},{"key":"2026020309263434000_2025.06.23.25330134v2.27","doi-asserted-by":"crossref","unstructured":"Chien I , Shi A , Chan A , Lindvall C. Identification of Serious Illness Conversations in Unstructured Clinical Notes Using Deep Neural Networks. In: Fernando Koch , Andrew Koster , Isabelle Bichindaritz , Pau Herrero , et al , eds. Artificial Intelligence in Health: First International Workshop, AIH 2018, Stockholm, Sweden, July 13-14, 2018, Revised Selected Papers. Lecture Notes in Artificial Intelligence. Springer Nature Switzerland; 2019:199\u2013212.","DOI":"10.1007\/978-3-030-12738-1_15"},{"key":"2026020309263434000_2025.06.23.25330134v2.28","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2012.0472"},{"key":"2026020309263434000_2025.06.23.25330134v2.29","unstructured":"Alsentzer E , Murphy JR , Boag W , et al. Publicly Available Clinical BERT Embeddings. arXiv. Preprint posted online June 20, 2019. Accessed August 29, 2024. https:\/\/arxiv.org\/abs\/1904.03323"},{"key":"2026020309263434000_2025.06.23.25330134v2.30","unstructured":"Alsentzer E. BioClinicalBERT. 2019. https:\/\/huggingface.co\/emilyalsentzer\/Bio_ClinicalBERT"},{"key":"2026020309263434000_2025.06.23.25330134v2.31","unstructured":"Devlin J , Chang MW , Lee K , Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv. Preprint posted online May 24, 2019. Accessed August 29, 2024. http:\/\/arxiv.org\/abs\/1810.04805"},{"key":"2026020309263434000_2025.06.23.25330134v2.32","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2018.0294"},{"key":"2026020309263434000_2025.06.23.25330134v2.33","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2021.06.025"},{"key":"2026020309263434000_2025.06.23.25330134v2.34","unstructured":"Vanderkam D. Local Turk. 2024. Accessed January 22, 2025. https:\/\/github.com\/danvk\/localturk"},{"key":"2026020309263434000_2025.06.23.25330134v2.35","unstructured":"Lee RY , Li KS , Sibley J , et al. NLP-screened EHR passage adjudication. REDCap Shared Library. January 22, 2025. https:\/\/redcap.vumc.org\/consortium\/library\/search.php (requires session authentication via local REDCap project)"},{"key":"2026020309263434000_2025.06.23.25330134v2.36","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2012.10.006"},{"key":"2026020309263434000_2025.06.23.25330134v2.37","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-5010"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2025.06.23.25330134","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,3]],"date-time":"2026-02-03T17:26:45Z","timestamp":1770139605000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.1101\/2025.06.23.25330134"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,24]]},"references-count":37,"URL":"https:\/\/doi.org\/10.1101\/2025.06.23.25330134","relation":{},"subject":[],"published":{"date-parts":[[2025,6,24]]},"subtype":"preprint"}}