{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T10:15:45Z","timestamp":1777544145961,"version":"3.51.4"},"reference-count":37,"publisher":"SAGE Publications","issue":"2","license":[{"start":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T00:00:00Z","timestamp":1770854400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["K12HL137940"],"award-info":[{"award-number":["K12HL137940"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["K23HL161503"],"award-info":[{"award-number":["K23HL161503"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["T32HL125195"],"award-info":[{"award-number":["T32HL125195"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000049","name":"National Institute on Aging","doi-asserted-by":"publisher","award":["R01AG062441"],"award-info":[{"award-number":["R01AG062441"]}],"id":[{"id":"10.13039\/100000049","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100014367","name":"National Palliative Care Research Center","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100014367","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006108","name":"National Center for Advancing Translational Sciences","doi-asserted-by":"publisher","award":["UL1TR002319"],"award-info":[{"award-number":["UL1TR002319"]}],"id":[{"id":"10.13039\/100006108","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100015348","name":"Cambia Health Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100015348","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Clinical Trials"],"published-print":{"date-parts":[[2026,4]]},"abstract":"<jats:sec>\n                    <jats:title>Background:<\/jats:title>\n                    <jats:p>Natural language processing allows efficient extraction of clinical variables and outcomes from electronic health records (EHRs). However, measuring pragmatic clinical trial outcomes may demand accuracy that exceeds natural language processing performance. Combining natural language processing with human adjudication can address this gap, yet few software solutions support such workflows. We developed a modular, scalable system for natural language processing-screened human abstraction to measure the primary outcomes of two clinical trials.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods:<\/jats:title>\n                    <jats:p>In two clinical trials of hospitalized patients with serious illness, a deep-learning natural language processing model screened electronic health record passages for documented goals-of-care discussions. Screen-positive passages were referred for human adjudication using a REDCap-based system to measure the trial outcomes. Dynamic pooling of passages using structured query language within the REDCap database reduced unnecessary abstraction while ensuring data completeness.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results:<\/jats:title>\n                    <jats:p>In the first trial (N\u2009=\u20092512), natural language processing identified 22,187 screen-positive passages (0.8%) from 2.6 million electronic health record passages. Human reviewers adjudicated 7494 passages over 34.3 abstractor-hours to measure the cumulative incidence and time to first documented goals-of-care discussion for all patients with 92.6% patient-level sensitivity. In the second trial (N\u2009=\u2009617), natural language processing identified 8952 screen-positive passages (1.6%) from 559,596 passages at a threshold with near-100% sensitivity. Human reviewers adjudicated 3509 passages over 27.9 abstractor-hours to measure the same outcome for all patients.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion:<\/jats:title>\n                    <jats:p>We present the design and source code for a scalable and efficient pipeline for measuring complex electronic health record-derived outcomes using natural language processing-screened human abstraction. This implementation is adaptable to diverse research needs, and its modular pipeline represents a practical middle ground between custom software and commercial platforms.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1177\/17407745251405386","type":"journal-article","created":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T02:42:35Z","timestamp":1770950555000},"page":"145-154","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["A modular pipeline for natural language processing-screened human abstraction of a pragmatic trial outcome from electronic health records"],"prefix":"10.1177","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5535-287X","authenticated-orcid":false,"given":"Robert Y","family":"Lee","sequence":"first","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington School of Medicine, Seattle, WA, USA"},{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-0786-1600","authenticated-orcid":false,"given":"Kevin S","family":"Li","sequence":"additional","affiliation":[{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"},{"name":"Division of Biomedical and Health Informatics, Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"James","family":"Sibley","sequence":"additional","affiliation":[{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"},{"name":"Department of Biobehavioral Nursing and Health Informatics, University of Washington, Seattle, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Trevor","family":"Cohen","sequence":"additional","affiliation":[{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"},{"name":"Division of Biomedical and Health Informatics, Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"William B","family":"Lober","sequence":"additional","affiliation":[{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"},{"name":"Division of Biomedical and Health Informatics, Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, USA"},{"name":"Department of Biobehavioral Nursing and Health Informatics, University of Washington, Seattle, WA, USA"},{"name":"Department of Global Health, University of Washington, Seattle, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Janaki","family":"O\u2019Brien","sequence":"additional","affiliation":[{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"},{"name":"Norcliffe Foundation Center for Integrative Brain Research, Seattle Children\u2019s Research Institute, Seattle Children\u2019s Hospital, Seattle, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nicole","family":"LeDuc","sequence":"additional","affiliation":[{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"},{"name":"Project ECHO, University of New Mexico Health Sciences Center, Albuquerque, NM, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kasey","family":"Mallon Andrews","sequence":"additional","affiliation":[{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"},{"name":"Department of Surgery, University of Washington, Seattle, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anna","family":"Ungar","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington School of Medicine, Seattle, WA, USA"},{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jessica","family":"Walsh","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington School of Medicine, Seattle, WA, USA"},{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elizabeth L","family":"Nielsen","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington School of Medicine, Seattle, WA, USA"},{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5838-4834","authenticated-orcid":false,"given":"Danae G","family":"Dotolo","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington School of Medicine, Seattle, WA, USA"},{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Erin K","family":"Kross","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care, and Sleep Medicine, University of Washington School of Medicine, Seattle, WA, USA"},{"name":"Cambia Palliative Care Center of Excellence, UW Medicine, University of Washington School of Medicine, WA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2026,2,12]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.2196\/12239"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.2196\/17984"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2022.5088"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cct.2022.106879"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2023.8812"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-019-05446-0"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2019.0059"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2021.12.020"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2020.08.024"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2022.02.006"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.1204"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2024.0496"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2008.08.010"},{"key":"e_1_3_3_15_2","volume-title":"Dying in America: improving quality and honoring individual preferences near the end of life","author":"Institute of Medicine","year":"2015","unstructured":"Institute of Medicine. Dying in America: improving quality and honoring individual preferences near the end of life. Washington, DC: National Academies Press, 2015."},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1046\/j.1532-5415.2002.50116.x"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1200\/JCO.2009.25.4672"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-007-0362-6"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/S2213-2600(19)30087-6"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2014.5271"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2020.5179"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1097\/00003246-200102001-00006"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1513\/AnnalsATS.201612-965OI"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2012.0317"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2014.12.007"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1503\/cmaj.121274"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2021.23695"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-12738-1_15"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2012.0472"},{"key":"e_1_3_3_30_2","unstructured":"Alsentzer E Murphy JR Boag W et al. Publicly available clinical BERT embeddings. arXiv. https:\/\/arxiv.org\/abs\/1904.03323 (2019 accessed 29 August 2024)."},{"key":"e_1_3_3_31_2","unstructured":"Alsentzer E. BioClinicalBERT 2019. https:\/\/huggingface.co\/emilyalsentzer\/Bio_ClinicalBERT"},{"key":"e_1_3_3_32_2","unstructured":"Devlin J Chang MW Lee K et al. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv. http:\/\/arxiv.org\/abs\/1810.04805 (2019 accessed 29 August 2024)."},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1089\/jpm.2018.0294"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpainsymman.2021.06.025"},{"key":"e_1_3_3_35_2","unstructured":"Vanderkam D. Local Turk. https:\/\/github.com\/danvk\/localturk (2024 accessed 22 January 2025)"},{"key":"e_1_3_3_36_2","unstructured":"Lee RY Li KS Sibley J et al. NLP-screened EHR passage adjudication. REDCap Shared Library. January 22 2025. https:\/\/redcap.vumc.org\/consortium\/library\/search.php"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2012.10.006"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-5010"}],"container-title":["Clinical Trials"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/17407745251405386","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/17407745251405386","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/17407745251405386","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T17:29:32Z","timestamp":1777397372000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/17407745251405386"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,12]]},"references-count":37,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,4]]}},"alternative-id":["10.1177\/17407745251405386"],"URL":"https:\/\/doi.org\/10.1177\/17407745251405386","relation":{},"ISSN":["1740-7745","1740-7753"],"issn-type":[{"value":"1740-7745","type":"print"},{"value":"1740-7753","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,12]]}}}