{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:54:18Z","timestamp":1782842058660,"version":"3.54.5"},"reference-count":63,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>A wealth of clinically relevant information is only obtainable within unstructured clinical narratives, leading to great interest in clinical natural language processing (NLP). While a multitude of approaches to NLP exist, current algorithm development approaches have limitations that can slow the development process. These limitations are exacerbated when the task is emergent, as is the case currently for NLP extraction of signs and symptoms of COVID-19 and postacute sequelae of SARS-CoV-2 infection (PASC).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aims to highlight the current limitations of existing NLP algorithm development approaches that are exacerbated by NLP tasks surrounding emergent clinical concepts and to illustrate our approach to addressing these issues through the use case of developing an NLP system for the signs and symptoms of COVID-19 and PASC.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We used 2 preexisting studies on PASC as a baseline to determine a set of concepts that should be extracted by NLP. This concept list was then used in conjunction with the Unified Medical Language System to autonomously generate an expanded lexicon to weakly annotate a training set, which was then reviewed by a human expert to generate a fine-tuned NLP algorithm. The annotations from a fully human-annotated test set were then compared with NLP results from the fine-tuned algorithm. The NLP algorithm was then deployed to 10 additional sites that were also running our NLP infrastructure. Of these 10 sites, 5 were used to conduct a federated evaluation of the NLP algorithm.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>An NLP algorithm consisting of 12,234 unique normalized text strings corresponding to 2366 unique concepts was developed to extract COVID-19 or PASC signs and symptoms. An unweighted mean dictionary coverage of 77.8% was found for the 5 sites.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The evolutionary and time-critical nature of the PASC NLP task significantly complicates existing approaches to NLP algorithm development. In this work, we present a hybrid approach using the Open Health Natural Language Processing Toolkit aimed at addressing these needs with a dictionary-based weak labeling step that minimizes the need for additional expert annotation while still preserving the fine-tuning capabilities of expert involvement.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/49997","type":"journal-article","created":{"date-parts":[[2024,3,1]],"date-time":"2024-03-01T03:22:57Z","timestamp":1709263377000},"page":"e49997","source":"Crossref","is-referenced-by-count":3,"title":["A Case Demonstration of the Open Health Natural Language Processing Toolkit From the National COVID-19 Cohort Collaborative and the Researching COVID to Enhance Recovery Programs for a Natural Language Processing System for COVID-19 or Postacute Sequelae of SARS CoV-2 Infection: Algorithm Development and Validation"],"prefix":"10.2196","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9090-8028","authenticated-orcid":false,"given":"Andrew","family":"Wen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9970-8604","authenticated-orcid":false,"given":"Liwei","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1312-4195","authenticated-orcid":false,"given":"Huan","family":"He","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1691-5179","authenticated-orcid":false,"given":"Sunyang","family":"Fu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9763-1164","authenticated-orcid":false,"given":"Sijia","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6931-3791","authenticated-orcid":false,"given":"David A","family":"Hanauer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9139-3433","authenticated-orcid":false,"given":"Daniel R","family":"Harris","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1238-9378","authenticated-orcid":false,"given":"Ramakanth","family":"Kavuluru","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8258-3585","authenticated-orcid":false,"given":"Rui","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9066-9431","authenticated-orcid":false,"given":"Karthik","family":"Natarajan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3135-3047","authenticated-orcid":false,"given":"Nishanth P","family":"Pavinkurve","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6251-9586","authenticated-orcid":false,"given":"Janos","family":"Hajagos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3991-8420","authenticated-orcid":false,"given":"Sritha","family":"Rajupet","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5335-3721","authenticated-orcid":false,"given":"Veena","family":"Lingam","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1554-5738","authenticated-orcid":false,"given":"Mary","family":"Saltz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5481-8305","authenticated-orcid":false,"given":"Corey","family":"Elowsky","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2723-5902","authenticated-orcid":false,"given":"Richard A","family":"Moffitt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6974-5674","authenticated-orcid":false,"given":"Farrukh M","family":"Koraishy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7737-8752","authenticated-orcid":false,"given":"Matvey B","family":"Palchuk","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-8820-7819","authenticated-orcid":false,"given":"Jordan","family":"Donovan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-1209-0983","authenticated-orcid":false,"given":"Lora","family":"Lingrey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-8708-7764","authenticated-orcid":false,"given":"Garo","family":"Stone-DerHagopian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1787-2855","authenticated-orcid":false,"given":"Robert T","family":"Miller","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0692-412X","authenticated-orcid":false,"given":"Andrew E","family":"Williams","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9730-1675","authenticated-orcid":false,"given":"Peter J","family":"Leese","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1562-1095","authenticated-orcid":false,"given":"Paul I","family":"Kovach","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6840-9756","authenticated-orcid":false,"given":"Emily R","family":"Pfaff","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2434-2537","authenticated-orcid":false,"given":"Mikhail","family":"Zemmel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-9949-4139","authenticated-orcid":false,"given":"Robert D","family":"Pates","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8409-4605","authenticated-orcid":false,"given":"Nick","family":"Guthe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9114-8737","authenticated-orcid":false,"given":"Melissa A","family":"Haendel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5437-2545","authenticated-orcid":false,"given":"Christopher G","family":"Chute","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2570-3741","authenticated-orcid":false,"given":"Hongfang","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"National COVID Cohort Collaborative","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"The RECOVER Initiative","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2024,9,9]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.15265\/IY-2014-0020"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2023.8823"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.n1648"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-021-03553-9"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1126\/science.abm8108"},{"key":"ref6","first-page":"67","volume":"2013","author":"Forbush, TB","year":"2013","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2017.11.011"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1007\/s10278-014-9762-4"},{"key":"ref9","first-page":"149","volume":"2013","author":"Liu, H","year":"2013","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1136\/jamia.2009.001560"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2013-002190"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1197\/jamia.M3378"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.1090\/mbk\/121\/79"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btq126"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-020-01241-8"},{"key":"ref16","first-page":"779","volume":"2012","author":"Roberts, K","year":"2012","journal-title":"AMIA Annu Symp Proc"},{"key":"ref17","first-page":"1812","volume":"2017","author":"Wu, Y","year":"2017","journal-title":"AMIA Annu Symp Proc"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1109\/jcsse.2019.8864178"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.2196\/17934"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.2196\/22797"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocx132"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-015-0160-8"},{"key":"ref23","first-page":"1326","volume":"2015","author":"Wu, Y","year":"2015","journal-title":"AMIA Annu Symp Proc"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.5121\/ijnlc.2012.1402"},{"key":"ref25","unstructured":"SkounakisMCravenMRaySHierarchical hidden Markov models for information extraction2003IJCAI'03: Proceedings of the 18th International Joint Conference on Artificial IntelligenceAugust 9-15, 2003Acapulco, MexicoSan Francisco, CAMorgan Kaufmann Publishers Inc427433"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.1109\/neurel.2008.4685557"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1109\/cisp-bmei48845.2019.8965823"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btz682"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocab126"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2021.104433"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/n19-1078"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.1186\/s12859-018-2467-9"},{"key":"ref33","author":"Kocaman, V","year":"2021","journal-title":"Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10-15, 2021, Proceedings, Part I"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-017-0468-7"},{"key":"ref35","doi-asserted-by":"publisher","DOI":"10.1136\/jamia.1999.0060393"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.1006\/jbin.2001.1029"},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2015.02.010"},{"key":"ref38","doi-asserted-by":"publisher","DOI":"10.1136\/jamia.2001.0080598"},{"key":"ref39","doi-asserted-by":"publisher","DOI":"10.3115\/1572392.1572408"},{"key":"ref40","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23533"},{"key":"ref41","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2011-000150"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0112774"},{"key":"ref43","doi-asserted-by":"crossref","unstructured":"BhatiaPCelikkayaBKhaliliaMJoint entity extraction and assertion detection for clinical text2019Proceedings of the 57th Annual Meeting of the Association for Computational LinguisticsJuly 28-August 2, 2019Florence, ItalyAssociation for Computational Linguistics954959","DOI":"10.18653\/v1\/P19-1091"},{"key":"ref44","first-page":"1149","volume":"2017","author":"Rumeng, L","year":"2017","journal-title":"AMIA Annu Symp Proc"},{"key":"ref45","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.nlpmc-1.5"},{"key":"ref46","doi-asserted-by":"publisher","DOI":"10.1017\/cts.2022.207"},{"key":"ref47","doi-asserted-by":"publisher","DOI":"10.1145\/3535508.3545555"},{"key":"ref48","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2021.103951"},{"key":"ref49","first-page":"1639","volume":"2011","author":"Zhou, L","year":"2011","journal-title":"AMIA Annu Symp Proc"},{"key":"ref50","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocad134"},{"key":"ref51","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.18"},{"key":"ref52","doi-asserted-by":"publisher","DOI":"10.1016\/j.ebiom.2021.103722"},{"key":"ref53","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btab880"},{"key":"ref54","first-page":"574","volume":"216","author":"Hripcsak, G","year":"2015","journal-title":"Stud Health Technol Inform"},{"key":"ref55","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaa196"},{"key":"ref56","unstructured":"National Center for Advancing Translational Sciences, National Institutes of Health. National COVID Cohort Collaborative (N3C)20202024-07-14https:\/\/ncats.nih.gov\/n3c"},{"key":"ref57","unstructured":"Researching COVID to Enhance Recovery20242024-07-14https:\/\/recovercovid.org"},{"key":"ref58","unstructured":"National Center for Advancing Translational Sciences, National Institutes of Health. Data Transfer Agreement Signatories20242024-07-14https:\/\/covid.cd2h.org\/dtas"},{"key":"ref59","unstructured":"National Center for Advancing Translational Sciences, National Institutes of Health. DUA signatories20242024-07-14https:\/\/covid.cd2h.org\/duas"},{"issue":"3","key":"ref60","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1093\/jamia\/ocaa196","volume":"28","author":"Haendel, MA","year":"2021","journal-title":"Journal of the American Medical Informatics Association"},{"key":"ref61","unstructured":"National Center for Advancing Translational Sciences, National Institutes of Health. National COVID Cohort Collaborative. Enclave essentials20242024-07-14https:\/\/covid.cd2h.org\/enclave\/"},{"key":"ref62","unstructured":"National Center for Advancing Translational Sciences, National Institutes of Health. National COVID Cohort Collaborative Forms and Resources20242024-07-14https:\/\/ncats.nih.gov\/research\/research-activities\/n3c\/resources"},{"key":"ref63","unstructured":"National Center for Advancing Translational Sciences, National Institutes of Health. N3C core contributors20242024-07-14https:\/\/covid.cd2h.org\/core-contributors"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2024\/1\/e49997","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:32:13Z","timestamp":1782840733000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2024\/1\/e49997"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,9]]},"references-count":63,"URL":"https:\/\/doi.org\/10.2196\/49997","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.49997","asserted-by":"object"}]},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,9]]}}}