{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,12]],"date-time":"2026-11-12T19:52:51Z","timestamp":1794513171164,"version":"4.9.1"},"reference-count":40,"publisher":"American Society of Hematology","issue":"12","license":[{"start":{"date-parts":[[2024,3,24]],"date-time":"2024-03-24T00:00:00Z","timestamp":1711238400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["ashpublications.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,6,25]]},"abstract":"<jats:sec>\n                  <jats:title>Abstract<\/jats:title>\n                  <jats:p>Venous thromboembolism (VTE) is a leading cause of preventable in-hospital mortality. Monitoring VTE cases is limited by the challenges of manual medical record review and diagnosis code interpretation. Natural language processing (NLP) can automate the process. Rule-based NLP methods are effective but time consuming. Machine learning (ML)-NLP methods present a promising solution. We conducted a systematic review and meta-analysis of studies published before May 2023 that use ML-NLP to identify VTE diagnoses in the electronic health records. Four reviewers screened all manuscripts, excluding studies that only used a rule-based method. A meta-analysis evaluated the pooled performance of each study\u2019s best performing model that evaluated for pulmonary embolism and\/or deep vein thrombosis. Pooled sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) with confidence interval (CI) were calculated by DerSimonian and Laird method using a random-effects model. Study quality was assessed using an adapted TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) tool. Thirteen studies were included in the systematic review and 8 had data available for meta-analysis. Pooled sensitivity was 0.931 (95%\u00a0CI, 0.881-0.962), specificity 0.984 (95%\u00a0CI, 0.967-0.992), PPV 0.910 (95%\u00a0CI, 0.865-0.941) and NPV 0.985 (95%\u00a0CI, 0.977-0.990). All studies met at least 13 of the 21 NLP-modified TRIPOD items, demonstrating fair quality. The highest performing models used vectorization rather than bag-of-words and deep-learning techniques such as convolutional neural networks. There was significant heterogeneity in the studies, and only 4 validated their model on an external data set. Further standardization of ML studies can help progress this novel technology toward real-world implementation.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1182\/bloodadvances.2023012200","type":"journal-article","created":{"date-parts":[[2024,3,24]],"date-time":"2024-03-24T20:59:24Z","timestamp":1711313964000},"page":"2991-3000","update-policy":"https:\/\/doi.org\/10.1182\/blood.2019cm0000","source":"Crossref","is-referenced-by-count":34,"title":["Machine learning natural language processing for identifying venous thromboembolism: systematic review and meta-analysis"],"prefix":"10.1182","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5585-3823","authenticated-orcid":false,"given":"Barbara D.","family":"Lam","sequence":"first","affiliation":[{"name":"1Division of Hematology, Department of Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA"},{"name":"2Division of Clinical Informatics, Department of Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pavlina","family":"Chrysafi","sequence":"additional","affiliation":[{"name":"3Department of Medicine, Mount Auburn Hospital, Harvard Medical School, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0443-1751","authenticated-orcid":false,"given":"Thita","family":"Chiasakul","sequence":"additional","affiliation":[{"name":"4Center of Excellence in Translational Hematology, Division of Hematology, Department of Medicine, Faculty of Medicine, Chulalongkorn University and King Chulalongkorn Memorial Hospital, Thai Red Cross Society, Bangkok, Thailand"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Harshit","family":"Khosla","sequence":"additional","affiliation":[{"name":"5Department of Medicine, Saint Vincent Hospital, Worcester, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dimitra","family":"Karagkouni","sequence":"additional","affiliation":[{"name":"6Department of Pathology, Cancer Research Institute, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Megan","family":"McNichol","sequence":"additional","affiliation":[{"name":"7Library Sciences, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6493-2796","authenticated-orcid":false,"given":"Alys","family":"Adamski","sequence":"additional","affiliation":[{"name":"8Division of Blood Disorders, National Center on Birth Defects and Developmental Disabilities, Centers for Disease Control and Prevention, Atlanta, GA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nimia","family":"Reyes","sequence":"additional","affiliation":[{"name":"8Division of Blood Disorders, National Center on Birth Defects and Developmental Disabilities, Centers for Disease Control and Prevention, Atlanta, GA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karon","family":"Abe","sequence":"additional","affiliation":[{"name":"8Division of Blood Disorders, National Center on Birth Defects and Developmental Disabilities, Centers for Disease Control and Prevention, Atlanta, GA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Simon","family":"Mantha","sequence":"additional","affiliation":[{"name":"9Division of Hematology, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8849-808X","authenticated-orcid":false,"given":"Ioannis S.","family":"Vlachos","sequence":"additional","affiliation":[{"name":"6Department of Pathology, Cancer Research Institute, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5810-6893","authenticated-orcid":false,"given":"Jeffrey I.","family":"Zwicker","sequence":"additional","affiliation":[{"name":"9Division of Hematology, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8426-3398","authenticated-orcid":false,"given":"Rushad","family":"Patell","sequence":"additional","affiliation":[{"name":"1Division of Hematology, Department of Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"234","published-online":{"date-parts":[[2024,6,12]]},"reference":[{"issue":"9610","key":"2024062414454849700_bib1","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1016\/S0140-6736(08)60202-0","article-title":"Venous thromboembolism risk and prophylaxis in the acute hospital care setting (ENDORSE study): a multinational cross-sectional study","volume":"371","author":"Cohen","year":"2008","journal-title":"Lancet"},{"issue":"4","key":"2024062414454849700_bib2","doi-asserted-by":"crossref","first-page":"688","DOI":"10.1160\/TH09-04-0266","article-title":"Long-term complications of medical patients with hospital-acquired venous thromboembolism","volume":"102","author":"Fanikos","year":"2009","journal-title":"Thromb Haemost"},{"issue":"24","key":"2024062414454849700_bib3","doi-asserted-by":"crossref","first-page":"e914","DOI":"10.1161\/CIR.0000000000000769","article-title":"Call to action to prevent venous thromboembolism in hospitalized patients: a policy statement from the American Heart Association","volume":"141","author":"Henke","year":"2020","journal-title":"Circulation"},{"key":"2024062414454849700_bib4","unstructured":"Agency of Healthcare Research and Quality\n          . Chapter 4. Choose the Model to Assess VTE and Bleeding Risk. Accessed 8 April 2024. https:\/\/www.ahrq.gov\/patient-safety\/settings\/hospital\/vtguide\/guide4.html."},{"key":"2024062414454849700_bib5","unstructured":"The Joint Commission\n          . Venous Thromboembolism. Accessed 8 April 2024. https:\/\/www.jointcommission.org\/measurement\/measures\/venous-thromboembolism\/."},{"issue":"6","key":"2024062414454849700_bib6","first-page":"326","article-title":"The validity of ICD-9-CM codes in identifying postoperative deep vein thrombosis and pulmonary embolism","volume":"33","author":"Zhan","year":"2007","journal-title":"Jt Comm J Qual Patient Saf"},{"issue":"12","key":"2024062414454849700_bib7","doi-asserted-by":"crossref","first-page":"e137","DOI":"10.1097\/MLR.0000000000000524","article-title":"Validity of using inpatient and outpatient administrative codes to identify acute venous thromboembolism: the CVRN VTE study","volume":"55","author":"Fang","year":"2017","journal-title":"Med Care"},{"issue":"3","key":"2024062414454849700_bib8","doi-asserted-by":"crossref","first-page":"3713","DOI":"10.1007\/s11042-022-13428-4","article-title":"Natural language processing: state of the art, current trends and challenges","volume":"82","author":"Khurana","year":"2023","journal-title":"Multimed Tools Appl"},{"key":"2024062414454849700_bib9","doi-asserted-by":"crossref","first-page":"106649","DOI":"10.1016\/j.compbiomed.2023.106649","article-title":"Natural language processing in electronic health records in relation to healthcare decision-making: a systematic review","volume":"155","author":"Hossain","year":"2023","journal-title":"Comput Biol Med"},{"issue":"2","key":"2024062414454849700_bib10","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1136\/jamia.1994.95236146","article-title":"A\u00a0general natural-language text processor for clinical radiology","volume":"1","author":"Friedman","year":"1994","journal-title":"J\u00a0Am Med Inform Assoc"},{"issue":"5","key":"2024062414454849700_bib11","doi-asserted-by":"crossref","first-page":"728","DOI":"10.1016\/j.jbi.2011.03.011","article-title":"Document-level classification of CT pulmonary angiography reports based on an extension of the ConText algorithm","volume":"44","author":"Chapman","year":"2011","journal-title":"J\u00a0Biomed Inform"},{"issue":"1","key":"2024062414454849700_bib12","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1186\/s12874-019-0792-y","article-title":"Current approaches to identify sections within clinical narratives from electronic health records: a systematic review","volume":"19","author":"Pomares-Quimbaya","year":"2019","journal-title":"BMC Med Res Methodol"},{"issue":"7","key":"2024062414454849700_bib13","doi-asserted-by":"crossref","first-page":"e048008","DOI":"10.1136\/bmjopen-2020-048008","article-title":"Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence","volume":"11","author":"Collins","year":"2021","journal-title":"BMJ Open"},{"issue":"2","key":"2024062414454849700_bib14","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1148\/radiol.16142770","article-title":"Natural language processing in radiology: a systematic review","volume":"279","author":"Pons","year":"2016","journal-title":"Radiology"},{"issue":"1","key":"2024062414454849700_bib15","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1186\/s12911-021-01533-7","article-title":"A\u00a0systematic review of natural language processing applied to radiology reports","volume":"21","author":"Casey","year":"2021","journal-title":"BMC Med Inform Decis Mak"},{"key":"2024062414454849700_bib16","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.jbi.2017.11.012","article-title":"Radiology report annotation using intelligent word embeddings: applied to multi-institutional chest CT cohort","volume":"77","author":"Banerjee","year":"2018","journal-title":"J\u00a0Biomed Inform"},{"key":"2024062414454849700_bib17","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.artmed.2018.11.004","article-title":"Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification","volume":"97","author":"Banerjee","year":"2019","journal-title":"Artif Intell Med"},{"issue":"3","key":"2024062414454849700_bib18","doi-asserted-by":"crossref","first-page":"845","DOI":"10.1148\/radiol.2017171115","article-title":"Deep learning to classify radiology free-text reports","volume":"286","author":"Chen","year":"2018","journal-title":"Radiology"},{"key":"2024062414454849700_bib19","first-page":"69","article-title":"Semiautomatic identification of pulmonary embolism in electronic health records through sentence labeling","volume":"289","author":"Danilov","year":"2022","journal-title":"Stud Health Technol Inform"},{"issue":"9","key":"2024062414454849700_bib20","doi-asserted-by":"crossref","first-page":"e54","DOI":"10.1097\/MLR.0000000000000831","article-title":"Improved identification of venous thromboembolism from electronic medical records using a novel information extraction software platform","volume":"56","author":"Dantes","year":"2018","journal-title":"Med Care"},{"issue":"1","key":"2024062414454849700_bib21","doi-asserted-by":"crossref","first-page":"266","DOI":"10.1186\/1471-2105-15-266","article-title":"Natural language processing of radiology reports for the detection of thromboembolic diseases and clinically relevant incidental findings","volume":"15","author":"Pham","year":"2014","journal-title":"BMC Bioinformatics"},{"issue":"1","key":"2024062414454849700_bib22","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1136\/amiajnl-2014-002768","article-title":"A\u00a0novel method of adverse event detection can accurately identify venous thromboembolisms (VTEs) from narrative electronic health record data","volume":"22","author":"Rochefort","year":"2015","journal-title":"J\u00a0Am Med Inform Assoc"},{"issue":"6","key":"2024062414454849700_bib23","doi-asserted-by":"crossref","first-page":"1300","DOI":"10.1016\/j.surg.2018.05.008","article-title":"Autonomous detection, grading, and reporting of postoperative complications using natural language processing","volume":"164","author":"Selby","year":"2018","journal-title":"Surgery"},{"key":"2024062414454849700_bib24","doi-asserted-by":"crossref","first-page":"108862","DOI":"10.1016\/j.ejrad.2020.108862","article-title":"Towards automated generation of curated datasets in radiology: Application of natural language processing to unstructured reports exemplified on CT for pulmonary embolism","volume":"125","author":"Weikert","year":"2020","journal-title":"Eur J Radiol"},{"key":"2024062414454849700_bib25","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1016\/j.jbi.2014.08.001","article-title":"Classification of CT pulmonary angiography reports by presence, chronicity, and location of pulmonary embolism with natural language processing","volume":"52","author":"Yu","year":"2014","journal-title":"J\u00a0Biomed Inform"},{"issue":"7_Supple_B","key":"2024062414454849700_bib26","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1302\/0301-620X.102B7.BJJ-2019-1574.R1","article-title":"Data for registry and quality review can be retrospectively collected using natural language processing from unstructured charts of arthroplasty patients","volume":"102-B","author":"Shah","year":"2020","journal-title":"Bone Joint J"},{"key":"2024062414454849700_bib27","first-page":"860","article-title":"Automatic extraction of PIOPED interpretations from ventilation\/perfusion lung scan reports","author":"Fiszman","year":"1998","journal-title":"Proc AMIA Symp"},{"issue":"1","key":"2024062414454849700_bib28","doi-asserted-by":"crossref","DOI":"10.2196\/36877","article-title":"Exploring the applicability of using natural language processing to support nationwide venous thromboembolism surveillance: model evaluation study","volume":"3","author":"Wendelboe","year":"2022","journal-title":"JMIR Bioinform Biotech"},{"key":"2024062414454849700_bib29","first-page":"1532","article-title":"GloVe: Global Vectors for Word Representation","volume":"14","author":"Pennington","year":"2014","journal-title":"EMNLP"},{"key":"2024062414454849700_bib30","article-title":"Efficient estimation of word representations in vector space","author":"Mikolov","year":"2013","journal-title":"arXiv"},{"key":"2024062414454849700_bib31","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1146\/annurev-biodatasci-080917-013315","article-title":"Advances in electronic phenotyping: from rule-based definitions to machine learning models","volume":"1","author":"Banda","year":"2018","journal-title":"Annu Rev Biomed Data Sci"},{"key":"2024062414454849700_bib32","article-title":"Comparing rule-based and deep learning models for patient phenotyping","author":"Gehrmann","year":"2017","journal-title":"arXiv"},{"issue":"Suppl 3","key":"2024062414454849700_bib33","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1186\/s12911-019-0781-4","article-title":"Clinical text classification with rule-based features and knowledge-guided convolutional neural networks","volume":"19","author":"Yao","year":"2019","journal-title":"BMC Med Inform Decis Mak"},{"issue":"11","key":"2024062414454849700_bib34","doi-asserted-by":"crossref","first-page":"1225","DOI":"10.1016\/S0895-4356(96)00002-9","article-title":"Advantages and disadvantages of using artificial neural networks versus logistic regression for predicting medical outcomes","volume":"49","author":"Tu","year":"1996","journal-title":"J\u00a0Clin Epidemiol"},{"key":"2024062414454849700_bib35","article-title":"BERT: pre-training on deep bidirectional transformers for language understanding","author":"Devlin","year":"24  2019","journal-title":"arXiv"},{"issue":"4","key":"2024062414454849700_bib36","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"issue":"1","key":"2024062414454849700_bib37","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1038\/s41746-022-00742-2","article-title":"A\u00a0large language model for electronic health records","volume":"5","author":"Yang","year":"2022","journal-title":"NPJ Digit Med"},{"issue":"25","key":"2024062414454849700_bib38","first-page":"2400","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. Reply","volume":"388","author":"Lee","year":"2023","journal-title":"N\u00a0Engl J Med"},{"issue":"1","key":"2024062414454849700_bib39","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1055\/s-0041-1726522","article-title":"A\u00a0review of recent work in transfer learning and domain adaptation for natural language processing of electronic health records","volume":"30","author":"Laparra","year":"2021","journal-title":"Yearb Med Inform"},{"issue":"7973","key":"2024062414454849700_bib40","doi-asserted-by":"crossref","first-page":"E19","DOI":"10.1038\/s41586-023-06455-0","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"}],"container-title":["Blood Advances"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ashpublications.org\/bloodadvances\/article-pdf\/8\/12\/2991\/2232012\/blooda_adv-2023-012200-main.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/ashpublications.org\/bloodadvances\/article-pdf\/8\/12\/2991\/2232012\/blooda_adv-2023-012200-main.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,24]],"date-time":"2024-06-24T14:46:32Z","timestamp":1719240392000},"score":1,"resource":{"primary":{"URL":"https:\/\/ashpublications.org\/bloodadvances\/article\/8\/12\/2991\/515446\/Machine-learning-natural-language-processing-for"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,12]]},"references-count":40,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2024,6,25]]}},"URL":"https:\/\/doi.org\/10.1182\/bloodadvances.2023012200","relation":{},"ISSN":["2473-9529","2473-9537"],"issn-type":[{"value":"2473-9529","type":"print"},{"value":"2473-9537","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,12]]}}}