{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T07:01:08Z","timestamp":1762930868132,"version":"3.45.0"},"reference-count":16,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T00:00:00Z","timestamp":1762905600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Patients within the US Department of Veterans Affairs (VA) healthcare system have the option of receiving care at facilities external to the VA network. This work presents a method for identifying external hospitalizations among the VA\u2019s patient population by utilizing data stored in patient records. The process of extracting this information is complicated by the fact that indicators of external hospitalizations come from two sources: well-defined structured data and free-form unstructured text. Though natural language processing (NLP) leveraging Large Language Models (LLMs) has advanced capabilities to automate information extraction from free text, deploying these systems remains complex and costly. Using structured data is low-cost, but its utility must be determined in order to optimally allocate resources. We describe a method for estimating the utility of using structured and unstructured data and show that if specific conditions are met, the level of effort to perform this estimate can be greatly reduced. For external hospitalizations in the VA, our analysis showed that 44.4% of cases identified using unstructured data could not be found using structured data alone.<\/jats:p>","DOI":"10.3390\/info16110978","type":"journal-article","created":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T06:55:04Z","timestamp":1762930504000},"page":"978","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Estimating the Utility of Using Structured and Unstructured Data for Extracting Incidents of External Hospitalizations from Patient Documents"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5160-2410","authenticated-orcid":false,"given":"Michael","family":"Davenport","sequence":"first","affiliation":[{"name":"Cooperative Studies Program Coordinating Center, VA Boston Healthcare System, Boston, MA 02111, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8556-5016","authenticated-orcid":false,"given":"Robert","family":"Hall","sequence":"additional","affiliation":[{"name":"Cooperative Studies Program Coordinating Center, VA Boston Healthcare System, Boston, MA 02111, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Saraswathi","family":"Kappala","sequence":"additional","affiliation":[{"name":"Cooperative Studies Program Coordinating Center, VA Boston Healthcare System, Boston, MA 02111, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Trevor","family":"Michelson","sequence":"additional","affiliation":[{"name":"Cooperative Studies Program Coordinating Center, VA Boston Healthcare System, Boston, MA 02111, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Robert","family":"Mitchell","sequence":"additional","affiliation":[{"name":"Cooperative Studies Program Coordinating Center, VA Boston Healthcare System, Boston, MA 02111, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1774-4535","authenticated-orcid":false,"given":"David","family":"Winski","sequence":"additional","affiliation":[{"name":"Cooperative Studies Program Coordinating Center, VA Boston Healthcare System, Boston, MA 02111, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-4141-9376","authenticated-orcid":false,"given":"Cynthia","family":"Hau","sequence":"additional","affiliation":[{"name":"Cooperative Studies Program Coordinating Center, VA Boston Healthcare System, Boston, MA 02111, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9821-9646","authenticated-orcid":false,"given":"Sarah","family":"Leatherman","sequence":"additional","affiliation":[{"name":"Cooperative Studies Program Coordinating Center, VA Boston Healthcare System, Boston, MA 02111, USA"},{"name":"School of Public Health, Boston University, Boston, MA 02118, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3065-1619","authenticated-orcid":false,"given":"Frank","family":"Meng","sequence":"additional","affiliation":[{"name":"Cooperative Studies Program Coordinating Center, VA Boston Healthcare System, Boston, MA 02111, USA"},{"name":"Chobanian & Avedisian School of Medicine, Boston University, Boston, MA 02118, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2025,11,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"e241626","DOI":"10.1001\/jamanetworkopen.2024.1626","article-title":"Community Emergency Care Use by Veterans in an Era of Expanding Choice","volume":"7","author":"Vashi","year":"2024","journal-title":"JAMA Netw Open"},{"key":"ref_2","unstructured":"Xu, Z., Jain, S., and Kankanhalli, M. (2024). Hallucination is Inevitable: An Innate Limitation of Large Language Models. arXiv."},{"key":"ref_3","unstructured":"Tonmoy, S.M., Zaman, S.M., Jain, V., Rani, A., Rawte, V., Chadha, A., and Das, A. (2024). A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models. arXiv."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"109713","DOI":"10.1016\/j.isci.2024.109713","article-title":"The application of large language models in medicine: A scoping review","volume":"27","author":"Meng","year":"2024","journal-title":"iScience"},{"key":"ref_5","first-page":"4","article-title":"Natural Language Processing in Medicine: A Review","volume":"38","author":"Locke","year":"2021","journal-title":"Trends Anaesth. Crit. Care"},{"key":"ref_6","unstructured":"Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems 30 (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"10535","DOI":"10.1109\/ACCESS.2023.3240898","article-title":"Systematic Literature Review of Information Extraction From Textual Data: Recent Methods, Applications, Trends, and Challenges","volume":"11","author":"Abdullah","year":"2023","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"e68998","DOI":"10.2196\/68998","article-title":"Evidence for Clinical Text Summarization Using Large Language Models: Scoping Review","volume":"27","author":"Bednarczyk","year":"2025","journal-title":"J. Med. Internet Res."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1038\/s41746-025-01684-1","article-title":"Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis","volume":"8","author":"Gaber","year":"2025","journal-title":"NPJ Digit. Med."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2589","DOI":"10.1007\/s00330-024-11107-6","article-title":"Large language models for structured reporting in radiology: Past, present, and future","volume":"35","author":"Busch","year":"2025","journal-title":"Eur. Radiol."},{"key":"ref_11","unstructured":"Kandpal, N., and Raffel, C. (2025). Position: The Most Expensive Part of an LLM should be its Training Data. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Chiticariu, L., Li, Y., and Reiss, F.R. (2013, January 18\u201321). Rule-Based Information Extraction is Dead! Long Live Rule-Based Information Extraction Systems!. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, WA, USA.","DOI":"10.18653\/v1\/D13-1079"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"980","DOI":"10.1093\/jamia\/ocv012","article-title":"Automating the generation of lexical patterns for processing free text in clinical documents","volume":"22","author":"Meng","year":"2015","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_14","unstructured":"Meng, F., Morioka, C.A., and Elbers, D.C. (2019). Generating Information Extraction Patterns from Overlapping and Variable Length Annotations using Sequence Alignment. arXiv."},{"key":"ref_15","unstructured":"SQL Server Technical Documentation. Available online: https:\/\/learn.microsoft.com\/en-us\/sql\/sql-server\/?view=sql-server-ver17."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1097\/NAQ.0000000000000118","article-title":"The Veterans Affairs\u2019s Corporate Data Warehouse: Uses and Implications for Nursing Research and Practice","volume":"39","author":"Price","year":"2015","journal-title":"Nurs. Adm. Q."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/11\/978\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,12]],"date-time":"2025-11-12T06:58:40Z","timestamp":1762930720000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/11\/978"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,12]]},"references-count":16,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2025,11]]}},"alternative-id":["info16110978"],"URL":"https:\/\/doi.org\/10.3390\/info16110978","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,12]]}}}