{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T16:36:22Z","timestamp":1776270982142,"version":"3.50.1"},"posted":{"date-parts":[[2026,4,1]]},"group-title":"Health Informatics","reference-count":41,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T00:00:00Z","timestamp":1775001600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2026,4,11]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:sec>\n                  <jats:title>Background<\/jats:title>\n                  <jats:p>\n                    Secure text messages (TMs) exchanged among interdisciplinary care teams in nursing homes (NHs) contain clinical information that aligns with the Age-Friendly Health Systems 4Ms: What\n                    <jats:bold>M<\/jats:bold>\n                    atters,\n                    <jats:bold>M<\/jats:bold>\n                    edication,\n                    <jats:bold>M<\/jats:bold>\n                    entation, and\n                    <jats:bold>M<\/jats:bold>\n                    obility, yet, this information is not captured in any structured form, making it unavailable for systematic monitoring or quality reporting. Automatically extracting 4M information accurately and efficiently from these messages could enable several downstream applications within long term care settings. This task, however, is challenging because of the fragmented syntax, brevity, abbreviations, and informality of TMs.\n                  <\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>This study aimed to develop and evaluate a multi-stage 4M Entity Recognition (4M-ER) pipeline that combines a fine-tuned token classifier with large language model (LLM) revision, using only locally deployed open-source models, to improve 4M information extraction from clinical TMs.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>We used an expert-annotated dataset of 1,169 TMs representing conversations between interdisciplinary care teams across 16 Midwest NHs. The pipeline first identifies candidate text spans using a fine-tuned Bio-ClinicalBERT token classifier. A semantic similarity retriever then selects in-context exemplars to guide an LLM revision in which the LLM (Gemma, Phi, Qwen, or Mistral) performs boundary correction, label evaluation, and selective acceptance or rejection of candidate spans. Baselines for comparison included single-stage zero-shot LLMs, single-stage fine-tuned Bio-ClinicalBERT, and a fine-tuned LLM (Gemma) from a prior study. Ablation studies assessed the contribution of each pipeline stage and the effect of message filtering. Robustness was evaluated across 5 repeated runs.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>\n                    The 4M-ER pipeline outperformed the previously fine-tuned Gemma LLM across all 4M domains, achieving F\n                    <jats:sub>1<\/jats:sub>\n                    (entity type) improvements of +2 to +11 percentage points without any additional fine-tuning and at roughly half the GPU memory (12 vs 24 GB). It also improved upon single-stage fine-tuned Bio-ClinicalBERT in Mobility, Mentation, and What Matters (+0.02 to +0.05 F\n                    <jats:sub>1<\/jats:sub>\n                    ). Error analysis showed that LLM revision reduced false positives by 25% to 35% by correcting misclassifications caused by conversational ambiguity, while the fine-tuned Bio-ClinicalBERT\u2019s high recall captured subtle entities that the fine-tuned Gemma missed. Silver data augmentation further improved the hardest domains, raising What Matters F\n                    <jats:sub>1<\/jats:sub>\n                    from 0.59 to 0.67 and Mobility from 0.64 to 0.67. Ablation studies confirmed that restricting LLMs to revision only yielded optimal accuracy and efficiency.\n                  <\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>The 4M-ER pipeline enables accurate and scalable extraction of 4M entities from clinical TMs by combining fine-tuned Bio-ClinicalBERT with LLM revision using only locally deployed open-source models. The structured 4M data produced by the pipeline can support 4M taxonomy and ontology construction, as demonstrated in the prior work, and provides a foundation for downstream applications including real-time clinical surveillance, compliance with emerging age-friendly quality measures, and predictive modeling in long-term care settings.<\/jats:p>\n                <\/jats:sec>","DOI":"10.64898\/2026.03.31.26349861","type":"posted-content","created":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T23:15:21Z","timestamp":1775085321000},"source":"Crossref","is-referenced-by-count":0,"title":["Combining Token Classification With Large Language Model Revision for Age-Friendly 4M Entity Recognition From Nursing Home Text Messages: Development and Evaluation Study"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3411-6143","authenticated-orcid":false,"given":"Philip","family":"Amewudah","sequence":"first","affiliation":[{"name":"Institute for Data Science and Informatics, University of Missouri, Columbia, Columbia, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6145-8096","authenticated-orcid":false,"given":"Mihail","family":"Popescu","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Biostatistics and Medical Epidemiology, University of Missouri School of Medicine, Columbia, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0989-2968","authenticated-orcid":false,"given":"Matthew S.","family":"Farmer","sequence":"additional","affiliation":[{"name":"School of Nursing, University of Missouri, Columbia, Columbia, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6144-1438","authenticated-orcid":false,"given":"Kimberly R.","family":"Powell","sequence":"additional","affiliation":[{"name":"Sinclair School of Nursing, University of Missouri, Columbia, Missouri, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"54368","reference":[{"key":"2026041508351095000_2026.03.31.26349861v3.1","first-page":"187","article-title":"To Text or Not to Text? That is the Question","volume":"2020","year":"2021","journal-title":"AMIA Annu Symp Proc"},{"key":"2026041508351095000_2026.03.31.26349861v3.2","doi-asserted-by":"publisher","DOI":"10.3928\/00989134-20210604-02"},{"key":"2026041508351095000_2026.03.31.26349861v3.3","doi-asserted-by":"publisher","DOI":"10.2196\/50231"},{"key":"2026041508351095000_2026.03.31.26349861v3.4","unstructured":"Fulmer T. Age-Friendly Health Systems: A Guide to Using the 4Ms While Caring for Older Adults. ( Pelton L , Zhang J , Huang W , eds.). Institute for Healthcare Improvement; 2022."},{"key":"2026041508351095000_2026.03.31.26349861v3.5","doi-asserted-by":"publisher","DOI":"10.1080\/02701960.2022.2117171"},{"key":"2026041508351095000_2026.03.31.26349861v3.6","doi-asserted-by":"publisher","DOI":"10.1016\/j.jamda.2021.12.028"},{"key":"2026041508351095000_2026.03.31.26349861v3.7","doi-asserted-by":"publisher","DOI":"10.5334\/ijic.5588"},{"key":"2026041508351095000_2026.03.31.26349861v3.8","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2017.11.011"},{"key":"2026041508351095000_2026.03.31.26349861v3.9","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2023.3240898"},{"key":"2026041508351095000_2026.03.31.26349861v3.10","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaf006"},{"key":"2026041508351095000_2026.03.31.26349861v3.11","doi-asserted-by":"publisher","DOI":"10.14745\/ccdr.v46i06a02"},{"key":"2026041508351095000_2026.03.31.26349861v3.12","doi-asserted-by":"publisher","DOI":"10.2196\/59782"},{"key":"2026041508351095000_2026.03.31.26349861v3.13","doi-asserted-by":"publisher","DOI":"10.1093\/JAMIA\/OCZ200"},{"key":"2026041508351095000_2026.03.31.26349861v3.14","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423"},{"key":"2026041508351095000_2026.03.31.26349861v3.15","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2023.104302"},{"key":"2026041508351095000_2026.03.31.26349861v3.16","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-022-01819-4"},{"key":"2026041508351095000_2026.03.31.26349861v3.17","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-024-40555-y"},{"key":"2026041508351095000_2026.03.31.26349861v3.18","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.130"},{"key":"2026041508351095000_2026.03.31.26349861v3.19","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocad259"},{"key":"2026041508351095000_2026.03.31.26349861v3.20","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00970-0"},{"key":"2026041508351095000_2026.03.31.26349861v3.21","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2503.17222"},{"key":"2026041508351095000_2026.03.31.26349861v3.22","doi-asserted-by":"publisher","DOI":"10.3390\/electronics14183676"},{"key":"2026041508351095000_2026.03.31.26349861v3.23","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2011.06489"},{"key":"2026041508351095000_2026.03.31.26349861v3.24","doi-asserted-by":"publisher","DOI":"10.1109\/ichi61247.2024.00046"},{"key":"2026041508351095000_2026.03.31.26349861v3.25","doi-asserted-by":"publisher","DOI":"10.1177\/10547738241292657"},{"key":"2026041508351095000_2026.03.31.26349861v3.26","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2311.15946"},{"key":"2026041508351095000_2026.03.31.26349861v3.27","unstructured":"Powell K , Isnainy M , Lee S , et al. Using Novel Natural Language Processing Approaches to Examine Age-Friendly Communication about Nursing Home Residents with Dementia. The Gerontologist."},{"key":"2026041508351095000_2026.03.31.26349861v3.28","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2024.114172"},{"key":"2026041508351095000_2026.03.31.26349861v3.29","unstructured":"Bio_ClinicalBERT Model. emilyalsentzer\/Bio_ClinicalBERT \u00b7 Hugging Face. Accessed December 1, 2025. https:\/\/huggingface.co\/emilyalsentzer\/Bio_ClinicalBERT"},{"key":"2026041508351095000_2026.03.31.26349861v3.30","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1904.03323"},{"key":"2026041508351095000_2026.03.31.26349861v3.31","unstructured":"Sentence-transformer. sentence-transformers\/all-MiniLM-L6-v2 \u00b7 Hugging Face. January 5, 2024. Accessed December 1, 2025. https:\/\/huggingface.co\/sentence-transformers\/all-MiniLM-L6-v2"},{"key":"2026041508351095000_2026.03.31.26349861v3.32","unstructured":"Gemma-2-9b-it model. google\/gemma-2-9b-it \u00b7 Hugging Face. July 10, 2025. Accessed December 1, 2025. https:\/\/huggingface.co\/google\/gemma-2-9b-it"},{"key":"2026041508351095000_2026.03.31.26349861v3.33","unstructured":"Phi-3-Medium-4K-Instruct. microsoft\/Phi-3-medium-4k-instruct \u00b7 Hugging Face. May 1, 2025. Accessed December 1, 2025. https:\/\/huggingface.co\/microsoft\/Phi-3-medium-4k-instruct"},{"key":"2026041508351095000_2026.03.31.26349861v3.34","unstructured":"Qwen2.5-14B-Instruct. Qwen\/Qwen2.5-14B-Instruct \u00b7 Hugging Face. July 21, 2025. Accessed December 1, 2025. https:\/\/huggingface.co\/Qwen\/Qwen2.5-14B-Instruct"},{"key":"2026041508351095000_2026.03.31.26349861v3.35","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2407.10671"},{"key":"2026041508351095000_2026.03.31.26349861v3.36","unstructured":"Mistral-Nemo-Instruct-2407. mistralai\/Mistral-Nemo-Instruct-2407 \u00b7 Hugging Face. Accessed December 1, 2025. https:\/\/huggingface.co\/mistralai\/Mistral-Nemo-Instruct-2407"},{"key":"2026041508351095000_2026.03.31.26349861v3.37","unstructured":"GPT-OSS. https:\/\/ollama.com\/library\/gpt-oss:20b. Accessed March 31, 2026. https:\/\/ollama.com\/library\/gpt-oss:20b"},{"key":"2026041508351095000_2026.03.31.26349861v3.38","unstructured":"Nervaluate: NER evaluation considering partial match scoring. Accessed December 1, 2025. https:\/\/github.com\/MantisAI\/nervaluate"},{"key":"2026041508351095000_2026.03.31.26349861v3.39","unstructured":"OpenAI. ChatGPT. Accessed March 25, 2026. https:\/\/chatgpt.com\/"},{"key":"2026041508351095000_2026.03.31.26349861v3.40","unstructured":"Anthropic. Claude. Accessed March 25, 2026. https:\/\/claude.ai\/new"},{"key":"2026041508351095000_2026.03.31.26349861v3.41","unstructured":"Amewudah P. 4M-ER Codes. https:\/\/github.com\/philipamewudah\/4M-ER. 2026. Accessed March 31, 2026. https:\/\/github.com\/philipamewudah\/4M-ER"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.64898\/2026.03.31.26349861","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T15:35:23Z","timestamp":1776267323000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.64898\/2026.03.31.26349861"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,1]]},"references-count":41,"URL":"https:\/\/doi.org\/10.64898\/2026.03.31.26349861","relation":{},"subject":[],"published":{"date-parts":[[2026,4,1]]},"subtype":"preprint"}}