{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T20:56:03Z","timestamp":1791406563978,"version":"4.3.3"},"reference-count":54,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Res Protoc"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Chronic pain is a leading cause of disability and requires multidimensional assessment of pain intensity and functioning, yet electronic health records rarely capture these measures systematically. By contrast, surveys collecting patient-reported outcomes can assess pain over multiple dimensions but remain resource-intensive and difficult to scale for continuous population-level monitoring.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n The objective of this study is to develop and validate a domain-informed natural language processing framework to derive pain severity and functional interference outcomes from unstructured clinical narratives. We aim to demonstrate that natural language processing\u2013derived outcomes can serve as a reliable, scalable surrogate for resource-intensive patient-reported surveys.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n This study uses a retrospective cohort of 3725 Veterans with chronic musculoskeletal pain initiating complementary and integrative health therapies across 18 Veterans Health Administration Whole Health Flagship sites (2021\u20102023). The dataset encompasses longitudinal patient-reported outcome surveys serving as the benchmark, linked with unstructured clinical narratives from the Veterans Health Administration electronic health record. Guided by established psychometric instruments and subject matter expert (SME) input, we developed a seed lexicon and annotation guidelines to identify language distinguishing 3 pain domains: pain severity, interference with enjoyment of life, and interference with general activities. Preliminary large language model (LLM) prompting was used to identify 600 candidate encounters (200 per domain) from 6747 notes across 260 patients for SME annotation, forming a ground truth validation sample. Two candidate LLMs will be evaluated on this sample; the best-performing LLM will generate a large library of span-level annotations to train a scalable, lightweight language model. The study uses a 3-stage validation process: (1) documentation completeness of pain interference in clinical narratives against SME-annotated references; (2) inference accuracy of the LLM-as-annotator and the fine-tuned lightweight model against SME annotations across note-level classification and span-level localization; and (3) concordance between the lightweight model\u2019s output and patient-reported pain, enjoyment, and general activity scores across a range of temporal windows.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n As of July 2026, the cohort of 3725 Veterans has been identified and linked to clinical notes. The seed lexicon and annotation guidelines have been developed. Applying a developmental LLM to screen 6747 text notes in 6642 unique encounters over a 7-month period for 260 patients, at least 1 of the 3 pain domains was identified in 75% of notes and 99% of patients. SME validation at the encounter level is in progress. Final results from the subsequent knowledge distillation and validation stages are expected in the first half of 2027.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This protocol outlines a framework for identifying severe pain intensity and interference from clinical narratives, addressing a critical gap in health care system surveillance. To our knowledge, this is the first study to validate clinical text-based pain outcome extraction against patient-reported outcomes in a nationwide longitudinal cohort. If successful, this approach will enable health care systems to continuously monitor reports of pain-related functional interference and support more holistic, patient-centered pain management at scale.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/96346","type":"journal-article","created":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T12:15:07Z","timestamp":1784722507000},"page":"e96346-e96346","source":"Crossref","is-referenced-by-count":0,"title":["Extraction of Pain Severity and Functional Interference From Clinical Narratives Using Domain-Informed Large Language Models: Protocol for a Development and Validation Study"],"prefix":"10.2196","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7185-0470","authenticated-orcid":false,"given":"Xiaoyi","family":"Zhang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1929-8033","authenticated-orcid":false,"given":"Christopher R","family":"Wilson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-2648-9574","authenticated-orcid":false,"given":"Hannah","family":"Eyre","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0109-5999","authenticated-orcid":false,"given":"David E","family":"Reed II","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4016-8732","authenticated-orcid":false,"given":"Travis Y","family":"Hee Wai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-5418-8971","authenticated-orcid":false,"given":"Alexander","family":"Kloehn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1060-7634","authenticated-orcid":false,"given":"Ethan W","family":"Rosser","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7217-4008","authenticated-orcid":false,"given":"Gang","family":"Luo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3900-743X","authenticated-orcid":false,"given":"Steven B","family":"Zeliadt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,7]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.15620\/cdc\/169630","article-title":"Chronic pain and high-impact chronic pain in U.S. adults, 2023","author":"Lucas","journal-title":"NCHS Data Brief"},{"issue":"9","key":"R2","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1097\/MLR.0000000000002181","article-title":"Economic costs of chronic pain-United States, 2021","volume":"63","author":"Guy","journal-title":"Med Care"},{"issue":"2","key":"R3","doi-asserted-by":"publisher","first-page":"e273","DOI":"10.1016\/j.bja.2019.03.023","article-title":"Chronic pain: a review of its epidemiology and associated factors in population-based studies","volume":"123","author":"Mills","journal-title":"Br J Anaesth"},{"issue":"5","key":"R4","doi-asserted-by":"publisher","first-page":"104420","DOI":"10.1016\/j.jpain.2023.11.003","article-title":"Seventeen-year national pain prevalence trends among U.S. military veterans","volume":"25","author":"Taylor","journal-title":"J Pain"},{"issue":"6","key":"R5","doi-asserted-by":"publisher","first-page":"e715","DOI":"10.1097\/j.pain.0000000000002477","article-title":"Measuring pain care quality in the Veterans Health Administration primary care setting","volume":"163","author":"Luther","journal-title":"Pain"},{"issue":"1-2","key":"R6","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1016\/j.pain.2004.09.012","article-title":"Core outcome measures for chronic pain clinical trials: IMMPACT recommendations","volume":"113","author":"Dworkin","journal-title":"Pain"},{"issue":"3","key":"R7","doi-asserted-by":"publisher","first-page":"370","DOI":"10.1016\/j.jpain.2021.08.005","article-title":"NIH\u2019s Helping to End Addiction Long-term Initiative (NIH HEAL Initiative) Clinical Pain Management Common Data Element Program","volume":"23","author":"Wandner","journal-title":"J Pain"},{"key":"R8","unstructured":"Pain as the 5th Vital Sign Toolkit. Geriatrics and Extended Care Strategic Healthcare Group, National Pain Management Coordinating Committee, Veterans Health Administration; 2000. URL: https:\/\/www.va.gov\/painmanagement\/docs\/toolkit.pdf [Accessed 12-11-2025]"},{"issue":"6","key":"R9","doi-asserted-by":"publisher","first-page":"607","DOI":"10.1111\/j.1525-1497.2006.00415.x","article-title":"Measuring pain as the 5th vital sign does not improve quality of pain management","volume":"21","author":"Mularski","journal-title":"J Gen Intern Med"},{"issue":"2","key":"R10","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1016\/j.pmn.2017.10.010","article-title":"Moving beyond pain as the fifth vital sign and patient satisfaction scores to improve pain care in the 21st century","volume":"19","author":"Scher","journal-title":"Pain Manag Nurs"},{"issue":"Suppl 1","key":"R11","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1007\/s11606-018-4326-9","article-title":"Automating collection of pain-related patient-reported outcomes to enhance clinical care and research","volume":"33","author":"Owen-Smith","journal-title":"J Gen Intern Med"},{"issue":"10","key":"R12","doi-asserted-by":"publisher","first-page":"2235","DOI":"10.1093\/pm\/pnab110","article-title":"Associations of pain numeric rating scale scores collected during usual care with research administered patient reported pain outcomes","volume":"22","author":"Nugent","journal-title":"Pain Med"},{"issue":"6","key":"R13","doi-asserted-by":"publisher","first-page":"733","DOI":"10.1007\/s11606-009-0981-1","article-title":"Development and initial validation of the PEG, a three-item scale assessing pain intensity and interference","volume":"24","author":"Krebs","journal-title":"J Gen Intern Med"},{"issue":"10","key":"R14","doi-asserted-by":"publisher","first-page":"1453","DOI":"10.1007\/s11606-007-0321-2","article-title":"Accuracy of the pain numeric rating scale as a screening test in primary care","volume":"22","author":"Krebs","journal-title":"J Gen Intern Med"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1177\/27536130241241259","article-title":"Measuring patient-reported use and outcomes from complementary and integrative health therapies: development of the Complementary and Integrative Health Therapy Patient Experience Survey","volume":"13","author":"Taylor","journal-title":"Glob Adv Integr Med Health"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1177\/27536130251358757","article-title":"Complementary and integrative health therapies and pain: delivery through Veterans Affairs and community care","volume":"14","author":"Calvert","journal-title":"Glob Adv Integr Med Health"},{"issue":"3","key":"R17","doi-asserted-by":"publisher","first-page":"570","DOI":"10.1093\/jamia\/ocac231","article-title":"Using artificial intelligence to improve pain assessment and pain management: a scoping review","volume":"30","author":"Zhang","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"R18","doi-asserted-by":"publisher","first-page":"467","DOI":"10.1080\/14737167.2024.2322664","article-title":"Using natural language processing to analyze unstructured patient-reported outcomes data derived from electronic health records for cancer populations: a systematic review","volume":"24","author":"Sim","journal-title":"Expert Rev Pharmacoecon Outcomes Res"},{"issue":"2","key":"R19","first-page":"E245","volume":"25","author":"Dave","journal-title":"Pain Physician"},{"issue":"11","key":"R20","doi-asserted-by":"publisher","DOI":"10.2196\/26777","article-title":"Natural language processing and machine learning methods to characterize unstructured patient-reported outcomes: validation study","volume":"23","author":"Lu","journal-title":"J Med Internet Res"},{"issue":"12","key":"R21","doi-asserted-by":"publisher","first-page":"e855","DOI":"10.1016\/S2589-7500(23)00202-9","article-title":"Using fine-tuned large language models to parse clinical notes in musculoskeletal pain disorders","volume":"5","author":"Vaid","journal-title":"Lancet Digit Health"},{"key":"R22","unstructured":"Vaswani A Shazeer N Parmar N et al. Attention is all you need. In: Guyon I von Luxburg U Bengio S Wallach H Fergus R Vishwanathan SVN Garnett R , editors. Advances in Neural Information Processing Systems 30 (NeurIPS 2017). Curran Associates Inc; 2017:5998-6008. ISBN: 9781510860964"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.2196\/65903","article-title":"Exploring the capacity of large language models to assess the chronic pain experience: algorithm development and validation","volume":"27","author":"Amidei","journal-title":"J Med Internet Res"},{"issue":"4","key":"R24","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2023-079923","article-title":"Distributions of recorded pain in mental health records: a natural language processing based study","volume":"14","author":"Chaturvedi","journal-title":"BMJ Open"},{"issue":"11","key":"R25","doi-asserted-by":"publisher","DOI":"10.2196\/18659","article-title":"Characterizing chronic pain episodes in clinical text at two health care systems: comprehensive annotation and corpus analysis","volume":"8","author":"Carlson","journal-title":"JMIR Med Inform"},{"issue":"7","key":"R26","doi-asserted-by":"publisher","first-page":"1285","DOI":"10.1007\/s11517-017-1772-1","article-title":"Classifying clinical notes with pain assessment using machine learning","volume":"56","author":"Fodeh","journal-title":"Med Biol Eng Comput"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2023-078378","article-title":"TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods","volume":"385","author":"Collins","journal-title":"BMJ"},{"key":"R28","unstructured":"Office of Research and Development. VHA operations activities that may constitute research. Department of Veterans Affairs; Jan 9, 2019. URL: https:\/\/www.research.va.gov\/resources\/policies\/ProgramGuide-1200-21-VHA-Operations-Activities.pdf [Accessed 12-09-2026]"},{"issue":"4","key":"R29","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1177\/1740774520928399","article-title":"The APPROACH trial: assessing pain, patient-reported outcomes, and complementary and integrative health","volume":"17","author":"Zeliadt","journal-title":"Clin Trials"},{"issue":"Suppl 2","key":"R30","doi-asserted-by":"publisher","first-page":"S100","DOI":"10.1093\/pm\/pnaa349","article-title":"Assessing the relative effectiveness of combining self-care with practitioner-delivered complementary and integrative health therapies to improve pain in a pragmatic trial","volume":"21","author":"Zeliadt","journal-title":"Pain Med"},{"issue":"2","key":"R31","doi-asserted-by":"publisher","first-page":"277","DOI":"10.1016\/0304-3959(94)00178-H","article-title":"When is cancer pain mild, moderate or severe? Grading pain severity by its interference with function","volume":"61","author":"Serlin","journal-title":"Pain"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2016.01466","article-title":"Cut-off points for mild, moderate, and severe pain on the Numeric Rating Scale for pain in patients with chronic musculoskeletal pain: variability and influence of sex and catastrophizing","volume":"7","author":"Boonstra","journal-title":"Front Psychol"},{"issue":"2","key":"R33","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1016\/j.jpain.2005.09.011","article-title":"Pain interference in persons with spinal cord injury: classification of mild, moderate, and severe pain","volume":"7","author":"Hanley","journal-title":"J Pain"},{"issue":"3","key":"R34","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1016\/S0304-3959(00)00459-0","article-title":"Pain site and the effects of amputation pain: further clarification of the meaning of mild, moderate, and severe pain","volume":"91","author":"Jensen","journal-title":"Pain"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1016\/j.pain.2010.04.025","article-title":"Development of a PROMIS item bank to measure pain interference","volume":"150","author":"Amtmann","journal-title":"Pain"},{"issue":"4","key":"R36","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1016\/0304-3959(85)90004-1","article-title":"The West Haven-Yale Multidimensional Pain Inventory (WHYMPI)","volume":"23","author":"Kerns","journal-title":"Pain"},{"issue":"8","key":"R37","first-page":"271","volume":"66","author":"Fairbank","journal-title":"Physiotherapy"},{"key":"R38","unstructured":"South B Shen S Leng J Forbush T DuVall S Chapman W . A prototype tool set to support machine-assisted annotation. In: Cohen KB Demner-Fushman D Ananiadou S Webber B Tsujii J Pestian J , editors. BioNLP: Proceedings of the 2012 Workshop on Biomedical Natural Language Processing. Association for Computational Linguistics; 2012:130-139. URL: https:\/\/aclanthology.org\/W12-2416\/ [Accessed 12-09-2026]"},{"issue":"7972","key":"R39","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature"},{"issue":"3","key":"R40","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","journal-title":"Nat Med"},{"key":"R41","unstructured":"Nori H King N McKinney SM Carignan D Horvitz E . Capabilities of GPT-4 on medical challenge problems. arXiv. Preprint posted online on Mar 20, 2023. [doi: 10.48550\/arXiv.2303.13375]"},{"issue":"14","key":"R42","doi-asserted-by":"publisher","first-page":"6421","DOI":"10.3390\/app11146421","article-title":"What disease does this patient have? A large-scale open domain question answering dataset from medical exams","volume":"11","author":"Jin","journal-title":"Applied Sciences"},{"key":"R43","unstructured":"Jiang AQ Sablayrolles A Roux A et al. Mixtral of experts. arXiv. Preprint posted online on Jan 8, 2024. [doi: 10.48550\/arXiv.2401.04088]"},{"key":"R44","unstructured":"google\/gemma-4-E4B. Hugging Face. 2026. URL: https:\/\/huggingface.co\/google\/gemma-4-E4B [Accessed 28-05-2026]"},{"key":"R45","unstructured":"Sellergren A Gao C Mahvar F et al. MedGemma 1.5 technical report. arXiv. Preprint posted online on Apr 6, 2026. [doi: 10.48550\/arXiv.2604.05081]"},{"key":"R46","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1800","article-title":"Chain-of-thought prompting elicits reasoning in large language models","author":"Wei","journal-title":"Proc Int Conf Neural Inf Process Syst"},{"key":"R47","doi-asserted-by":"crossref","unstructured":"Alsentzer E Murphy J Boag W et al. Publicly available clinical BERT embeddings. In: Rumshisky A Roberts K Bethard S Naumann T , editors. Proceedings of the 2nd Clinical Natural Language Processing Workshop. Association for Computational Linguistics; 2019:72-78. [doi: 10.18653\/v1\/W19-1909]","DOI":"10.18653\/v1\/W19-1909"},{"key":"R48","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423","article-title":"BERT: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","journal-title":"Proc Conf North Am Chapter Assoc Comput Linguist Hum Lang Technol"},{"key":"R49","volume":"2021","author":"Eyre","journal-title":"AMIA Annu Symp Proc"},{"key":"R50","unstructured":"Lee SA Wu A Chiang JN . Clinical ModernBERT: an efficient and long context encoder for biomedical text. arXiv. Preprint posted online on Apr 4, 2025. [doi: 10.48550\/arXiv.2504.03964]"},{"key":"R51","doi-asserted-by":"publisher","DOI":"10.52202\/075280-0441","article-title":"QLORA: efficient finetuning of quantized LLMs","author":"Dettmers","journal-title":"Adv Neural Inf Process Syst"},{"key":"R52","doi-asserted-by":"publisher","DOI":"10.1101\/2025.04.22.25326219","article-title":"Automating evaluation of AI text generation in healthcare with a large language model (LLM)-as-a-judge","author":"Croxford","journal-title":"medRxiv"},{"issue":"5","key":"R53","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1097\/MLR.0000000000002295","article-title":"Adding self-care complementary and integrative health therapies to care for chronic pain: the Assessing Pain, Patient Reported Outcomes and Complementary Health (APPROACH) study","volume":"64","author":"Zeliadt","journal-title":"Med Care"},{"key":"R54","unstructured":"VA Informatics and Computing Infrastructure (VINCI). US Department of Veterans Affairs. URL: https:\/\/www.research.va.gov\/programs\/vinci\/ [Accessed 12-09-2026]"}],"container-title":["JMIR Research Protocols"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T20:41:53Z","timestamp":1791405713000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.researchprotocols.org\/2026\/1\/e96346"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,7]]},"references-count":54,"URL":"https:\/\/doi.org\/10.2196\/96346","relation":{},"ISSN":["1929-0748"],"issn-type":[{"value":"1929-0748","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,7]]},"article-number":"v15i9e96346"}}