{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T21:03:53Z","timestamp":1790283833103,"version":"4.1.0"},"reference-count":28,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Ment Health"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Mental health clinical notes contain decision-critical information often absent from structured electronic health record fields. Large language models (LLMs) can extract clinically relevant signals from narrative text; however, variability in output format, limited reproducibility, and inconsistent evaluation remain barriers to clinical deployment. Despite rapid advances in LLM-based information extraction, clear and reproducible guidance for interdisciplinary clinical teams is limited.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This tutorial aims to present a structured workflow for zero-shot information extraction from mental health clinical notes using locally deployed open-source LLMs. It aims to reduce barriers for clinicians and researchers with limited familiarity with natural language processing (NLP) or LLM-based pipelines. Each stage includes key decision points and examples. The workflow is illustrated on two tasks using synthetic notes: (1) detection of self-injurious thoughts and behaviors (SITB) in pediatric emergency department (ED) notes and (2) antipsychotic medication nonadherence detection in outpatient notes, using schema-constrained outputs and standardized evaluation.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n We describe a five-stage zero-shot LLM pipeline: (1) infrastructure setup with local deployment via\n Ollama<\/jats:italic>\n to prevent protected health information (PHI) transmission; (2) task definition specifying the clinical construct, output format, and evaluation; (3) dataset preparation using synthetic notes; (4) iterative prompt development using a hold-out development set with binary and Likert scale outputs constrained via JSON schemas; and (5) output parsing, normalization, and validation. We generated 300 synthetic notes per task using separate LLMs for generation and evaluation; 200 notes were used for evaluation, and 100 notes (50 positive and 50 negative) were used as a prompt-development set and excluded from final metrics. Evaluation used Large Language Model Meta AI (Llama) 3.2 and Llama 3.3 with deterministic decoding (temperature=0). Performance was assessed using accuracy, precision, recall, and\n F<\/jats:italic>\n 1<\/jats:sub>\n -score; Likert thresholds were optimized using the Youden index with bootstrapped CIs.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n We demonstrated the pipeline\u2019s functionality using 2 example behavioral health detection tasks. Across both examples, the more capable model (Llama 3.3) performed better than the lighter model used earlier in development (Llama 3.2), and we described how the pipeline\u2019s evaluation and error-analysis steps work in practice. These examples also illustrated 2 useful design choices: requiring the model to output in a fixed format reduced errors, and using a graded rating scale, rather than a simple yes\/no format, allowed the detection threshold to be adjusted based on clinical risk tolerance. These results are meant to show that the pipeline works as intended, not to serve as a benchmark of real-world accuracy.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n A schema-driven, zero-shot LLM workflow can support reproducible extraction of clinically relevant information from narrative notes. Local deployment enables processing without transmitting PHI to external servers. This tutorial provides a transferable methodology for institutional adaptation and validation prior to clinical use. All prompts, code, and datasets are publicly available via Zenodo (European Organization for Nuclear Research [CERN]).<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/97498","type":"journal-article","created":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T20:44:03Z","timestamp":1790282643000},"page":"e97498-e97498","source":"Crossref","is-referenced-by-count":0,"title":["Practical Guide to Large Language Models for Information Extraction in Behavioral Health Notes: Tutorial"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-3473-6537","authenticated-orcid":false,"given":"Diya","family":"Saha","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5530-8262","authenticated-orcid":false,"given":"Juliet B","family":"Edgcomb","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,24]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1093\/jamia\/ocae260","article-title":"Extracting social support and social isolation information from clinical psychiatry notes: comparing a rule-based natural language processing system and a large language model","volume":"32","author":"Patra","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-025-02851-w","article-title":"Natural language processing to identify suicidal ideation and anhedonia in major depressive disorder","volume":"25","author":"Vance","journal-title":"BMC Med Inform Decis Mak"},{"issue":"5","key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/30405","article-title":"Impact of electronic health records on information practices in mental health contexts: scoping review","volume":"24","author":"Kariotis","journal-title":"J Med Internet Res"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1016\/j.tacc.2021.02.007","article-title":"Natural language processing in medicine: a review","volume":"38","author":"Locke","journal-title":"Trends Anaesth Crit Care"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.2196\/69284","article-title":"The applications of large language models in mental health: scoping review","volume":"27","author":"Jin","journal-title":"J Med Internet Res"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1016\/j.xjmad.2025.100109","article-title":"Applying large language models to stratify suicide risk using narrative clinical notes","volume":"10","author":"McCoy","journal-title":"J Mood Anxiety Disord"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.2196\/63126","article-title":"Applications of large language models in the field of suicide prevention: scoping review","volume":"27","author":"Holmes","journal-title":"J Med Internet Res"},{"issue":"6","key":"R8","doi-asserted-by":"publisher","first-page":"532","DOI":"10.1192\/bjp.2024.134","article-title":"Detection of suicidality from medical text using privacy-preserving large language models","volume":"225","author":"Wiest","journal-title":"Br J Psychiatry"},{"key":"R9","unstructured":"Goel A Gueta A Gilon O et al. LLMs accelerate annotation for medical information extraction. In: Proceedings of Machine Learning Research. Vol 225. PMLR; 2023:82-100. URL: https:\/\/proceedings.mlr.press\/v225\/goel23a.html [Accessed 17-09-2026]"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1038\/s41698-025-01103-4","article-title":"A software pipeline for medical information extraction with large language models, open source and suitable for oncology","volume":"9","author":"Wiest","journal-title":"NPJ Precis Oncol"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.2196\/57306","article-title":"Exploring the efficacy of large language models in summarizing mental health counseling sessions: benchmark study","volume":"11","author":"Adhikary","journal-title":"JMIR Ment Health"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-28767-z","article-title":"Human level information extraction from clinical reports with finetuned language models","volume":"15","author":"Liu","journal-title":"Sci Rep"},{"key":"R13","unstructured":"Liu P Yuan W Fu J Jiang Z Hayashi H Neubig G . Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. arXiv. Preprint posted online on Jul 28, 2021. [doi: 10.48550\/arXiv.2107.13586]"},{"key":"R14","unstructured":"Brown TB Mann B Ryder N et al. Language models are few-shot learners. Presented at: NeurIPS 2020: Advances in Neural Information Processing Systems 33; Dec 6-12, 2020:1877-1901; Virtual (online). 2020.[doi: 10.5555\/3495724.3495883]"},{"key":"R15","doi-asserted-by":"crossref","unstructured":"Wei J Wang X Schuurmans D et al. Chain-of-thought prompting elicits reasoning in large language models. Presented at: Advances in Neural Information Processing Systems 35; 24824-24837; New Orleans, Louisiana. 2022.URL: http:\/\/www.proceedings.com\/68431.html [Accessed 05-09-2026] [doi: 10.52202\/068431-1800]","DOI":"10.52202\/068431-1800"},{"key":"R16","author":"Wei","journal-title":"Trans Mach Learn Res"},{"key":"R17","doi-asserted-by":"crossref","unstructured":"Ouyang L Wu J Jiang X et al. Training language models to follow instructions with human feedback. Presented at: Advances in Neural Information Processing Systems 35; 27730-27744; New Orleans, Louisiana. 2022.URL: http:\/\/www.proceedings.com\/68431.html [Accessed 05-09-2026] [doi: 10.52202\/068431-2011]","DOI":"10.52202\/068431-2011"},{"key":"R18","unstructured":"Geng S Cooper H Moskal M et al. JSONSchemaBench: a rigorous benchmark of structured outputs for language models. arXiv. Preprint posted online on Feb 27, 2025. [doi: 10.48550\/arXiv.2501.10868]"},{"issue":"4","key":"R19","doi-asserted-by":"publisher","first-page":"458","DOI":"10.1002\/bimj.200410135","article-title":"Estimation of the Youden Index and its associated cutoff point","volume":"47","author":"Fluss","journal-title":"Biom J"},{"key":"R20","unstructured":"Grattafiori A Dubey A Jauhri A et al. The Llama 3 herd of models. arXiv. Preprint posted online on Nov 23, 2024. [doi: 10.48550\/arXiv.2407.21783]"},{"key":"R21","unstructured":"Crosby AE Ortega L Melanson C . Self-directed violence surveillance: uniform definitions and recommended data elements, Version 1.0. Centers for Disease Control and Prevention; 2011. CDC Stacks. 2011. URL: https:\/\/stacks.cdc.gov\/view\/cdc\/11997 [Accessed 05-09-2026]"},{"issue":"7","key":"R22","doi-asserted-by":"publisher","first-page":"1035","DOI":"10.1176\/ajp.2007.164.7.1035","article-title":"Columbia Classification Algorithm of Suicide Assessment (C-CASA): classification of suicidal events in the FDA\u2019s pediatric suicidal risk analysis of antidepressants","volume":"164","author":"Posner","journal-title":"Am J Psychiatry"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.2196\/82371","article-title":"Detecting pediatric emergency service use for suicide and self-harm: multimodal analysis of 3828 encounters","volume":"13","author":"Edgcomb","journal-title":"JMIR Ment Health"},{"issue":"3","key":"R24","doi-asserted-by":"publisher","first-page":"491","DOI":"10.1093\/jamia\/ocz192","article-title":"A governance model for the application of AI in health care","volume":"27","author":"Reddy","journal-title":"J Am Med Inform Assoc"},{"key":"R25","unstructured":"Patil A Tao S Gedhu A . Evaluating reasoning LLMs for suicide screening with the Columbia-Suicide Severity Rating Scale. arXiv. Preprint posted online on May 11, 2025. [doi: 10.48550\/arXiv.2505.13480]"},{"issue":"1","key":"R26","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-22402-7","article-title":"Large language model performance versus human expert ratings in automated suicide risk assessment","volume":"15","author":"Thomas","journal-title":"Sci Rep"},{"issue":"1","key":"R27","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01258-7","article-title":"A framework for human evaluation of large language models in healthcare derived from literature review","volume":"7","author":"Tam","journal-title":"NPJ Digit Med"},{"key":"R28","unstructured":"Saha D Edgcomb JB . DiyadotSaha\/Synthetic-Data: synthetic behavioral health clinical notes: SITB and medication adherence detection. Version 1.0.1. Zenodo; 2026. [doi: 10.5281\/zenodo.19489710]"}],"container-title":["JMIR Mental Health"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T20:44:11Z","timestamp":1790282651000},"score":1,"resource":{"primary":{"URL":"https:\/\/mental.jmir.org\/2026\/1\/e97498"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,24]]},"references-count":28,"URL":"https:\/\/doi.org\/10.2196\/97498","relation":{},"ISSN":["2368-7959"],"issn-type":[{"value":"2368-7959","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,24]]},"article-number":"v13i1e97498"}}