{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T17:29:59Z","timestamp":1791480599886,"version":"4.3.4"},"posted":{"date-parts":[[2025,8,12]]},"group-title":"Addiction Medicine","reference-count":18,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2025,8,12]],"date-time":"2025-08-12T00:00:00Z","timestamp":1754956800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2025,8,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:sec>\n                  <jats:title>RATIONALE<\/jats:title>\n                  <jats:p>Recent studies have found that stigmatizing terms can incline physicians to pursue punitive approaches to patient care. The intensive care unit (ICU) contains large volumes of progress notes that may contain stigmatizing language, which could perpetuate negative biases against patients and affect healthcare delivery. Patients with substance use disorders (alcohol, opioid, and non-opioid drugs) are particularly vulnerable to stigma. This study aimed to examine the performance of Large Language Models (LLMs) in the identification of stigmatizing language from ICU progress notes of patients with substance use disorders (SUD).<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>METHODS<\/jats:title>\n                  <jats:p>Clinical notes were sampled from the Medical Information Mart for Intensive Care (MIMIC)-III, which contains 2,083,180 ICU notes. These 2,083,180 notes were passed into a rule-based labeling approach followed by manual verification for more ambiguous cases. The labeling approach followed the NIH guidelines on stigma in SUD. The labeling process resulted in identifying 38,552 stigmatizing encounters. To design our cohort, we randomly sampled an equivalent amount of non-stigmatizing encounters to create a dataset with 77,104 notes. This cohort was organized into train\/development\/test datasets (70\/15\/15). We utilized Meta\u2019s Llama-3 8B Instruct LLM to run the following experiments for stigma detection: (1) prompts with instructions that adhere to the NIH terms (Zero-Shot); (2) prompts with instructions and examples (in-context learning); (3) in-context learning with a selective retrieval system for the NIH terms (Retrieval Augmented Generation-RAG); and (4) supervised fine-tuning (SFT). We also created a baseline model using keyword search. Evaluation was performed on the held-out test set for accuracy, macro F1 score, and error analysis. The LLM-based approaches were prompted to provide their reasoning for label prediction. Additionally, all approaches were evaluated on an external validation dataset from the University of Wisconsin (UW) Health System with 288,130 ICU notes.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>RESULTS<\/jats:title>\n                  <jats:p>SFT had the best performance with 97.2% accuracy, followed by in-context learning. The LLMs with in-context learning and SFT provided appropriate reasoning for false positives during human review. Both approaches identified clinical notes with stigmatizing language that were missed during annotation (10\/93 false positives for SFT and 22\/186 false positives for the in-context learning approach were considered valid after human review). SFT maintained its accuracy at 97.9% on a similarly balanced external validation dataset.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>CONCLUSION<\/jats:title>\n                  <jats:p>Our findings demonstrate that LLMs, particularly using SFT and in-context learning, effectively identify stigmatizing language in ICU notes with high accuracy while explaining their reasoning in an asynchronous fashion without needing rigorous and time-intensive manual verification involved in labeling. These models also demonstrated the ability to identify novel stigmatizing language not explicitly in training data nor existing guidelines. This study highlights the potential of LLMs in reducing stigma in clinical documentation, especially for patients with SUD. These LLMs enable identification of stigmatizing language in clinical notes that can perpetuate negative stigma towards patients and encourage rewriting of notes.<\/jats:p>\n                <\/jats:sec>","DOI":"10.1101\/2025.08.08.25333315","type":"posted-content","created":{"date-parts":[[2025,8,12]],"date-time":"2025-08-12T15:11:42Z","timestamp":1755011502000},"source":"Crossref","is-referenced-by-count":1,"title":["Detecting Stigmatizing Language in Clinical Notes with Large Language Models for Addiction Care"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9587-3722","authenticated-orcid":false,"given":"R.","family":"Sethi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"J.","family":"Caskey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Y.","family":"Gao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"M.M.","family":"Churpek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"T.A.","family":"Miller","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"A.","family":"Mayampurath","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"E.S.","family":"Afshar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"M.","family":"Afshar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"D.","family":"Dligach","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2025081404251119000_2025.08.08.25333315v1.1","unstructured":"US Senate Committee on Health, Education, Labor, and Pensions. Implementing the 21st Century Cures Act: Making electronic health information available to patients and providers. US Government Publishing Office, Washington, DC (2021)."},{"key":"2025081404251119000_2025.08.08.25333315v1.2","doi-asserted-by":"publisher","DOI":"10.1016\/j.drugpo.2009.10.010"},{"key":"2025081404251119000_2025.08.08.25333315v1.3","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-024-00601-z"},{"key":"2025081404251119000_2025.08.08.25333315v1.4","doi-asserted-by":"publisher","DOI":"10.1101\/2025.03.12.243154"},{"key":"2025081404251119000_2025.08.08.25333315v1.5","unstructured":"Johnson A , Pollard T , Mark R. MIMIC-III clinical database (version 1.4). https:\/\/physionet.org\/content\/mimiciii\/1.4\/ (Accessed 27 May 2025)."},{"key":"2025081404251119000_2025.08.08.25333315v1.6","unstructured":"National Institutes of Health. Words matter \u2013 terms to use and avoid when talking about addiction. https:\/\/nida.nih.gov\/nidamed-medical-health-professionals\/health-professions-education\/words-matter-terms-to-use-avoid-when-talking-about-addiction (Accessed 27 May 2025)."},{"key":"2025081404251119000_2025.08.08.25333315v1.7","doi-asserted-by":"crossref","first-page":"424","DOI":"10.1097\/ADM.0000000000001145","article-title":"The incidence and disparities in use of stigmatizing language in clinical notes for patients with substance use disorder","volume":"17","year":"2023","journal-title":"J Addict Med"},{"key":"2025081404251119000_2025.08.08.25333315v1.8","doi-asserted-by":"publisher","DOI":"10.1377\/hlthaff.2015.1357"},{"key":"2025081404251119000_2025.08.08.25333315v1.9","doi-asserted-by":"publisher","DOI":"10.1007\/s10995-023-03772-2"},{"key":"2025081404251119000_2025.08.08.25333315v1.10","doi-asserted-by":"crossref","first-page":"e2144967","DOI":"10.1001\/jamanetworkopen.2021.44967","article-title":"Examination of stigmatizing language in the electronic health record","volume":"5","year":"2022","journal-title":"JAMA Netw Open"},{"key":"2025081404251119000_2025.08.08.25333315v1.11","doi-asserted-by":"crossref","first-page":"308","DOI":"10.1093\/jamia\/ocae290","article-title":"Identifying stigmatizing and positive\/preferred language in obstetric clinical notes using natural language processing","volume":"32","year":"2025","journal-title":"J Am Med Inform Assoc"},{"key":"2025081404251119000_2025.08.08.25333315v1.12","first-page":"312","volume-title":"Proc 61st Annu Meet Assoc Comput Linguist","volume":"2","year":"2023"},{"key":"2025081404251119000_2025.08.08.25333315v1.13","doi-asserted-by":"publisher","DOI":"10.1007\/s11606-022-07830-0"},{"key":"2025081404251119000_2025.08.08.25333315v1.14","unstructured":"AI@Meta. LLaMA 3 model card. https:\/\/github.com\/meta-llama\/llama3\/blob\/main\/MODEL_CARD.md (Accessed 27 May 2025)."},{"key":"2025081404251119000_2025.08.08.25333315v1.15","doi-asserted-by":"crossref","unstructured":"Wolf T , Debut L , Sanh V et al. Transformers: State-of-the-art natural language processing. In: Proc 2020 Conf Empir Methods Nat Lang Process: Syst Demonstrations, 38\u201345. Association for Computational Linguistics (2020).","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"2025081404251119000_2025.08.08.25333315v1.16","unstructured":"Paszke A , Gross S , Massa F , et al. PyTorch: An imperative style, high-performance deep learning library. In: Adv Neural Inf Process Syst 32. Curran Associates, Inc. (2019)."},{"key":"2025081404251119000_2025.08.08.25333315v1.17","doi-asserted-by":"crossref","unstructured":"Zheng Y , Zhang R , Zhang J et al. LLaMAFactory: Unified efficient fine-tuning of 100+ language models. In: Proc 62nd Annu Meet Assoc Comput Linguist (Vol. 3: System Demonstrations), Bangkok, Thailand. Association for Computational Linguistics (2024).","DOI":"10.18653\/v1\/2024.acl-demos.38"},{"key":"2025081404251119000_2025.08.08.25333315v1.18","doi-asserted-by":"publisher","DOI":"10.1016\/j.cpc.2010.04.018"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2025.08.08.25333315","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T20:42:55Z","timestamp":1768509775000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.1101\/2025.08.08.25333315"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,12]]},"references-count":18,"URL":"https:\/\/doi.org\/10.1101\/2025.08.08.25333315","relation":{},"subject":[],"published":{"date-parts":[[2025,8,12]]},"subtype":"preprint"}}