{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T19:09:45Z","timestamp":1788980985005,"version":"build-2803163510"},"reference-count":33,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n \n Overdose and suicide due to nonprescribed fentanyl use have increased significantly, yet health care systems lack reliable methods to identify patients who use nonprescribed fentanyl.\n International Classification of Diseases<\/jats:italic>\n codes are inconsistent and do not specify nonprescribed fentanyl use.\n <\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to develop natural language processing approaches to identifying nonprescribed fentanyl use in electronic health record (EHR) documentation.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n This retrospective study included Veterans Health Administration patients seen between April 5, 2023, and December 23, 2024. A term list was developed to identify fentanyl-related mentions in clinical text, and 250-character snippets surrounding identified mentions were extracted. Veterans (n=3878) were randomly sampled from 5 predefined groups based on the presence of 1 of 4 terms (\u201cfent,\u201d \u201cblues,\u201d \u201cM30s,\u201d and \u201ctranq\u201d) in their EHR documentation. Physician annotators classified snippets into \u201cnonprescribed fentanyl use,\u201d \u201cprescribed fentanyl use,\u201d or \u201cother,\u201d with interannotator agreement evaluated using the mean pairwise Cohen \u03ba. Cross-validation folds were constructed at the patient level between training and test sets. Penalized logistic regression, Bio-ClinicalBERT, Llama 3-8B, and Mistral-7B were trained on labeled data and compared. Model performance was evaluated using precision, recall, and\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores for each class, with a focus on the nonprescribed fentanyl use class as the primary label of clinical interest using bootstrapped 95% CIs. A fairness analysis and Shapley additive explanations analysis were performed using Bio-ClinicalBERT. External validation was performed using Bio-ClinicalBERT on an independent sample of 200 snippets, each representing a unique patient from January 2025 to June 2026, with precision reported as the primary validation metric.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Of 7389 snippets, 9.6% (n=709) were classified as \u201cnonprescribed fentanyl use,\u201d 40.3% (n=2981) were classified as \u201cprescribed fentanyl use,\u201d and 50% (n=3699) were classified as \u201cother.\u201d Interannotator agreement was high (\u03ba=0.822). Llama 3-8B achieved the highest\n F<\/jats:italic>\n 1<\/jats:sub>\n -score for nonprescribed fentanyl use (0.87, 95% CI 0.83-0.92), followed by Mistral-7B (0.80, 95% CI 0.75-0.84), Bio-ClinicalBERT (0.80, 95% CI 0.74-0.85), and penalized logistic regression (0.74, 95% CI 0.73-0.75). Performance was consistent across demographic subgroups, with lower performance for the nonprescribed fentanyl use class observed in female and Hispanic subgroups. Shapley additive explanations analysis revealed clinically meaningful discriminating terms for each class, although subword tokens required contextual interpretation. External validation of Bio-ClinicalBERT demonstrated a precision of 0.79 for nonprescribed fentanyl use.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Natural language processing can identify nonprescribed fentanyl use in EHR documentation, although model performance for this class was lower than overall model performance, reflecting the clinical complexity of identifying nonprescribed use and the variable ways in which clinicians document this problem. This approach may support risk prediction and targeting of interventions to patients exposed to nonprescribed fentanyl.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/97485","type":"journal-article","created":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T19:55:06Z","timestamp":1784577306000},"page":"e97485-e97485","source":"Crossref","is-referenced-by-count":0,"title":["Natural Language Processing Identification of Nonprescribed Fentanyl Use in Electronic Health Records: Algorithm Development and Validation Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0882-463X","authenticated-orcid":false,"given":"Joshua","family":"Trujeque","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6971-6021","authenticated-orcid":false,"given":"Joseph A","family":"Simonetti","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-0407-2559","authenticated-orcid":false,"given":"Isai","family":"Ortiz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0292-0594","authenticated-orcid":false,"given":"Nicholas E","family":"Ingraham","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8419-0339","authenticated-orcid":false,"given":"Nathan","family":"Mesfin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-4131-6343","authenticated-orcid":false,"given":"Jeremy","family":"Yeung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8258-3585","authenticated-orcid":false,"given":"Rui","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2629-214X","authenticated-orcid":false,"given":"Lisa A","family":"Brenner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8532-8552","authenticated-orcid":false,"given":"R Adams","family":"Dudley","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,9]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.15620\/cdc\/170565","article-title":"Drug overdose deaths in the United States, 2003\u20132023","author":"Garnett","journal-title":"NCHS Data Brief"},{"issue":"48","key":"R2","doi-asserted-by":"publisher","first-page":"1099","DOI":"10.15585\/mmwr.mm7348a2","article-title":"Detection of illegally manufactured fentanyls and carfentanil in drug overdose deaths - United States, 2021-2024","volume":"73","author":"Tanz","journal-title":"MMWR Morb Mortal Wkly Rep"},{"issue":"6","key":"R3","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.14997","article-title":"Decline in US drug overdose deaths by region, substance, and demographics","volume":"8","author":"Post","journal-title":"JAMA Netw Open"},{"issue":"2","key":"R4","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.58090","article-title":"Recent drug overdose mortality decline compared with pre-COVID-19 trend","volume":"8","author":"Kiang","journal-title":"JAMA Netw Open"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1016\/j.drugalcdep.2021.108537","article-title":"Using ICD-10-CM codes to detect illicit substance use: a comparison with retrospective self-report","volume":"221","author":"Rowe","journal-title":"Drug Alcohol Depend"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1016\/j.drugalcdep.2019.107825","article-title":"Validity of International Classification of Diseases codes in identifying illicit drug use target conditions using medical record data as a reference standard: a systematic review","volume":"208","author":"McGrew","journal-title":"Drug Alcohol Depend"},{"issue":"5","key":"R7","doi-asserted-by":"publisher","first-page":"577","DOI":"10.1002\/pds.5591","article-title":"Development and validation of an electronic health records-based opioid use disorder algorithm by expert clinical adjudication among patients with prescribed opioids","volume":"32","author":"Ranapurwala","journal-title":"Pharmacoepidemiol Drug Saf"},{"issue":"6","key":"R8","doi-asserted-by":"publisher","first-page":"691","DOI":"10.1001\/jamainternmed.2023.7282","article-title":"Caring for hospitalized adults with opioid use disorder in the era of fentanyl: a review","volume":"184","author":"Englander","journal-title":"JAMA Intern Med"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.drugalcdep.2022.109630","article-title":"Characteristics and correlates of fentanyl preferences among people with opioid use disorder","volume":"240","author":"Hochstatter","journal-title":"Drug Alcohol Depend"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1016\/j.drugalcdep.2018.09.006","article-title":"Characterizing fentanyl-related overdoses and implications for overdose response: findings from a rapid ethnographic study in Vancouver, Canada","volume":"193","author":"Mayer","journal-title":"Drug Alcohol Depend"},{"issue":"8","key":"R11","doi-asserted-by":"publisher","first-page":"1143","DOI":"10.1002\/pds.4810","article-title":"Using natural language processing of clinical text to enhance identification of opioid-related overdoses in electronic health records data","volume":"28","author":"Hazlehurst","journal-title":"Pharmacoepidemiol Drug Saf"},{"issue":"8","key":"R12","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0309424","article-title":"A neural network approach to predict opioid misuse among previously hospitalized patients using electronic health records","volume":"19","author":"Vega","journal-title":"PLOS ONE"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.2196\/53366","article-title":"Developing a framework to infer opioid use disorder severity from clinical notes to inform natural language processing methods: characterization study","volume":"11","author":"Poulsen","journal-title":"JMIR Ment Health"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2022.104963","article-title":"Using natural language processing to identify opioid use disorder in electronic health record data","volume":"170","author":"Singleton","journal-title":"Int J Med Inform"},{"issue":"11","key":"R15","doi-asserted-by":"publisher","first-page":"1600","DOI":"10.7326\/M21-2392","article-title":"Health care for our nation's veterans: a policy paper from the American College of Physicians","volume":"174","author":"Crowley","journal-title":"Ann Intern Med"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1016\/j.drugalcdep.2022.109296","article-title":"Veteran drug overdose mortality, 2010-2019","volume":"233","author":"Begley","journal-title":"Drug Alcohol Depend"},{"key":"R17","unstructured":"Harm reduction for veterans with substance use disorder. U.S. Department of Veterans Affairs. URL: https:\/\/www.mentalhealth.va.gov\/substance-use\/harm-reduction.asp [Accessed 05-12-2025]"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","first-page":"1826","DOI":"10.1080\/07853890.2022.2092896","article-title":"U.S. military veterans and the opioid overdose crisis: a review of risk factors and prevention efforts","volume":"54","author":"Bennett","journal-title":"Ann Med"},{"issue":"24","key":"R19","doi-asserted-by":"publisher","first-page":"2374","DOI":"10.1001\/jama.2019.20562","article-title":"VA efforts to reduce opioid overdose deaths in at-risk veterans","volume":"322","author":"Rubin","journal-title":"JAMA"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1037\/ser0000099","article-title":"Development and applications of the Veterans Health Administration\u2019s Stratification Tool for Opioid Risk Mitigation (STORM) to improve opioid safety and prevent overdose and suicide","volume":"14","author":"Oliva","journal-title":"Psychol Serv"},{"issue":"9596","key":"R21","doi-asserted-by":"publisher","first-page":"1453","DOI":"10.1016\/S0140-6736(07)61602-X","article-title":"The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies","volume":"370","author":"von Elm","journal-title":"Lancet"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","first-page":"113","DOI":"10.1093\/jamia\/ocae169","article-title":"Comparison of six natural language processing approaches to assessing firearm access in Veterans Health Administration electronic health records","volume":"32","author":"Trujeque","journal-title":"J Am Med Inform Assoc"},{"key":"R23","unstructured":"Labeling images and text documents. Microsoft. URL: https:\/\/learn.microsoft.com\/en-us\/azure\/machine-learning\/how-to-label-data?view=azureml-api-2 [Accessed 06-02-2025]"},{"issue":"1","key":"R24","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"Landis","journal-title":"Biometrics"},{"key":"R25","unstructured":"Jiang AQ Sablayrolles A Mensch A et al. Mistral 7B. arXiv. Preprint posted online on Oct 10, 2023. [doi: 10.48550\/arXiv.2310.06825]"},{"key":"R26","unstructured":"Grattafiori A Dubey A Jauhri A et al. The Llama 3 herd of models. arXiv. Preprint posted online on Jul 31, 2024. [doi: 10.48550\/arXiv.2407.21783]"},{"key":"R27","unstructured":"Lundberg S Lee SI . A unified approach to interpreting model predictions. arXiv. Preprint posted online on May 22, 2017. [doi: 10.48550\/arXiv.1705.07874]"},{"issue":"4","key":"R28","doi-asserted-by":"publisher","first-page":"493","DOI":"10.1177\/1077558715617382","article-title":"Do Veterans Health Administration enrollees generalize to other populations?","volume":"73","author":"Wong","journal-title":"Med Care Res Rev"},{"issue":"2","key":"R29","doi-asserted-by":"publisher","DOI":"10.71465\/fair724","article-title":"Interpretable and lightweight predictive modeling for congestive heart failure using ICU electronic health records","volume":"3","author":"Wang","journal-title":"Front Artif Intell Res"},{"issue":"9","key":"R30","doi-asserted-by":"publisher","first-page":"1935","DOI":"10.2105\/AJPH.2015.302737","article-title":"Predictive modeling and concentration of the risk of suicide: implications for preventive interventions in the US Department of Veterans Affairs","volume":"105","author":"McCarthy","journal-title":"Am J Public Health"},{"issue":"10","key":"R31","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2021.29900","article-title":"Evaluation of the Recovery Engagement and Coordination for Health-Veterans Enhanced Treatment suicide risk modeling clinical program in the Veterans Health Administration","volume":"4","author":"McCarthy","journal-title":"JAMA Netw Open"},{"issue":"14","key":"R32","doi-asserted-by":"publisher","first-page":"3746","DOI":"10.1007\/s11606-022-07622-1","article-title":"Randomized policy evaluation of the Veterans Health Administration Stratification Tool for Opioid Risk Mitigation (STORM)","volume":"37","author":"Minegishi","journal-title":"J Gen Intern Med"},{"key":"R33","doi-asserted-by":"publisher","DOI":"10.2196\/90759","article-title":"Toward retrieval-grounded evaluation for conversational large language model-based risk assessment","volume":"5","author":"Hu","journal-title":"JMIR AI"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,9]],"date-time":"2026-09-09T18:42:54Z","timestamp":1788979374000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e97485"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,9]]},"references-count":33,"URL":"https:\/\/doi.org\/10.2196\/97485","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,9]]},"article-number":"v28i48e97485"}}