{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T14:12:05Z","timestamp":1772115125947,"version":"3.50.1"},"posted":{"date-parts":[[2024,12,12]]},"group-title":"Emergency Medicine","reference-count":33,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2024,12,12]],"date-time":"2024-12-12T00:00:00Z","timestamp":1733961600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2024,12,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>The accurate identification of Emergency Department (ED) encounters involving opioid misuse is critical for health services, research, and surveillance. We sought to develop natural language processing (NLP)-based models for the detection of ED encounters involving opioid misuse.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>A sample of ED encounters enriched for opioid misuse was manually annotated and clinical notes extracted. We evaluated classic machine learning (ML) methods, fine-tuning of publicly available pretrained language models, and a previously developed convolutional neural network opioid classifier for use on hospitalized patients (SMART-AI). Performance was compared to ICD-10-CM codes. Both raw text and text transformed to the United Medical Language System were evaluated. Face validity was evaluated by term feature importance.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>There were 1123 encounters used for training, validation, and testing. Of the classic ML methods, XGBoost had the highest AU_PRC (0.936), accuracy (0.887), and F1 score (0.863) which outperformed ICD-10-CM codes [accuracy 0.870; F1 0.830]. Logistic regression, support vector machine, and XGBoost models had higher AU_PRC using transformed text, while decision trees performed better using raw text. Excluding XGBoost, fine-tuned pre-trained language models outperformed classic ML methods. The best performing model was the fine-tuned SMART-AI based model with domain adaptation [AU_PRC 0.948; accuracy 0.882; F1 0.851]. Explainability analyses showed the most predictive terms were \u2018heroin\u2019, \u2018opioids\u2019, \u2018alcoholic intoxication, chronic\u2019, \u2018cocaine\u2019, \u2018opiates\u2019, and \u2018suboxone\u2019.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>NLP-based models outperform entry of ICD-10-CM diagnosis codes for the detection of ED encounters with opioid misuse. Fine tuning with domain adaptation for pre-trained language models resulted in improved performance.<\/jats:p>\n                <\/jats:sec>","DOI":"10.1101\/2024.12.11.24318875","type":"posted-content","created":{"date-parts":[[2024,12,12]],"date-time":"2024-12-12T23:21:38Z","timestamp":1734045698000},"source":"Crossref","is-referenced-by-count":2,"title":["Development and Evaluation of Machine Learning Models for the Detection of Emergency Department Patients with Opioid Misuse from Clinical Notes"],"prefix":"10.64898","author":[{"given":"Usman","family":"Shahid","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0072-7499","authenticated-orcid":false,"given":"Natalie","family":"Parde","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dale L.","family":"Smith","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Grayson","family":"Dickinson","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joseph","family":"Bianco","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dillon","family":"Thorpe","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Madhav","family":"Hota","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6368-4652","authenticated-orcid":false,"given":"Majid","family":"Afshar","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7650-3008","authenticated-orcid":false,"given":"Niranjan S.","family":"Karnik","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0945-1233","authenticated-orcid":false,"given":"Neeraj","family":"Chhabra","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"54368","reference":[{"key":"2024121404002785000_2024.12.11.24318875v1.1","unstructured":"Abuse NI on D. Drug Overdose Death Rates | National Institute on Drug Abuse (NIDA) [Internet]. 2024 [cited 2024 Aug 5];Available from: https:\/\/nida.nih.gov\/research-topics\/trends-statistics\/overdose-death-rates"},{"key":"2024121404002785000_2024.12.11.24318875v1.2","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMp1811988"},{"issue":"25","key":"2024121404002785000_2024.12.11.24318875v1.3","doi-asserted-by":"crossref","first-page":"567","DOI":"10.15585\/mmwr.mm7325a1","article-title":"Treatment for Opioid Use Disorder: Population Estimates \u2014 United States, 2022","volume":"73","year":"2024","journal-title":"Morb Mortal Wkly Rep"},{"issue":"3","key":"2024121404002785000_2024.12.11.24318875v1.4","doi-asserted-by":"crossref","first-page":"965","DOI":"10.1007\/s11606-019-05249-3","article-title":"US ED Opioid-Related Visits Increase, While Use of Medication for Opioid Use Disorder Undetectable, 2011-2016","volume":"35","year":"2020","journal-title":"J Gen Intern Med"},{"issue":"6","key":"2024121404002785000_2024.12.11.24318875v1.5","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1016\/j.jemermed.2019.03.018","article-title":"Screening Tools Validated in the Outpatient Pain Management Setting Poorly Predict Opioid Misuse in the Emergency Department: A Pilot Study","volume":"56","year":"2019","journal-title":"J Emerg Med"},{"key":"2024121404002785000_2024.12.11.24318875v1.6","unstructured":"Abuse NI on D. Summary of Misuse of Prescription Drugs [Internet]. Natl. Inst. Drug Abuse. -- [cited 2023 Jan 30];Available from: https:\/\/nida.nih.gov\/publications\/research-reports\/misuse-prescription-drugs\/overview"},{"key":"2024121404002785000_2024.12.11.24318875v1.7","unstructured":"Commonly Used Terms | Opioids | CDC [Internet]. 2023 [cited 2023 Oct 5];Available from: https:\/\/www.cdc.gov\/opioids\/basics\/terms.html"},{"key":"2024121404002785000_2024.12.11.24318875v1.8","unstructured":"2015 National Survey on Drug Use and Health: Methodological Summary and Definitions | CBHSQ Data [Internet]. [cited 2023 Jan 30];Available from: https:\/\/www.samhsa.gov\/data\/report\/2015-national-survey-drug-use-and-health-methodological-summary-and-definitions"},{"issue":"4","key":"2024121404002785000_2024.12.11.24318875v1.9","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1016\/j.annemergmed.2023.04.030","article-title":"Death by a Thousand Screens: A Practical Role for Machine Learning in Emergency Medicine","volume":"82","year":"2023","journal-title":"Ann Emerg Med"},{"key":"2024121404002785000_2024.12.11.24318875v1.10","doi-asserted-by":"crossref","unstructured":"Chhabra N , Smith D , Pachwicewicz P , et al. Performance of International Classification of Disease-10 codes in detecting emergency department patients with opioid misuse. Addict Abingdon Engl 2023;","DOI":"10.1111\/add.16394"},{"key":"2024121404002785000_2024.12.11.24318875v1.11","doi-asserted-by":"crossref","unstructured":"Sharma B , Dligach D , Swope K , et al. Publicly available machine learning models for identifying opioid misuse from the clinical notes of hospitalized patients. BMC Med Inform Decis Mak [Internet] 2020 [cited 2021 Jan 17];20. Available from: https:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC7191715\/","DOI":"10.1186\/s12911-020-1099-y"},{"issue":"1","key":"2024121404002785000_2024.12.11.24318875v1.12","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1186\/s13722-021-00229-7","article-title":"External validation of an opioid misuse machine learning classifier in hospitalized adult patients","volume":"16","year":"2021","journal-title":"Addict Sci Clin Pract"},{"issue":"6","key":"2024121404002785000_2024.12.11.24318875v1.13","doi-asserted-by":"crossref","first-page":"e426","DOI":"10.1016\/S2589-7500(22)00041-3","article-title":"Development and multimodal validation of a substance misuse algorithm for referral to treatment using artificial intelligence (SMART-AI): a retrospective deep learning study","volume":"4","year":"2022","journal-title":"Lancet Digit Health"},{"key":"2024121404002785000_2024.12.11.24318875v1.14","unstructured":"Anthony K . Cook County closes 2020 with record highs of 875 gun-related homicides, 1,599 opioid deaths [Internet]. Chic. Sun-Times. 2021 [cited 2021 Jan 26];Available from: https:\/\/chicago.suntimes.com\/metro-state\/2021\/1\/2\/22210281\/cook-county-homicide-total-2020-970-opioid-covid"},{"key":"2024121404002785000_2024.12.11.24318875v1.15","unstructured":"\u2022. As Opioid Overdose Deaths Hit New Record, Pressure Grows for Safe Places to Inject Drugs in Chicago [Internet]. NBC Chic. 2022 [cited 2024 Oct 23];Available from: https:\/\/www.nbcchicago.com\/news\/local\/as-opioid-overdose-deaths-hit-new-record-pressure-grows-for-safe-places-to-inject-drugs-in-chicago\/2730602\/"},{"issue":"7","key":"2024121404002785000_2024.12.11.24318875v1.16","doi-asserted-by":"crossref","first-page":"e0219717","DOI":"10.1371\/journal.pone.0219717","article-title":"Subtypes in patients with opioid misuse: A prognostic enrichment strategy using electronic health record data in hospitalized patients","volume":"14","year":"2019","journal-title":"PloS One"},{"key":"2024121404002785000_2024.12.11.24318875v1.17","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2008.08.010"},{"key":"2024121404002785000_2024.12.11.24318875v1.18","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2019.103208"},{"key":"2024121404002785000_2024.12.11.24318875v1.19","doi-asserted-by":"crossref","first-page":"103626","DOI":"10.1016\/j.jbi.2020.103626","article-title":"Pre-training phenotyping classifiers","volume":"113","year":"2021","journal-title":"J Biomed Inform"},{"key":"2024121404002785000_2024.12.11.24318875v1.20","unstructured":"Devlin J , Chang M-W , Lee K , Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [Internet]. 2019 [cited 2023 Jun 8];Available from: http:\/\/arxiv.org\/abs\/1810.04805"},{"key":"2024121404002785000_2024.12.11.24318875v1.21","unstructured":"Beltagy I , Peters ME , Cohan A. Longformer: The Long-Document Transformer [Internet]. 2020 [cited 2024 Oct 30];Available from: http:\/\/arxiv.org\/abs\/2004.05150"},{"key":"2024121404002785000_2024.12.11.24318875v1.22","unstructured":"Lee J , Yoon W , Kim S , et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining [Internet]. 2019 [cited 2024 Oct 30];Available from: http:\/\/arxiv.org\/abs\/1901.08746"},{"key":"2024121404002785000_2024.12.11.24318875v1.23","unstructured":"Li Y , Wehbe RM , Ahmad FS , Wang H , Luo Y. Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences [Internet]. 2022 [cited 2024 Oct 30];Available from: http:\/\/arxiv.org\/abs\/2201.11838"},{"issue":"5","key":"2024121404002785000_2024.12.11.24318875v1.24","doi-asserted-by":"crossref","first-page":"507","DOI":"10.1136\/jamia.2009.001560","article-title":"Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications","volume":"17","year":"2010","journal-title":"J Am Med Inform Assoc JAMIA"},{"issue":"Suppl 3","key":"2024121404002785000_2024.12.11.24318875v1.25","first-page":"74","article-title":"Comparison of MetaMap and cTAKES for entity extraction in clinical notes","volume":"18","year":"2018","journal-title":"BMC Med Inform Decis Mak"},{"issue":"2","key":"2024121404002785000_2024.12.11.24318875v1.26","first-page":"262","article-title":"ICD-10-CM-Based Definitions for Emergency Department Opioid Poisoning Surveillance: Electronic Health Record Case Confirmation Study","volume":"135","year":"2020","journal-title":"Public Health Rep Wash DC 1974"},{"key":"2024121404002785000_2024.12.11.24318875v1.27","unstructured":"Weiss AJ , McDermott KW , Heslin KC. Table 1, ICD-10-CM diagnosis codes defining different opioid-related conditions [Internet]. 2019 [cited 2021 Jan 18];Available from: http:\/\/www.ncbi.nlm.nih.gov\/books\/NBK538344\/table\/sb247.tab1\/"},{"key":"2024121404002785000_2024.12.11.24318875v1.28","doi-asserted-by":"crossref","unstructured":"Ribeiro MT , Singh S , Guestrin C . \u201cWhy Should I Trust You?\u201d: Explaining the Predictions of Any Classifier [Internet]. 2016 [cited 2024 Oct 31];Available from: http:\/\/arxiv.org\/abs\/1602.04938","DOI":"10.18653\/v1\/N16-3020"},{"key":"2024121404002785000_2024.12.11.24318875v1.29","unstructured":"Paszke A , Gross S , Massa F , et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library [Internet]. 2019 [cited 2024 Oct 31];Available from: http:\/\/arxiv.org\/abs\/1912.01703"},{"key":"2024121404002785000_2024.12.11.24318875v1.30","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2023-078378"},{"issue":"2","key":"2024121404002785000_2024.12.11.24318875v1.31","first-page":"262","article-title":"ICD-10-CM-Based Definitions for Emergency Department Opioid Poisoning Surveillance: Electronic Health Record Case Confirmation Study","volume":"135","year":"2020","journal-title":"Public Health Rep Wash DC 1974"},{"issue":"5","key":"2024121404002785000_2024.12.11.24318875v1.32","doi-asserted-by":"crossref","first-page":"577","DOI":"10.1002\/pds.5591","article-title":"Development and validation of an electronic health records-based opioid use disorder algorithm by expert clinical adjudication among patients with prescribed opioids","volume":"32","year":"2023","journal-title":"Pharmacoepidemiol Drug Saf"},{"issue":"9","key":"2024121404002785000_2024.12.11.24318875v1.33","doi-asserted-by":"crossref","first-page":"e2435603","DOI":"10.1001\/jamanetworkopen.2024.35603","article-title":"Trends and Disparities in Initiation of Buprenorphine in US Emergency Departments, 2013-2022","volume":"7","year":"2024","journal-title":"JAMA Netw Open"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2024.12.11.24318875","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T20:22:43Z","timestamp":1768508563000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.1101\/2024.12.11.24318875"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,12]]},"references-count":33,"URL":"https:\/\/doi.org\/10.1101\/2024.12.11.24318875","relation":{},"subject":[],"published":{"date-parts":[[2024,12,12]]},"subtype":"preprint"}}