{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T06:45:22Z","timestamp":1790318722809,"version":"4.1.0"},"reference-count":34,"publisher":"Public Library of Science (PLoS)","issue":"9","license":[{"start":{"date-parts":[[2025,9,30]],"date-time":"2025-09-30T00:00:00Z","timestamp":1759190400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosdigitalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Digit Health"],"abstract":"<jats:p>Adverse event (AE) detection is labor-intensive and costly given the task is to find rare events. Automated solutions to enhance efficiency, reduce costs, and capture unnoticed safety signals are needed. To develop and evaluate an automated machine learning tool, \u201cWaldo,\u201d for AE detection from unstructured social media text data, specifically targeting consumer health products that lack traditional post-market surveillance channels. We tested three models \u2013 (i) N-gram model, (ii) BERT (Bidirectional Encoder Representations from Transformers), and (iii) RoBERTa (Robustly optimized BERT approach) \u2013 trained on 10,000 previously published unstructured reports on cannabis-derived products (CDPs) annotated by humans for the presence of adverse events to determine the best-performing AE detection method. This method was then benchmarked against an AI chatbot (ChatGPT: gpt-3.5-turbo-0613) and applied to previously unstudied user narratives about CDPs from 20 subreddits.RoBERTa demonstrated the highest accuracy at 99.7%, hereafter referred to as Waldo, with 22 false positives and 12 false negatives, yielding an F1-score of 95.1% for the positive class. In contrast, the chatbot had an accuracy of 94.4%, with 401 false positives (18.23-fold more than Waldo) and 163 false negatives (13.58-fold more than Waldo), yielding an F1-score of 38% for the positive class. Applying Waldo to 437,132 posts identified 28,832 potential AEs. The subreddit r\/Marijuana had the highest AE rate (12.7%), followed by r\/weed (10.5%) and r\/AskTrees (10.0%). r\/weedstocks (0.1%), r\/macrogrowery (0.2%), and r\/weedbiz (0.2%) had the lowest rates of potential AEs. Waldo addresses critical gaps in safety surveillance for unregulated consumer health products by automatically detecting adverse events from social media\u2014a capability absent in traditional industry systems. Unlike existing approaches limited to structured databases or narrow domains, Waldo processes informal user narratives at scale with high precision. We have open-sourced Waldo for immediate application by the health community [<jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/waldo-ae-detection.github.io\/WALDO\/\" xlink:type=\"simple\">https:\/\/waldo-ae-detection.github.io\/WALDO\/<\/jats:ext-link>].<\/jats:p>","DOI":"10.1371\/journal.pdig.0001011","type":"journal-article","created":{"date-parts":[[2025,9,30]],"date-time":"2025-09-30T17:21:13Z","timestamp":1759252873000},"page":"e0001011","update-policy":"https:\/\/doi.org\/10.1371\/journal.pdig.corrections_policy","source":"Crossref","is-referenced-by-count":1,"title":["Waldo: Automated discovery of adverse events from unstructured self reports"],"prefix":"10.1371","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0515-125X","authenticated-orcid":true,"given":"Karan S.","family":"Desai","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vijay M.","family":"Tiyyala","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pranav","family":"Tiyyala","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Atharva","family":"Yeola","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alejandra","family":"Gallegos-Rangel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alejandro","family":"Montiel-Torres","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-6372-9405","authenticated-orcid":true,"given":"Matthew R.","family":"Allen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark","family":"Dredze","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ryan G.","family":"Vandrey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8929-9579","authenticated-orcid":true,"given":"Johannes","family":"Thrul","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eric C.","family":"Leas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mike","family":"Hogarth","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Davey M.","family":"Smith","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John W.","family":"Ayers","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2025,9,30]]},"reference":[{"issue":"5","key":"pdig.0001011.ref001","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1007\/s40264-014-0155-x","article-title":"Digital drug safety surveillance: monitoring pharmaceutical products in twitter","volume":"37","author":"CC Freifeld","year":"2014","journal-title":"Drug Saf"},{"issue":"1","key":"pdig.0001011.ref002","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1002\/pds.2261","article-title":"Disproportionality analysis for signal detection of implantable cardioverter-defibrillator-related adverse events in the Food and Drug Administration Medical Device Reporting System","volume":"21","author":"HJ Duggirala","year":"2012","journal-title":"Pharmacoepidemiol Drug Saf"},{"key":"pdig.0001011.ref003","article-title":"Marketing medicine to millennials: Preparing institutions and regulations for direct-to-consumer healthcare","volume":"55","author":"M Levy","year":"2019","journal-title":"California Western Law Review"},{"key":"pdig.0001011.ref004","volume-title":"Making the healthcare shift: The transformation to consumer-centricity","author":"SM Davis","year":"2018"},{"issue":"7","key":"pdig.0001011.ref005","doi-asserted-by":"crossref","first-page":"402","DOI":"10.1097\/NHH.0000000000000107","article-title":"Medical devices, the FDA, and the home healthcare clinician","volume":"32","author":"LK Simone","year":"2014","journal-title":"Home Healthc Nurse"},{"issue":"1","key":"pdig.0001011.ref006","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1186\/s42238-023-00191-y","article-title":"Self-reported adverse events associated with \u22068-Tetrahydrocannabinol (Delta-8-THC) Use","volume":"5","author":"EC Leas","year":"2023","journal-title":"J Cannabis Res"},{"key":"pdig.0001011.ref007","doi-asserted-by":"crossref","first-page":"309","DOI":"10.3389\/fphar.2016.00309","article-title":"Current Therapeutic Cannabis Controversies and Clinical Trial Design Issues","volume":"7","author":"EB Russo","year":"2016","journal-title":"Front Pharmacol"},{"key":"pdig.0001011.ref008","unstructured":"CDC Archives. https:\/\/archive.cdc.gov\/#\/details?url=https:\/\/www.cdc.gov\/tobacco\/basic_information\/e-cigarettes\/severe-lung-disease.html. 2024 October 7."},{"key":"pdig.0001011.ref009","first-page":"1","article-title":"Pharmacovigilance in the Age of Legalized Cannabis: Using Social Media to Monitor Drug\u2013Drug Interactions Between Immunosuppressants and Cannabis-Derived Products","author":"MR Allen","year":"2024","journal-title":"Drug Saf"},{"issue":"1","key":"pdig.0001011.ref010","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1186\/s12859-020-3509-7","article-title":"Predicting potential adverse events using safety data from marketed drugs","volume":"21","author":"C Daluwatte","year":"2020","journal-title":"BMC Bioinformatics"},{"issue":"5","key":"pdig.0001011.ref011","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1093\/ijpp\/riab037","article-title":"The cost of pharmacovigilance: a time and motion study of an adverse drug reaction program","volume":"29","author":"GB McLachlan","year":"2021","journal-title":"Int J Pharm Pract"},{"issue":"17","key":"pdig.0001011.ref012","doi-asserted-by":"crossref","first-page":"1708","DOI":"10.1001\/jama.2017.11909","article-title":"Labeling Accuracy of Cannabidiol Extracts Sold Online","volume":"318","author":"MO Bonn-Miller","year":"2017","journal-title":"JAMA"},{"key":"pdig.0001011.ref013","doi-asserted-by":"crossref","first-page":"2491","DOI":"10.1001\/jama.2015.6613","article-title":"Cannabinoid dose label accuracy \u2018edible\u2019 medical cannabis products","volume":"313","author":"R Vandrey","year":"2015","journal-title":"J Am Med Assoc"},{"key":"pdig.0001011.ref014","doi-asserted-by":"crossref","first-page":"1335441","DOI":"10.3389\/fphar.2024.1335441","article-title":"Product labeling accuracy and contamination analysis of commercially available cannabidiol product samples","volume":"15","author":"BE Gidal","year":"2024","journal-title":"Front Pharmacol"},{"key":"pdig.0001011.ref015","author":"Office of the Commissioner","year":"2024"},{"key":"pdig.0001011.ref016","article-title":"Legalized marijuana products still go largely unregulated","author":"C Smith","year":"2023","journal-title":"Governing"},{"issue":"35","key":"pdig.0001011.ref017","doi-asserted-by":"crossref","DOI":"10.1503\/cmaj.181035","article-title":"Minimizing the adverse public health effects of cannabis legalization","volume":"190","author":"W Hall","year":"2018","journal-title":"CMAJ"},{"key":"pdig.0001011.ref018","unstructured":"Devlin J, Chang MW, Lee K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2018."},{"key":"pdig.0001011.ref019","unstructured":"Liu Y, Ott M, Goyal N. RoBERTa: A Robustly Optimized BERT Pretraining Approach. 2019."},{"key":"pdig.0001011.ref020","unstructured":"FDA\u2019s Sentinel Initiative. https:\/\/www.fda.gov\/safety\/fdas-sentinel-initiative. 2024 September 14."},{"key":"pdig.0001011.ref021","doi-asserted-by":"crossref","first-page":"1399","DOI":"10.1001\/jama.2014.1505","article-title":"Could behavioral medicine lead the web data revolution?","volume":"311","author":"JW Ayers","year":"2014","journal-title":"JAMA"},{"issue":"1","key":"pdig.0001011.ref022","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1093\/jamia\/ocad197","article-title":"Using natural language processing to characterize and predict homeopathic product-associated adverse events in consumer reviews: comparison to reports to FDA Adverse Event Reporting System (FAERS)","volume":"31","author":"K Konkel","year":"2023","journal-title":"J Am Med Inform Assoc"},{"key":"pdig.0001011.ref023","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1007\/s40264-022-01155-6","article-title":"Machine learning in causal inference: application in pharmacovigilance","volume":"45","author":"Y Zhao","year":"2022","journal-title":"Drug Saf"},{"issue":"1","key":"pdig.0001011.ref024","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1093\/jamia\/ocx045","article-title":"Mining e-cigarette adverse events in social media using Bi-LSTM recurrent neural network with word embedding representation","volume":"25","author":"J Xie","year":"2018","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"pdig.0001011.ref025","doi-asserted-by":"crossref","first-page":"14604582221136712","DOI":"10.1177\/14604582221136712","article-title":"Identifying adverse drug reactions from patient reviews on social media using natural language processing","volume":"29","author":"O Oyebode","year":"2023","journal-title":"Health Informatics J"},{"key":"pdig.0001011.ref026","doi-asserted-by":"crossref","first-page":"1605","DOI":"10.1109\/BIBM52615.2021.9669410","author":"MY Yan","year":"2021","journal-title":"2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)"},{"issue":"12","key":"pdig.0001011.ref027","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pdig.0000409","article-title":"BERT based natural language processing for triage of adverse drug reaction reports shows close to human-level performance","volume":"2","author":"E Bergman","year":"2023","journal-title":"PLOS Digit Health"},{"issue":"4","key":"pdig.0001011.ref028","doi-asserted-by":"crossref","DOI":"10.1093\/jamiaopen\/ooab081","article-title":"Deep learning models in detection of dietary supplement adverse event signals from Twitter","volume":"4","author":"Y Wang","year":"2021","journal-title":"JAMIA Open"},{"issue":"22","key":"pdig.0001011.ref029","doi-asserted-by":"crossref","first-page":"2163","DOI":"10.1001\/jama.2019.4432","article-title":"The Need for Federal Regulation of Marijuana Marketing","volume":"321","author":"JW Ayers","year":"2019","journal-title":"JAMA"},{"issue":"10","key":"pdig.0001011.ref030","article-title":"Self-reported Cannabidiol (CBD) Use for Conditions With Proven Therapies","volume":"3","author":"EC Leas","year":"2020","journal-title":"JAMA Netw Open"},{"key":"pdig.0001011.ref031","doi-asserted-by":"crossref","first-page":"1282831","DOI":"10.3389\/fphar.2024.1282831","article-title":"Systematic review of drug-drug interactions of delta-9-tetrahydrocannabinol, cannabidiol, and Cannabis","volume":"15","author":"R Nachnani","year":"2024","journal-title":"Front Pharmacol"},{"key":"pdig.0001011.ref032","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1016\/j.transproceed.2023.11.013","article-title":"Drug-drug interaction between cannabidiol, cyclosporine, and mycophenolate mofetil: A case report","volume":"56","author":"L Cu\u00f1etti","year":"2024","journal-title":"Transplant Proc"},{"issue":"3","key":"pdig.0001011.ref033","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1109\/MIC.2020.3037151","article-title":"Chatbots as Conversational Healthcare Services","volume":"25","author":"M Jovanovic","year":"2021","journal-title":"IEEE Internet Comput"},{"key":"pdig.0001011.ref034","doi-asserted-by":"crossref","first-page":"7949","DOI":"10.18653\/v1\/2020.emnlp-main.638","article-title":"Active learning for BERT: An empirical study","author":"L Ein-Dor","year":"2020","journal-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing"}],"container-title":["PLOS Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001011","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,30]],"date-time":"2025-09-30T17:21:17Z","timestamp":1759252877000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001011"}},"subtitle":[],"editor":[{"given":"Ali","family":"Nabavizadeh","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2025,9,30]]},"references-count":34,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2025,9,30]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pdig.0001011","relation":{},"ISSN":["2767-3170"],"issn-type":[{"value":"2767-3170","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,30]]}}}