{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T06:50:16Z","timestamp":1790232616166,"version":"4.1.0"},"reference-count":47,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"\n Background<\/jats:title>\n Sentiment analysis is one of the most widely used methods for mining and examining text. Social media researchers need guidance on choosing between manual and automated sentiment analysis methods.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n Popular sentiment analysis tools based on natural language processing (NLP; VADER [Valence Aware Dictionary for Sentiment Reasoning], TEXT2DATA [T2D], and Linguistic Inquiry and Word Count [LIWC-22]), and a large language model (ChatGPT 4.0) were compared with manually coded sentiment scores, as applied to the analysis of YouTube comments on videos discussing the opioid epidemic. Sentiment analysis methods were also examined regarding ease of programming, monetary cost, and other practical considerations.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Evaluation methods included descriptive statistics, receiver operating characteristic (ROC) curve analysis, confusion matrices, Cohen \u03ba, accuracy, specificity, precision, sensitivity (recall), F1-score harmonic mean, and the Matthews correlation coefficient. An inductive, iterative approach to content analysis of the data was used to obtain manual sentiment codes.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n A subset of comments were analyzed by a second coder, producing good agreement between the 2 coders\u2019 judgments (\u03ba=0.734). YouTube social media about the opioid crisis had many more negative comments (4286\/4871, 88%) than positive comments (79\/662, 12%), making it possible to evaluate the performance of sentiment analysis models in an unbalanced dataset. The tone summary measure from LIWC-22 performed better than other tools for estimating the prevalence of negative versus positive sentiment. According to the ROC curve analysis, VADER was best at classifying manually coded negative comments. A comparison of Cohen \u03ba values indicated that NLP tools (VADER, followed by LIWC\u2019s tone and T2D) showed only fair agreement with manual coding. In contrast, ChatGPT 4.0 had poor agreement and failed to generate binary sentiment scores in 2 out of 3 attempts. Variations in accuracy, specificity, precision, sensitivity, F1-score, and MCC did not reveal a single superior model. F1-score harmonic means were 0.34-0.38 (SD 0.02) for NLP tools and very low (0.13) for ChatGPT 4.0. None of the MCCs reached a strong correlation level.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Researchers studying negative emotions, public worries, or dissatisfaction with social media face unique challenges in selecting models suitable for unbalanced datasets. We recommend VADER, the only cost-free tool we evaluated, due to its excellent discrimination, which can be further improved when the comments are at least 100 characters long. If estimating the prevalence of negative comments in an unbalanced dataset is important, we recommend the tone summary measure from LIWC-22. Researchers using T2D must know that it may only score some data and, compared with other methods, be more time-consuming and cost-prohibitive. A general-purpose large language model, ChatGPT 4.0, has yet to surpass the performance of NLP models, at least for unbalanced datasets with highly prevalent (7:1) negative comments.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/57395","type":"journal-article","created":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T01:25:18Z","timestamp":1726795518000},"page":"e57395","source":"Crossref","is-referenced-by-count":28,"title":["Public Health Discussions on Social Media: Evaluating Automated Sentiment Analysis Methods"],"prefix":"10.2196","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6487-8064","authenticated-orcid":false,"given":"Lisa M","family":"Gandy","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4138-6646","authenticated-orcid":false,"given":"Lana V","family":"Ivanitskaya","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-8229-816X","authenticated-orcid":false,"given":"Leeza L","family":"Bacon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8769-9597","authenticated-orcid":false,"given":"Rodina","family":"Bizri-Baryak","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,1,8]]},"reference":[{"key":"ref1","unstructured":"Social Media Fact Sheet2024-10-02https:\/\/www.pewresearch.org\/internet\/fact-sheet\/social-media\/"},{"key":"ref2","unstructured":"Ortiz-OspinaEThe rise of social mediaOur World in Data2024-10-02https:\/\/ourworldindata.org\/rise-of-social-media"},{"key":"ref3","doi-asserted-by":"crossref","first-page":"651","DOI":"10.4135\/9781529782943.n45","author":"Khan, ML","year":"2022","journal-title":"The Sage Handbook of Social Media Research Methods"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyt.2023.1121583"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.1403"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2020.102221"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1016\/j.aej.2023.08.038"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2016.05.040"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v8i1.14550"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1177\/0047287517747753"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1140\/epjds\/s13688-016-0085-1"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1145\/3361785.3361814"},{"key":"ref13","author":"Fasha, EF","year":"2021","journal-title":"Advances on Intelligent Informatics and Computing"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1017\/pan.2022.10"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.5860\/choice.45-0602"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/w17-3110"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.2196\/21418"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1109\/iccmc51019.2021.9418320"},{"key":"ref19","unstructured":"TymannKLutzMPalsbr\u00f6kerPGipsCGerVADER-A German Adaptation of the VADER Sentiment Analysis tool for Social Media Texts2021Lernen, Wissen, Daten, Analysen2024 September 23-25Germany178189"},{"key":"ref20","unstructured":"TEXT2DATA2024-10-02https:\/\/text2data.com\/sentiment-analysis-api"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.1109\/icsssm.2015.7170341"},{"key":"ref22","unstructured":"RameshDThe correlation between Twitter sentiments and polling results for the 2016 Presidential raceSouth Carolina Junior Academy of Science20162024-10-02https:\/\/scholarexchange.furman.edu\/scjas\/2016\/all\/273"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-05061-9_28"},{"key":"ref24","unstructured":"BoydRLAshokkumarASerajSPennebakerJWThe development and psychometric properties of LIWC-22The University of Texas at Austin20222024-10-02https:\/\/www.liwc.app"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2017.11.151"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.1177\/1750481317699347"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1109\/access.2020.3000093"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1080\/19312458.2019.1671966"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijresmar.2018.09.009"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1016\/j.metrad.2023.100017"},{"key":"ref31","doi-asserted-by":"crossref","first-page":"343","DOI":"10.21093\/ijeltal.v7i2.1387","volume":"7","author":"Baskara, R","year":"2023","journal-title":"IJELTAL"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.3389\/feduc.2023.1166682"},{"key":"ref33","unstructured":"BaconLLOpioid epidemic: CNN and Fox News Youtube viewers' stories, claims, and solutions2024-10-02https:\/\/tinyurl.com\/yemxye4f"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1111\/j.0956-7976.2004.00741.x"},{"key":"ref35","doi-asserted-by":"publisher","DOI":"10.1186\/s13040-023-00322-4"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.1186\/s12864-019-6413-7"},{"key":"ref37","doi-asserted-by":"crossref","DOI":"10.1201\/9781420075779","author":"Hilbe, JM","year":"2009","journal-title":"Logistic Regression Models"},{"key":"ref38","first-page":"362","author":"Gonen, M","year":"2007","journal-title":"Analyzing receiver operating characteristic curves with SAS"},{"key":"ref39","doi-asserted-by":"crossref","DOI":"10.1002\/9781118548387","author":"Hosmer, D","year":"2013","journal-title":"Applied Logistic Regression"},{"key":"ref40","unstructured":"Laerd Statistics2002-10-24https:\/\/statistics.laerd.com\/premium\/spss\/ck\/cohens-kappa-in-spss-7.php"},{"key":"ref41","doi-asserted-by":"publisher","DOI":"10.1016\/b978-0-12-824145-5.00005-8"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.3390\/vaccines10101713"},{"key":"ref43","doi-asserted-by":"publisher","DOI":"10.1101\/2022.07.20.22277865"},{"key":"ref44","doi-asserted-by":"publisher","DOI":"10.1093\/pnasnexus\/pgad219"},{"key":"ref45","doi-asserted-by":"publisher","DOI":"10.31234\/osf.io\/x9e7u"},{"key":"ref46","doi-asserted-by":"publisher","DOI":"10.1080\/23808985.2013.11679143"},{"key":"ref47","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2019.01065"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/formative.jmir.org\/2025\/1\/e57395","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:47:58Z","timestamp":1782841678000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2025\/1\/e57395"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,8]]},"references-count":47,"URL":"https:\/\/doi.org\/10.2196\/57395","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.57395","asserted-by":"object"}]},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,8]]}}}