{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,12]],"date-time":"2026-10-12T16:07:39Z","timestamp":1791821259222,"version":"4.3.4"},"reference-count":44,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n In the digital age, social media has become a crucial platform for public discourse on diverse health-related topics, including vaccines. Efficient sentiment analysis and hesitancy detection are essential for understanding public opinions and concerns. Large language models (LLMs) offer advanced capabilities for processing complex linguistic patterns, potentially providing valuable insights into vaccine-related discourse.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to evaluate the performance of various LLMs in sentiment analysis and hesitancy detection related to vaccine discussions on social media and identify the most efficient, accurate, and cost-effective model for detecting vaccine-related public sentiment and hesitancy trends.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n We used several LLMs\u2014generative pretrained transformer (GPT-3.5), GPT-4, Claude-3 Sonnet, and Llama 2\u2014to process and classify complex linguistic data related to human papillomavirus; measles, mumps, and rubella; and vaccines overall from X (formerly known as Twitter), Reddit, and YouTube. The models were tested across different learning paradigms: zero-shot, 1-shot, and few-shot to determine their adaptability and learning efficiency with varying amounts of training data. We evaluated the models\u2019 performance using accuracy,\n F<\/jats:italic>\n 1<\/jats:sub>\n -score, precision, and recall. In addition, we conducted a cost analysis focused on token usage to assess the computational efficiency of each approach.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n GPT-4 (\n F<\/jats:italic>\n 1<\/jats:sub>\n -score=0.85 and accuracy=0.83) outperformed GPT-3.5, Llama 2, and Claude-3 Sonnet across various metrics, regardless of the sentiment type or learning paradigm. Few-shot learning did not significantly enhance performance compared with the zero-shot paradigm. Moreover, the increased computational costs and token usage associated with few-shot learning did not justify its application, given the marginal improvement in model performance. The analysis highlighted challenges in classifying neutral sentiments and convenience, correctly interpreting sarcasm, and accurately identifying indirect expressions of vaccine hesitancy, emphasizing the need for model refinement.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n GPT-4 emerged as the most accurate model, excelling in sentiment and hesitancy analysis. Performance differences between learning paradigms were minimal, making zero-shot learning preferable for its balance of accuracy and computational efficiency. However, the zero-shot GPT-4 model is not the most cost-effective compared with traditional machine learning. A hybrid approach, using LLMs for initial annotation and traditional models for training, could optimize cost and performance. Despite reliance on specific LLM versions and a limited focus on certain vaccine types and platforms, our findings underscore the capabilities and limitations of LLMs in vaccine sentiment and hesitancy analysis, highlighting the need for ongoing evaluation and adaptation in public health communication strategies.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/64723","type":"journal-article","created":{"date-parts":[[2025,10,15]],"date-time":"2025-10-15T18:51:56Z","timestamp":1760554316000},"page":"e64723-e64723","source":"Crossref","is-referenced-by-count":13,"title":["Evaluating Large Language Models for Sentiment Analysis and Hesitancy Analysis on Vaccine Posts From Social Media: Qualitative Study"],"prefix":"10.2196","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2696-6765","authenticated-orcid":false,"given":"Augustine","family":"Annan","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9422-1215","authenticated-orcid":false,"given":"Amanda L","family":"Eiden","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-3322-6284","authenticated-orcid":false,"given":"Dong","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0322-4566","authenticated-orcid":false,"given":"Jingcheng","family":"Du","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6962-3554","authenticated-orcid":false,"given":"Majid","family":"Rastegar-Mojarad","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-6710-6239","authenticated-orcid":false,"given":"Varun Kumar","family":"Nomula","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4193-4120","authenticated-orcid":false,"given":"Xiaoyan","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,10,15]]},"reference":[{"key":"R1","unstructured":"OpenAI. GPT-4 technical report. Mar 15, 2023. URL: https:\/\/cdn.openai.com\/papers\/gpt-4.pdf [Accessed 18-09-2025]"},{"key":"R2","unstructured":"OpenAI API documentation (GPT-3.5 series). OpenAI. Mar 15, 2022. URL: https:\/\/platform.openai.com\/docs\/api-reference\/introduction [Accessed 29-09-2025]"},{"key":"R3","unstructured":"Introducing the next generation of Claude. Anthropic. 2024. URL: https:\/\/docs.claude.com\/en\/home [Accessed 29-09-2025]"},{"key":"R4","unstructured":"Llama 2. Meta AI. URL: https:\/\/www.llama.com\/llama2\/ [Accessed 29-09-2025]"},{"issue":"6","key":"R5","doi-asserted-by":"publisher","first-page":"1105","DOI":"10.1007\/s11596-021-2474-3","article-title":"Application of artificial intelligence in medicine: An overview","volume":"41","author":"Liu","journal-title":"Curr Med Sci"},{"issue":"2","key":"R6","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198","article-title":"Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models","volume":"2","author":"Kung","journal-title":"PLOS Digit Health"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.wneu.2023.08.042","article-title":"GPT-4 artificial intelligence model outperforms ChatGPT, medical students, and neurosurgery residents on neurosurgery written board-like questions","volume":"179","author":"Guerra","journal-title":"World Neurosurg"},{"issue":"3","key":"R8","doi-asserted-by":"publisher","DOI":"10.2106\/JBJS.OA.23.00056","article-title":"Evaluating ChatGPT performance on the orthopaedic in-training examination","volume":"8","author":"Kung","journal-title":"JB JS Open Access"},{"issue":"5","key":"R9","doi-asserted-by":"publisher","first-page":"269","DOI":"10.4174\/astr.2023.104.5.269","article-title":"ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models","volume":"104","author":"Oh","journal-title":"Ann Surg Treat Res"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-43436-9","article-title":"Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments","volume":"13","author":"Brin","journal-title":"Sci Rep"},{"issue":"6","key":"R11","doi-asserted-by":"publisher","first-page":"e116","DOI":"10.1016\/j.jsurg.2020.06.033","article-title":"Sustained clinical performance during surgical rotations predicts NBME shelf exam outcomes","volume":"77","author":"Tracy","journal-title":"J Surg Educ"},{"issue":"11","key":"R12","doi-asserted-by":"publisher","first-page":"1238","DOI":"10.1097\/00001888-199911000-00019","article-title":"Early clinical experience enhances third-year pediatrics clerkship performance","volume":"74","author":"Kossoff","journal-title":"Acad Med"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/j.ebiom.2019.07.019","article-title":"Artificial intelligence to support clinical decision-making processes","volume":"46","author":"Garcia-Vidal","journal-title":"EBioMedicine"},{"issue":"8","key":"R14","doi-asserted-by":"publisher","first-page":"895","DOI":"10.1016\/j.amjmed.2020.03.033","article-title":"Artificial intelligence: the future for diabetes care","volume":"133","author":"Ellahham","journal-title":"Am J Med"},{"issue":"7972","key":"R15","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature New Biol"},{"key":"R16","unstructured":"World Health Organization. Vaccine hesitancy: a growing challenge for immunization programmes. 2015. URL: https:\/\/www.who.int\/news\/item\/18-08-2015-vaccine-hesitancy-a-growing-challenge-for-immunization-programmes [Accessed 29-9-2025]"},{"key":"R17","unstructured":"Naveed H Khan AU Qiu S Yao L Haq AU Ullah S et al. A comprehensive overview of large language models. Preprint posted online on 2023. URL: https:\/\/arxiv.org\/abs\/2307.06435 [Accessed 11-02-2024]"},{"issue":"8","key":"R18","first-page":"9","volume":"1","author":"Radford","journal-title":"OpenAI blog"},{"key":"R19","unstructured":"Radford A Narasimhan K Salimans T Sutskever I . Improving language understanding by generative pre-training. OpenAI; 2018. URL: https:\/\/cdn.openai.com\/research-covers\/language-unsupervised\/language_understanding_paper.pdf [Accessed 11-01-2024]"},{"issue":"140","key":"R20","first-page":"1","volume":"21","author":"Raffel","journal-title":"J Mach Learn Res"},{"key":"R21","unstructured":"Anthropic Blog. Claude: a next-generation AI assistant. 2023. URL: https:\/\/www.anthropic.com\/claude [Accessed 15-07-2024]"},{"key":"R22","unstructured":"Touvron H Lavril T Izacard G Martinet X Lachaux MA Lacroix T et al. Llama: open and efficient foundation language models. arXiv. Preprint posted online on 2023. URL: https:\/\/arxiv.org\/abs\/2302.13971 [Accessed 20-05-2024]"},{"issue":"4","key":"R23","doi-asserted-by":"publisher","DOI":"10.2196\/57164","article-title":"Natural language processing-powered real-time monitoring solution for vaccine sentiments and hesitancy on social media: system development and validation","volume":"12","author":"Huang","journal-title":"JMIR Med Inform"},{"key":"R24","unstructured":"World Health Organization. Report of the SAGE working group on vaccine hesitancy. 2014. URL: https:\/\/www.asset-scienceinsociety.eu\/sites\/default\/files\/sage_working_group_revised_report_vaccine_hesitancy.pdf [Accessed 29-9-2025]"},{"key":"R25","unstructured":"Alarcon N . NVIDIA Technical Blog. OpenAI presents GPT-3, a 175 billion parameters language model. Apr 7, 2020. URL: https:\/\/developer.nvidia.com\/blog\/openai-presents-gpt-3-a-175-billion-parameters-language-model\/ [Accessed 10-04-2024]"},{"key":"R26","unstructured":"Bastian M . The Decoder. GPT-3.5 might be a strong example of the efficiency potential of large AI models. Mar 31, 2023. URL: https:\/\/the-decoder.com\/gpt-3-5-might-be-a-strong-example-of-the-efficiency-potential-of-large-ai-models\/ [Accessed 10-03-2024]"},{"key":"R27","unstructured":"Amazon Web Services. Anthropic\u2019s Claude 3 Sonnet Model now on Amazon Bedrock. 2024. URL: https:\/\/aws.amazon.com\/about-aws\/whats-new\/2024\/03\/anthropics-claude-3-sonnet-model-amazon-bedrock\/ [Accessed 09-05-2024]"},{"key":"R28","unstructured":"Bergmann D . IBM What is zero-shot learning?. 2024. URL: https:\/\/www.ibm.com\/think\/topics\/zero-shot-learning [Accessed 29-9-2025]"},{"key":"R29","unstructured":"Hugging Face. What is zero-shot classification? Nov 16, 2022. URL: https:\/\/huggingface.co\/tasks\/zero-shot-classification [Accessed 05-02-2024]"},{"key":"R30","volume":"33","author":"Brown","journal-title":"Adv Neural Inf Process Syst"},{"issue":"9","key":"R31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3560815","article-title":"Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing","volume":"55","author":"Liu","journal-title":"ACM Comput Surv"},{"issue":"1","key":"R32","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-45914-8","article-title":"Extracting accurate materials data from research papers with conversational language models and prompt engineering","volume":"15","author":"Polak","journal-title":"Nat Commun"},{"key":"R33","doi-asserted-by":"crossref","unstructured":"Gao T Fisch A Chen D . Making pre-trained language models better few-shot learners. Presented at: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1); 3816-3830; Online. 2021.[doi: 10.18653\/v1\/2021.acl-long.295]","DOI":"10.18653\/v1\/2021.acl-long.295"},{"key":"R34","unstructured":"Walker SM . Klu F-score: what are accuracy, precision, recall, and F1 score?. Jul 4, 2023. URL: https:\/\/klu.ai\/glossary\/accuracy-precision-recall-f1 [Accessed 15-03-2024]"},{"key":"R35","unstructured":"Roberts M Russo R . A Student\u2019s Guide to Analysis of Variance. 1st ed. Routledge; 1999."},{"key":"R36","unstructured":"OpenAI. Pricing. URL: https:\/\/openai.com\/api\/pricing\/ [Accessed 25-05-2024]"},{"key":"R37","unstructured":"Anthropic. Claude API. URL: https:\/\/docs.claude.com\/en\/home [Accessed 29-9-2025]"},{"key":"R38","unstructured":"Microsoft. Introducing Llama 2 on Azure. URL: https:\/\/techcommunity.microsoft.com\/blog\/azure-ai-foundry-blog\/introducing-llama-2-on-azure\/3881233 [Accessed 22-02-2024]"},{"key":"R39","unstructured":"Amazon Web Services. Fine-tune and deploy Llama 2 models cost-effectively in Amazon SageMaker JumpStart with AWS Inferentia and AWS Trainium. 2024. URL: https:\/\/aws.amazon.com\/blogs\/machine-learning\/fine-tune-and-deploy-llama-2-models-cost-effectively-in-amazon-sagemaker-jumpstart-with-aws-inferentia-and-aws-trainium\/ [Accessed 21-05-2024]"},{"key":"R40","unstructured":"Baktash JA Dawodi M . Gpt-4: a review on advancements and opportunities in natural language processing. arXiv. Preprint posted online on 2023. URL: https:\/\/arxiv.org\/abs\/2305.03195 [Accessed 13-05-2024]"},{"key":"R41","unstructured":"Fatemi S Hu Y . A comparative analysis of fine-tuned LLMS and few-shot learning of LLMS for financial sentiment analysis. arXiv. Preprint posted online on 2023. URL: https:\/\/arxiv.org\/abs\/2312.08725 [Accessed 14-05-2024]"},{"key":"R42","doi-asserted-by":"crossref","unstructured":"Liu YH Deng Y Liu B Pan SJ Bing L . Understanding LLMS: a comprehensive overview from training to inference. Preprint posted online on 2024. URL: https:\/\/arxiv.org\/abs\/2401.02038","DOI":"10.2139\/ssrn.4706201"},{"key":"R43","doi-asserted-by":"crossref","unstructured":"Zhang W Deng Y Liu B Pan SJ Bing L . Sentiment analysis in the era of large language models: a reality check. arXiv. Preprint posted online on 2023. [doi: 10.48550\/arXiv.2305.15005]","DOI":"10.18653\/v1\/2024.findings-naacl.246"},{"key":"R44","doi-asserted-by":"crossref","unstructured":"Ghosh A Veale T . Fracking sarcasm using neural network. Presented at: Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis; Jun 16-17, 2016:161-169; San Diego, CA. 2016.[doi: 10.18653\/v1\/W16-0425]","DOI":"10.18653\/v1\/W16-0425"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/formative.jmir.org\/2025\/1\/e64723","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:30:14Z","timestamp":1785537014000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2025\/1\/e64723"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,15]]},"references-count":44,"URL":"https:\/\/doi.org\/10.2196\/64723","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,15]]},"article-number":"v9i6e64723"}}