{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T21:58:49Z","timestamp":1791583129183,"version":"4.3.4"},"reference-count":38,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Symptom detection is essential in global disease surveillance to detect potential outbreaks, as symptoms are the first observable signs of infection. To reflect real-time ground truth conditions during pandemics, social media has emerged as a valuable data source. Moreover, effective digital disease surveillance systems must operate across diverse linguistic settings, and large language models (LLMs) have been shown to perform inconsistently across languages, tending to have lower performance in low-resource languages. While multilingual approaches have been explored in various health-related natural language processing tasks, a critical gap remains in understanding whether LLM-based symptom detection can perform consistently across languages for global disease surveillance. Southeast Asia demonstrates this challenge, combining diverse languages and the potential for emerging infectious disease outbreaks, making it a case for evaluating how multilingual performance disparities manifest in symptom detection.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to evaluate multilingual disparities in symptom detection using a LLM, as well as the associated error mechanisms across languages and symptom types, to better understand their implications for global disease surveillance.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n This study uses the MedWeb dataset, a multilingual pseudo\u2013social media text dataset with multiple symptom labels. The data consist of 12 languages, covering diverse regions and language resource classifications. The symptoms included in this study are fever, headache, runny nose, cough, diarrhea, hay fever, influenza, and cold. We used GPT-5 as the symptom detection system, representing a strong model for health-related tasks. The results are evaluated using the macro precision, recall, and\n F<\/jats:italic>\n 1<\/jats:sub>\n -score. An error analysis was conducted to identify the error mechanisms underlying incorrect symptom predictions across languages and to calculate the impact of each error category.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Our findings show that performance varied across languages, language resource groups, and symptom types. Southeast Asian (SEA) languages generally achieved lower scores than non-SEA languages, with Japanese obtaining the highest\n F<\/jats:italic>\n 1<\/jats:sub>\n -score (0.812) and Lao the lowest (0.716). High-resource languages achieved the most consistent performance, while low-resource languages obtained the lowest overall scores. At the symptom level, diarrhea, headache, cough, and fever showed stable detection across languages, while hay fever, runny nose, influenza, and cold exhibited greater variability. Hay fever showed the widest variability, forming 2 distinct performance clusters aligned with language resource and region classifications. The error analysis revealed 4 misclassification patterns: explicitly mentioned symptoms, cross-lingual variation, symptom overgeneralization, and context misinterpretation. Cross-lingual variation was the most frequent error category, while errors related to explicitly mentioned symptoms showed the largest potential impact on model performance.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Due to performance disparities, achieving more reliable and equitable LLM-based symptom detection from social media text for global disease surveillance would benefit from broader representation in training data for low-resource languages, improved cultural and linguistic sensitivity, and stronger contextual understanding of symptom-related expressions.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/103321","type":"journal-article","created":{"date-parts":[[2026,8,31]],"date-time":"2026-08-31T18:55:05Z","timestamp":1788202505000},"page":"e103321-e103321","source":"Crossref","is-referenced-by-count":0,"title":["Multilingual Disparities in Large Language Model\u2013Based Symptom Detection for Global Disease Surveillance: Evaluation Study"],"prefix":"10.2196","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7420-4530","authenticated-orcid":false,"given":"Sa'idah Zahrotul","family":"Jannah","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1538-8266","authenticated-orcid":false,"given":"Tomohiro","family":"Nishiyama","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4020-9100","authenticated-orcid":false,"given":"Shaowen","family":"Peng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9371-1340","authenticated-orcid":false,"given":"Shoko","family":"Wakamiya","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0201-3609","authenticated-orcid":false,"given":"Eiji","family":"Aramaki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,9]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","first-page":"100071","DOI":"10.1016\/j.medp.2025.100071","article-title":"Progress and challenges in infectious disease surveillance and early warning","volume":"2","author":"Shen","journal-title":"Med Plus"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","first-page":"1075","DOI":"10.1057\/s41599-024-03589-y","article-title":"Leveraging social media data for pandemic detection and prediction","volume":"11","author":"Shi","journal-title":"Humanit Soc Sci Commun"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-publhealth-040119-094402","article-title":"Social media\u2013and internet-based disease surveillance for public health","volume":"41","author":"Aiello","journal-title":"Annu Rev Public Health"},{"issue":"2","key":"R4","doi-asserted-by":"publisher","first-page":"1167","DOI":"10.32604\/cmes.2023.043921","article-title":"Social media-based surveillance systems for health informatics using machine and deep learning techniques: a comprehensive review and open challenges","volume":"139","author":"Amin","journal-title":"Comput Model Eng Sci"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","DOI":"10.1017\/ash.2021.225","article-title":"Social media: a new tool for outbreak surveillance","volume":"1","author":"Wilson","journal-title":"Antimicrob Steward Healthc Epidemiol"},{"key":"R6","doi-asserted-by":"crossref","unstructured":"Joshi P Santy S Budhiraja A Bali K Choudhury M . The state and fate of linguistic diversity and inclusion in the NLP world. In: Jurafsky D Chai J Schluter N Tetreault J , editors. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics; 2020:6282-6293. [doi: 10.18653\/v1\/2020.acl-main.560]","DOI":"10.18653\/v1\/2020.acl-main.560"},{"key":"R7","unstructured":"Pava JN Meinhardt C Uz Zaman HB et al. Mind the (language) gap: mapping the challenges of LLM development in low-resource language contexts. Stanford Institute for Human-Centered Artificial Intelligence (HAI), Stanford University; 2025. URL: https:\/\/hai.stanford.edu\/assets\/files\/hai-taf-pretoria-white-paper-mind-the-language-gap.pdf [Accessed 12-09-2026]"},{"key":"R8","doi-asserted-by":"crossref","unstructured":"Zhang X Li S Hauer B Shi N Kondrak G . Don\u2019t trust ChatGPT when your question is not in English: a study of multilingual abilities and types of LLMs. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2023:7915-7927. [doi: 10.18653\/v1\/2023.emnlp-main.491]","DOI":"10.18653\/v1\/2023.emnlp-main.491"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.2196\/81399","article-title":"Performance evaluation of large language models in multilingual medical multiple-choice questions: mixed methods study","volume":"12","author":"Strasser","journal-title":"JMIR Med Educ"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/49139","article-title":"Use of large language models to assess the likelihood of epidemics from the content of tweets: infodemiology study","volume":"26","author":"Deiner","journal-title":"J Med Internet Res"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1093\/database\/baae087","article-title":"Recognition and normalization of multilingual symptom entities using in-domain-adapted BERT models and classification layers","volume":"2024","author":"Gallego","journal-title":"Database (Oxford)"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI250880","article-title":"Challenges in multilingual adverse drug reaction detection on social media: insights from case studies","volume":"329","author":"Yeh","journal-title":"Stud Health Technol Inform"},{"key":"R13","doi-asserted-by":"crossref","unstructured":"Jannah SZ Aco E Peng S Wakamiya S Aramaki E . Multilingual symptom detection on social media: enhancing health-related fact-checking with LLMs. In: Proceedings of the Eighth Fact Extraction and VERification Workshop (FEVER). Association for Computational Linguistics; 2025:54-68. [doi: 10.18653\/v1\/2025.fever-1.4]","DOI":"10.18653\/v1\/2025.fever-1.4"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Susanto Y Hulagadri AV Montalan JR et al. SEA-HELM: Southeast Asian holistic evaluation of language models. In: Che W Nabende J Shutova E Pilehvar MT , editors. Findings of the Association for Computational Linguistics. Association for Computational Linguistics; 2025:12308-12336. [doi: 10.18653\/v1\/2025.findings-acl.636]","DOI":"10.18653\/v1\/2025.findings-acl.636"},{"key":"R15","doi-asserted-by":"crossref","unstructured":"Liu C Zhang W Ying J Aljunied M Luu AT Bing L . SeaExam and seabench: benchmarking llms with local multilingual questions in Southeast Asia. In: Chiruzzo L Ritter A Wang L , editors. Findings of the Association for Computational Linguistics: NAACL 2025. Association for Computational Linguistics; 2025:6134-6151. [doi: 10.18653\/v1\/2025.findings-naacl.341]","DOI":"10.18653\/v1\/2025.findings-naacl.341"},{"key":"R16","doi-asserted-by":"publisher","DOI":"10.1016\/j.ecoenv.2025.117751","article-title":"Predicting dengue incidence in high-risk areas of China through the integration of Southeast Asian and local meteorological factors","volume":"290","author":"Sang","journal-title":"Ecotoxicol Environ Saf"},{"issue":"9765","key":"R17","doi-asserted-by":"publisher","first-page":"599","DOI":"10.1016\/S0140-6736(10)62004-1","article-title":"Emerging infectious diseases in Southeast Asia: regional challenges to control","volume":"377","author":"Coker","journal-title":"The Lancet"},{"issue":"9","key":"R18","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare9091220","article-title":"Early spatiotemporal patterns and population characteristics of the COVID-19 pandemic in Southeast Asia","volume":"9","author":"Zhu","journal-title":"Health Care (Don Mills)"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2023.1141483","article-title":"Emerging and re-emerging zoonotic viral diseases in Southeast Asia: One Health challenge","volume":"11","author":"Saba Villarroel","journal-title":"Front Public Health"},{"issue":"2","key":"R20","doi-asserted-by":"publisher","DOI":"10.2196\/12783","article-title":"Tweet classification toward Twitter-based disease surveillance: new data, methods, and evaluations","volume":"21","author":"Wakamiya","journal-title":"J Med Internet Res"},{"key":"R21","unstructured":"Introducing GPT-5. OpenAI. 2026. URL: https:\/\/openai.com\/index\/introducing-gpt-5\/ [Accessed 22-04-2026]"},{"issue":"4","key":"R22","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1016\/j.ipm.2009.03.002","article-title":"A systematic analysis of performance measures for classification tasks","volume":"45","author":"Sokolova","journal-title":"Inf Process Manag"},{"key":"R23","unstructured":"Ethical Guidelines for Medical and Biological Research Involving Human Subjects. Ministry of Education, Culture, Sports, Science and Technology (MEXT); Ministry of Health, Labour and Welfare (MHLW); Ministry of Economy, Trade and Industry (METI); 2021. URL: https:\/\/www.mext.go.jp\/content\/20250325-mxt_life-000035486-01.pdf [Accessed 12-09-2026]"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.2196\/69857","article-title":"Performance of open-source large language models in psychiatry: usability study through comparative analysis of non-English records and English translations","volume":"27","author":"Kim","journal-title":"J Med Internet Res"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-025-00802-0","article-title":"Large language model comparisons between English and Chinese query performance for cardiovascular prevention","volume":"5","author":"Ji","journal-title":"Commun Med"},{"issue":"1","key":"R26","doi-asserted-by":"publisher","DOI":"10.1186\/s12967-024-05128-4","article-title":"Language and cultural bias in AI: comparing the performance of large language models developed in different countries on Traditional Chinese Medicine highlights the need for localized models","volume":"22","author":"Zhu","journal-title":"J Transl Med"},{"key":"R27","doi-asserted-by":"crossref","unstructured":"Bagheri Nezhad S Agrawal A . What drives performance in multilingual language models? In: Scherrer Y Jauhiainen T Ljube\u0161i\u0107 N Zampieri M Nakov P Tiedemann J , editors. Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024). Association for Computational Linguistics; 2024:16-27. [doi: 10.18653\/v1\/2024.vardial-1.2]","DOI":"10.18653\/v1\/2024.vardial-1.2"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2024.101118","article-title":"A survey of multilingual large language models","volume":"6","author":"Qin","journal-title":"Patterns"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.3389\/falgy.2023.1224988","article-title":"Common cold","volume":"4","author":"Eccles","journal-title":"Front Allergy"},{"issue":"11","key":"R30","doi-asserted-by":"publisher","first-page":"718","DOI":"10.1016\/S1473-3099(05)70270-X","article-title":"Understanding the symptoms of the common cold and influenza","volume":"5","author":"Eccles","journal-title":"Lancet Infect Dis"},{"issue":"7","key":"R31","doi-asserted-by":"publisher","DOI":"10.1016\/j.waojou.2023.100799","article-title":"Pollen respiratory allergy: is it really seasonal?","volume":"16","author":"D\u2019Amato","journal-title":"World Allergy Organ J"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.3389\/fpubh.2022.1067034","article-title":"Exploring the effects of seasons, diurnal cycle, and heights on airborne pollen load in a Southeast Asian atmospheric condition","volume":"10","author":"Juprasong","journal-title":"Front Public Health"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1016\/j.iac.2020.09.007","article-title":"Climate change and pollen allergy in India and South Asia","volume":"41","author":"Singh","journal-title":"Immunol Allergy Clin North Am"},{"issue":"5","key":"R34","doi-asserted-by":"publisher","DOI":"10.1016\/j.waojou.2025.101053","article-title":"Monitoring, distribution and clinical relevance of airborne pollen and fern spores in Southeast Asia - a systematic review","volume":"18","author":"Pham","journal-title":"World Allergy Organ J"},{"issue":"3","key":"R35","doi-asserted-by":"publisher","DOI":"10.1002\/clt2.12130","article-title":"Worldwide prevalence of rhinitis in adults: a review of definitions and temporal evolution","volume":"12","author":"Savour\u00e9","journal-title":"Clin Transl Allergy"},{"issue":"2","key":"R36","doi-asserted-by":"publisher","first-page":"186","DOI":"10.1111\/j.1365-2222.2011.03891.x","article-title":"Prevalence and diversity of allergic rhinitis in regions of the world beyond Europe and North America","volume":"42","author":"Katelaris","journal-title":"Clin Exp Allergy"},{"issue":"3","key":"R37","doi-asserted-by":"publisher","DOI":"10.1080\/08989621.2025.2544331","article-title":"GAIDeT (Generative AI Delegation Taxonomy): a taxonomy for humans to delegate tasks to generative artificial intelligence in scientific research and publishing","volume":"33","author":"Suchikova","journal-title":"Account Res"},{"key":"R38","unstructured":"NTCIR-13 MedWeb [Article in Japanese]. NTCIR Project. URL: https:\/\/research.nii.ac.jp\/ntcir\/permission\/ntcir-13\/perm-ja-MedWeb.html [Accessed 01-06-2026]"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T21:43:07Z","timestamp":1791582187000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e103321"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,9]]},"references-count":38,"URL":"https:\/\/doi.org\/10.2196\/103321","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,9]]},"article-number":"v5i1e103321"}}