{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T14:17:17Z","timestamp":1783520237186,"version":"3.55.0"},"reference-count":20,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"\n Background<\/jats:title>\n Causality assessment is central to pharmacovigilance but remains resource-intensive and subjective. The applicability of large language models (LLMs) to formal World Health Organization\u2013Uppsala Monitoring Centre (WHO-UMC) drug\u2013adverse event causality assessment has not been well established.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to evaluate the performance of LLMs in WHO-UMC causality assessment.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n A curated set of 55 cases derived from the US Food and Drug Administration Adverse Event Reporting System, comprising 337 drug-level assessments, was constructed. Cases involving 2 to 11 suspected drugs were stratified by drug count, and 5 cases were sampled from each stratum. To ensure representation of rare but clinically important categories, 5 additional cases containing at least 1 \u201cCertain\u201d drug\u2013adverse event pair were included. Case data were reorganized into a standardized semistructured format that preserved key elements required for WHO-UMC causality assessment. Domain experts conducted a pilot evaluation to align interpretation criteria prior to independently assessing the final dataset, yielding an interexpert agreement (Fleiss \u03ba) of 0.762 across 337 drug-level assessments. Multiple prompting strategies, including standard prompting, chain-of-thought (CoT), CoT with self-consistency, few-shot, reasoning and acting, and tree-of-thought prompting, were applied across multiple LLMs, including GPT-5.4 and its mini variant and Gemini 2.5 Flash and Pro, via their respective application programming interfaces. Agreement with expert assessments was quantified using Cohen \u03ba, weighted \u03ba, and accuracy metrics. Internal consistency across repeated inferences was evaluated using Fleiss \u03ba.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Performance varied across models and prompting strategies. Cohen \u03ba ranged from 0.368 to 0.641, weighted \u03ba ranged from 0.641 to 0.821, accuracy ranged from 0.583 to 0.804, and balanced accuracy ranged from 0.513 to 0.735. Fleiss \u03ba ranged from 0.730 to 0.915, corresponding to substantial to almost perfect agreement. The highest Cohen \u03ba was observed for Gemini 2.5 Flash with CoT prompting (0.641). Gemini 2.5 Flash with CoT\u2013self-consistency prompting showed a Cohen \u03ba of 0.640 and achieved the highest observed point estimates for weighted \u03ba (0.821), accuracy (0.804), and Fleiss \u03ba (0.915), although the gains over other prompting strategies were modest. Category-level performance for this model showed higher performance for \u201cCertain\u201d (F1-score=0.793), \u201cProbable\/Likely\u201d (F1-score=0.794), and \u201cUnlikely\u201d (F1-score=0.898), whereas performance for \u201cPossible\u201d remained substantially lower (F1-score=0.293), reflecting the difficulty of intermediate causality assessment.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n LLMs demonstrated moderate to substantial agreement in WHO-UMC causality assessment, indicating meaningful but still limited performance relative to expert judgment. Although LLMs are not suitable for independent decision-making, they may serve as supportive tools in pharmacovigilance workflows, particularly for preliminary case triage. Further studies using larger and more diverse datasets and evaluating performance on raw narrative reports are warranted.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/93237","type":"journal-article","created":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T13:25:07Z","timestamp":1783517107000},"page":"e93237","source":"Crossref","is-referenced-by-count":0,"title":["Large Language Models for World Health Organization\u2013Uppsala Monitoring Centre Drug\u2013Adverse Event Causality Assessment Using Food and Drug Administration Adverse Event Reporting System Cases: Comparative Performance Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-1772-331X","authenticated-orcid":false,"given":"Young Mi","family":"Ha","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8718-8469","authenticated-orcid":false,"given":"Minjung","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2850-0432","authenticated-orcid":false,"given":"YoungIn","family":"Bang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5070-360X","authenticated-orcid":false,"given":"Daejin","family":"Choi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8609-7135","authenticated-orcid":false,"given":"Jae Hyun","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2554-3370","authenticated-orcid":false,"given":"Sandy Jeong","family":"Rhie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9110-9604","authenticated-orcid":false,"given":"Yoshihiro","family":"Noguchi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5593-7672","authenticated-orcid":false,"given":"Myeong Gyu","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,7,8]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.7150\/ijms.6048"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1007\/s00228-008-0475-9"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.4103\/jfmpc.jfmpc_1148_24"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.59975"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(00)02799-9"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1139\/cjpp-2024-0268"},{"key":"ref7","unstructured":"The use of the WHO-UMC system for standardised case causality assessmentWorld Health Organization2013652026-01-28https:\/\/www.who.int\/publications\/m\/item\/WHO-causality-assessment"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1007\/s40264-013-0123-x"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0000198"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.1177\/20420986251405082"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1177\/20420986251339358"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI250495"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.2427\/12267"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2412.07923"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1037\/h0026256"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.1037\/h0031619"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.60638"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.1007\/s40264-025-01531-y"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e93237","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T13:25:08Z","timestamp":1783517108000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e93237"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,8]]},"references-count":20,"URL":"https:\/\/doi.org\/10.2196\/93237","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.93237","asserted-by":"object"}]},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,8]]}}}