{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:43:01Z","timestamp":1785544981851,"version":"3.56.0"},"reference-count":31,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Administrative costs in the health care system are driven in part by complex insurance eligibility determinations. Large language models (LLMs) are increasingly used for health insurance\u2013related queries, yet their reliability for structured logical reasoning over coverage criteria has not been systematically evaluated.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to develop a benchmark for anticancer drug reimbursement eligibility determination and evaluate whether LLMs can reliably perform eligibility verification.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We constructed a benchmark based on South Korea\u2019s National Health Insurance reimbursement guidelines for 3 gynecologic cancers (cervical, uterine, and ovarian), using a tristate adjudication framework (eligible, ineligible, and undeterminable). Three gynecologic oncology experts and a utilization review nurse validated the benchmark. Six LLMs from 3 providers (Anthropic, Google, and OpenAI) were evaluated using the official guideline document as input. Each case was evaluated 3 times per model, with final predictions determined by majority vote, and performance was compared across the 3 outcome classes.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n The benchmark comprises 74 anticancer regimens with 222 cases. Overall verification accuracy ranged from 77.9% to 88.7% across the 6 models. Eligible and ineligible cases were classified with high recall (86.5%\u201098.6%), but undeterminable cases showed a marked decline across all models (44.6%\u201070.3%). Performance varied by cancer type, with uterine cancer showing the lowest undeterminable recall (16.7%), corresponding to the highest guideline complexity. Undeterminable cases were predominantly misclassified as eligible rather than ineligible. The tristate framework enabled logic-based error analysis of 235 incorrect predictions, revealing information gap-filling as the dominant failure pattern (n=196, 83.4%), followed by criterion misapplication (n=20, 8.5%) and false uncertainty (n=19, 8.1%). Subtype analysis indicated that information gap-filling errors were concentrated at hierarchical elements of the guideline. Sensitivity analyses showed that converting the guideline document to structured text degraded performance, while web search\u2013enabled condition (0%\u20103.2% tool invocation across models) and structure-guided prompting did not produce significant changes from baseline.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n In this benchmark, LLMs classified clearly eligible and ineligible cases with relatively high recall but showed limited reliability on undeterminable cases. The dominant error pattern was information gap-filling, in which models inferred eligibility rather than withholding judgment. These findings indicate that LLMs, in their current form, should be deployed as supervised decision-support tools rather than as independent adjudicators in reimbursement review.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/95877","type":"journal-article","created":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T19:21:52Z","timestamp":1781551312000},"page":"e95877-e95877","source":"Crossref","is-referenced-by-count":0,"title":["Assessing Eligibility for Anticancer Drug Health Insurance Reimbursement Using Large Language Models: Benchmark Development and Comparative Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-8289-4432","authenticated-orcid":false,"given":"Junhyuk","family":"Seo","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8340-9511","authenticated-orcid":false,"given":"Taerim","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4007-5739","authenticated-orcid":false,"given":"Ju-Hyun","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,6,15]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-014-0556-7","article-title":"Billing and insurance-related administrative costs in United States\u2019 health care: synthesis of micro-costing evidence","volume":"14","author":"Jiwani","journal-title":"BMC Health Serv Res"},{"issue":"7","key":"R2","doi-asserted-by":"publisher","first-page":"691","DOI":"10.1001\/jama.2017.19148","article-title":"Administrative costs associated with physician billing and insurance-related activities at an academic health care system","volume":"319","author":"Tseng","journal-title":"JAMA"},{"issue":"9","key":"R3","doi-asserted-by":"publisher","first-page":"659","DOI":"10.7326\/M16-2697","article-title":"Putting patients first by reducing administrative tasks in health care: a position paper of the American College of Physicians","volume":"166","author":"Erickson","journal-title":"Ann Intern Med"},{"issue":"11","key":"R4","doi-asserted-by":"publisher","DOI":"10.1093\/haschl\/qxaf216","article-title":"Patient administrative burden: a scoping review","volume":"3","author":"Kyle","journal-title":"Health Aff Sch"},{"issue":"8","key":"R5","doi-asserted-by":"publisher","first-page":"1098","DOI":"10.1377\/hlthaff.2022.00241","article-title":"Billing and insurance-related administrative costs: a cross-national analysis","volume":"41","author":"Richman","journal-title":"Health Aff (Millwood)"},{"issue":"1","key":"R6","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-016-1746-2","article-title":"Effects of public and private health insurance on medical service utilization in the National Health Insurance System: national panel study in the Republic of Korea","volume":"16","author":"Sohn","journal-title":"BMC Health Serv Res"},{"issue":"5","key":"R7","doi-asserted-by":"publisher","first-page":"718","DOI":"10.3346\/jkms.2017.32.5.718","article-title":"Towards actualizing the value potential of Korea Health Insurance Review and Assessment (HIRA) data as a resource for health research: strengths, limitations, applications, and strategies for optimal use of HIRA data","volume":"32","author":"Kim","journal-title":"J Korean Med Sci"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.4178\/epih\/e2014008","article-title":"A guide for the utilization of Health Insurance Review and Assessment Service National Patient Samples","volume":"36","author":"Kim","journal-title":"Epidemiol Health"},{"issue":"3","key":"R9","doi-asserted-by":"publisher","first-page":"184","DOI":"10.4258\/hir.2015.21.3.184","article-title":"Healthcare utilization monitoring system in Korea","volume":"21","author":"Shin","journal-title":"Healthc Inform Res"},{"issue":"5","key":"R10","doi-asserted-by":"publisher","DOI":"10.3346\/jkms.2025.40.e16","article-title":"National expenditures on anticancer and immunomodulating agents during 2013-2022 in Korea","volume":"40","author":"Yun","journal-title":"J Korean Med Sci"},{"key":"R11","unstructured":"Cancer drug and regimen guidelines: public announcements [Report in Korean]. Health Insurance Review & Assessment Service; 2026. URL: https:\/\/www.hira.or.kr\/bbsDummy.do?pgmid=HIRAA030023010000 [Accessed 02-06-2026]"},{"key":"R12","unstructured":"AI as a healthcare ally: how Americans are navigating the system with ChatGPT. OpenAI; 2026. URL: https:\/\/cdn.openai.com\/pdf\/2cb29276-68cd-4ec6-a5f4-c01c5e7a36e9\/OpenAI-AI-as-a-Healthcare-Ally-Jan-2026.pdf [Accessed 02-06-2026]"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.2196\/58329","article-title":"Evaluation framework of large language models in medical documentation: development and usability study","volume":"26","author":"Seo","journal-title":"J Med Internet Res"},{"issue":"10","key":"R14","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2025.38427","article-title":"Large language model assistant for emergency department discharge documentation","volume":"8","author":"Song","journal-title":"JAMA Netw Open"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/81915","article-title":"Improving the understandability of clinical guidelines: development and evaluation of a GPT-4-based pipeline","volume":"28","author":"Jones","journal-title":"J Med Internet Res"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1053\/j.gastro.2024.09.014","article-title":"Detection of gastrointestinal bleeding with large language models to aid quality improvement and appropriate reimbursement","volume":"168","author":"Zheng","journal-title":"Gastroenterology"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.1038\/s44401-025-00018-3","article-title":"Enhancing medical coding efficiency through domain-specific fine-tuned large language models","volume":"2","author":"Hou","journal-title":"Npj Health Syst"},{"issue":"5","key":"R18","doi-asserted-by":"publisher","first-page":"565","DOI":"10.1001\/jama.293.5.565","article-title":"Missing clinical information during primary care visits","volume":"293","author":"Smith","journal-title":"JAMA"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6963-11-114","article-title":"Missing clinical information in NHS hospital outpatient clinics: prevalence, causes and effects on patient care","volume":"11","author":"Burnett","journal-title":"BMC Health Serv Res"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","article-title":"The TRIPOD-LLM reporting guideline for studies using large language models","volume":"31","author":"Gallifant","journal-title":"Nat Med"},{"issue":"14","key":"R21","doi-asserted-by":"publisher","first-page":"6421","DOI":"10.3390\/app11146421","article-title":"What disease does this patient have? A large-scale open domain question answering dataset from medical exams","volume":"11","author":"Jin","journal-title":"Appl Sci"},{"issue":"4","key":"R22","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","journal-title":"JAMA"},{"issue":"3","key":"R23","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-025-04151-2","article-title":"Holistic evaluation of large language models for medical tasks with MedHELM","volume":"32","author":"Bedi","journal-title":"Nat Med"},{"key":"R24","unstructured":"Li X Gao M Hao Y et al. MedGUIDE: benchmarking clinical decision-making in large language models. Presented at: 39th Conference on Neural Information Processing Systems (NeurIPS 2025); Dec 2-7, 2025; San Diego, CA, USA. URL: https:\/\/openreview.net\/pdf\/07c2a85f2355803fe92312f6a9808c0e5065ddef.pdf [Accessed 02-06-2026]"},{"issue":"4","key":"R25","doi-asserted-by":"publisher","first-page":"855","DOI":"10.1093\/jamia\/ocag026","article-title":"CPGPrompt: translating clinical guidelines into large language model-executable decision support","volume":"33","author":"Deng","journal-title":"J Am Med Inform Assoc"},{"key":"R26","doi-asserted-by":"crossref","unstructured":"Zhou R Hua W Pan L et al. RuleArena: a benchmark for rule-guided reasoning with LLMs in real-world scenarios. Presented at: The 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers); Jul 27 to Aug 1, 2025; Vienna, Austria. [doi: 10.18653\/v1\/2025.acl-long.27]","DOI":"10.18653\/v1\/2025.acl-long.27"},{"key":"R27","unstructured":"Kirichenko P Ibrahim M Chaudhuri K Bell SJ . AbstentionBench: reasoning llms fail on unanswerable questions. Presented at: 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Track on Datasets and Benchmarks; Dec 2-7, 2025; San Diego Convention Center, San Diego, CA, USA. URL: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2025\/hash\/fb122bfc3f0127a94ded048b5b03496f-Abstract-Datasets_and_Benchmarks_Track.html [Accessed 02-06-2026]"},{"key":"R28","doi-asserted-by":"crossref","unstructured":"Machcha S Yerra S Gupta S et al. Knowing when to abstain: medical LLMs under clinical uncertainty. Presented at: 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers); Mar 24-29, 2026; Rabat, Morocco. [doi: 10.18653\/v1\/2026.eacl-long.291]","DOI":"10.18653\/v1\/2026.eacl-long.291"},{"issue":"1","key":"R29","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-025-01194-x","article-title":"Neuro-symbolic AI for auditable cognitive information extraction from medical reports","volume":"5","author":"Prenosil","journal-title":"Commun Med (Lond)"},{"key":"R30","unstructured":"Seo-see\/k-NHIB. GitHub. URL: https:\/\/github.com\/seo-see\/K-NHIB [Accessed 07-06-2026]"},{"key":"R31","unstructured":"K-NHIB: Korean National Health Insurance Benchmark. Zenodo. URL: https:\/\/doi.org\/10.5281\/zenodo.20563360 [Accessed 07-06-2026]"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e95877","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:23:17Z","timestamp":1785536597000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e95877"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,15]]},"references-count":31,"URL":"https:\/\/doi.org\/10.2196\/95877","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,15]]},"article-number":"v28i7e95877"}}