{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:00:15Z","timestamp":1782846015763,"version":"3.54.5"},"reference-count":34,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n This study develops an open-source large language model\u2013based chatbot tailored for Korean health consultations. The chatbot was implemented using the retrieval-augmented generation (RAG) technique alongside metadata filtering to enhance its performance.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to analyze and compare the performance of a RAG-based chatbot with other leading language models in the context of Korean health consultations.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n A 10.4 GB Korean medical document corpus (487,277 segments) was constructed from official websites of major Korean hospitals, public health sources, and medical textbooks. This study quantitatively compared 5 open-source large language models (Qwen3:4B, Mistral:7B, Llama-3.1:8B, Gpt-Oss:20B, and Gemma3:27B) in 3 configurations: baseline (model only), RAG-only, and RAG with metadata filtering. The RAG system used a specialized Korean embedding model (upskyy\/bge-m3-korean) and an Elasticsearch store. Performance was assessed by an emergency medicine specialist using a validation set of 226 questions across 7 common diseases and scoring responses based on accuracy, safety, and helpfulness.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The application of RAG alone failed to yield statistically significant performance improvements and, in some cases (Llama 3.1: 8B and Gemma 3: 27B), resulted in decreased scores. However, the combination of RAG with metadata filtering yielded statistically significant (\n P<\/jats:italic>\n <.05) performance increases in most models. Notably, the average score for Mistral:7B increased from 3.79, SD 0.08, to 4.10, SD 0.10, and Gpt-Oss:20B increased from 4.43, SD 0.05, to 4.51, SD 0.04, with the latter achieving the highest safety score (4.61, SD 0.03). The Gemma3:27B model, which possessed a high baseline performance (4.42, SD 0.03), was an exception, exhibiting no significant improvement (\n P<\/jats:italic>\n =.14) even with filtering.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The effectiveness of RAG for specialized domains such as Korean medical consultation is highly dependent on a metadata filtering process that controls the quality of retrieved information; simple information augmentation is insufficient. Furthermore, the benefit of RAG is limited when a model\u2019s intrinsic knowledge (eg, Gemma3:27B) already meets or exceeds the quality of the external knowledge base. This finding indicates that performance enhancement strategies must account for both the retrieval mechanism\u2019s quality and the model\u2019s preexisting capabilities.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/72604","type":"journal-article","created":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T15:45:06Z","timestamp":1775144706000},"page":"e72604-e72604","source":"Crossref","is-referenced-by-count":0,"title":["Korean Medical Consultation With Open-Weight Large Language Models: Pilot Comparative Evaluation of Retrieval-Augmented Generation With Metadata Filtering"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-9730-6298","authenticated-orcid":false,"given":"Saeyoun","family":"Choi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0841-3830","authenticated-orcid":false,"given":"Donghyun","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-9165-8175","authenticated-orcid":false,"given":"Ji-Hwan","family":"Jeon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-8492-4084","authenticated-orcid":false,"given":"Minji","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-2234-9089","authenticated-orcid":false,"given":"Dong Hun","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-1409-3361","authenticated-orcid":false,"given":"DaeHwan","family":"Ahn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6424-9946","authenticated-orcid":false,"given":"Eu Sun","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3767-3889","authenticated-orcid":false,"given":"Yoon Ji","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4631-1504","authenticated-orcid":false,"given":"Hyun","family":"Youk","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,4,30]]},"reference":[{"issue":"8","key":"R1","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-023-00370-1","article-title":"The future landscape of large language models in medicine","volume":"3","author":"Clusmann","journal-title":"Commun Med (Lond)"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","DOI":"10.1186\/s12910-025-01316-z","article-title":"Performance of large language models in non-English medical ethics-related multiple choice questions: comparison of ChatGPT performance across versions and languages","volume":"26","author":"Kim","journal-title":"BMC Med Ethics"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.naacl-long.206","article-title":"KMMLU: measuring massive multitask language understanding in Korean","author":"Son","journal-title":"Assoc Comput Linguist"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2025.1504805","article-title":"Bridging the gap: a practical step-by-step approach to warrant safe implementation of large language models in healthcare","volume":"8","author":"Workum","journal-title":"Front Artif Intell"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2025.1706383","article-title":"Epistemic and ethical limits of large language models in evidence-based medicine: from knowledge to judgment","volume":"7","author":"Qi","journal-title":"Front Digit Health"},{"issue":"6","key":"R7","doi-asserted-by":"publisher","first-page":"519","DOI":"10.3348\/kjr.2025.0257","article-title":"Overview of South Korean guidelines for approval of large language or multimodal models as medical devices: key features and areas for improvement","volume":"26","author":"Park","journal-title":"Korean J Radiol"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","first-page":"4","DOI":"10.3348\/kjr.2025.1647","article-title":"Access and reimbursement for artificial intelligence in radiology: a Korean perspective","volume":"27","author":"Lee","journal-title":"Korean J Radiol"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-015-1059-x","article-title":"Between government policy, clinical autonomy, and market demands: a qualitative study of the impact of the prescribing analysis system on behavior of physicians in South Korea","volume":"15","author":"Kim","journal-title":"BMC Health Serv Res"},{"issue":"17","key":"R10","doi-asserted-by":"publisher","DOI":"10.3390\/jcm14176169","article-title":"Challenges of implementing LLMs in clinical practice: perspectives","volume":"14","author":"Artsi","journal-title":"J Clin Med"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-67689-0","article-title":"Disparities in medical recommendations from AI-based chatbots across different countries\/regions","volume":"14","author":"Gumilar","journal-title":"Sci Rep"},{"issue":"9","key":"R12","doi-asserted-by":"publisher","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","journal-title":"Nat Med"},{"key":"R13","doi-asserted-by":"crossref","unstructured":"Kim Y Jeong H Chen S et al. Medical hallucination in foundation models and their impact on healthcare. medRxiv. Preprint posted online on Mar 3, 2025. [doi: 10.1101\/2025.02.28.25323115]","DOI":"10.1101\/2025.02.28.25323115"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Lee WB . Regulating artificial intelligence in medical care in South Korea. In: Solaiman B Cohen IG , editors. Research Handbook on Health, AI and the Law. Edward Elgar Publishing Ltd; 2024. [doi: 10.4337\/9781802205657.ch22]","DOI":"10.4337\/9781802205657.ch22"},{"issue":"Suppl 7","key":"R15","doi-asserted-by":"publisher","first-page":"242","DOI":"10.1186\/s12911-020-01241-8","article-title":"Korean clinical entity recognition from diagnosis text using BERT","volume":"20","author":"Kim","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-12845-3","article-title":"A patient-centered approach to developing and validating a natural language processing model for extracting patient-reported symptoms","volume":"15","author":"Watabe","journal-title":"Sci Rep"},{"issue":"3","key":"R17","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1097\/NNR.0000000000000488","article-title":"Identifying symptom information in clinical notes using natural language processing","volume":"70","author":"Koleck","journal-title":"Nurs Res"},{"key":"R18","volume":"2024","author":"Adam","journal-title":"AMIA Annu Symp Proc"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-022-17806-8","article-title":"A pre-trained BERT for Korean medical natural language processing","volume":"12","author":"Kim","journal-title":"Sci Rep"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","first-page":"108","DOI":"10.31803\/tg-20250410040449","article-title":"Real-time hybrid query transformation method for enhancing search system performance in Korean language applications","volume":"20","author":"Yoo","journal-title":"Teh glas (Online)"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.findings-acl.350","article-title":"Can we trust AI doctors? A survey of medical hallucination in large language and large vision-language models","author":"Zhu","journal-title":"Assoc Comput Linguist"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.2196\/70190","article-title":"Enhancing large language models for improved accuracy and safety in medical question answering: comparative study","volume":"11","author":"Wang","journal-title":"JMIR Med Educ"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-02167-z","article-title":"Bridging the mentorship divide: how large language models could reshape medical workforce equity","volume":"9","author":"Nguyen","journal-title":"npj Digit Med"},{"issue":"4","key":"R24","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1093\/jamia\/ocaf008","article-title":"Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines","volume":"32","author":"Liu","journal-title":"J Am Med Inform Assoc"},{"key":"R25","unstructured":"Lewis P Perez E Piktus A et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. arXiv. Preprint posted online on Apr 12, 2021. [doi: 10.48550\/arXiv.2005.11401]"},{"key":"R26","unstructured":"Liu NF Lin K Hewitt J et al. Lost in the middle: how language models use long contexts. arXiv. Preprint posted online on Nov 20, 2023. [doi: 10.48550\/arXiv.2307.03172]"},{"key":"R27","doi-asserted-by":"crossref","unstructured":"Poliakov M Shvai N . Multi-meta-RAG: improving RAG for multi-hop queries using database filtering with LLM-extracted metadata. 2025;2359:334-342. [doi: 10.1007\/978-3-031-81372-6_25]","DOI":"10.1007\/978-3-031-81372-6_25"},{"key":"R28","unstructured":"Bruni D Avvenuti M Tonellotto N Tesconi M . AMAQA: a metadata-based QA dataset for RAG systems. arXiv. Preprint posted online on Feb 12, 2026. [doi: 10.48550\/arXiv.2505.13557]"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.2196\/69857","article-title":"Performance of open-source large language models in psychiatry: usability study through comparative analysis of non-English records and English translations","volume":"27","author":"Kim","journal-title":"J Med Internet Res"},{"key":"R30","unstructured":"Kweon S Choi B Chu G et al. KorMedMCQA: multi-choice question answering benchmark for Korean healthcare professional licensing examinations. arXiv. Preprint posted online on Dec 9, 2024. [doi: 10.48550\/arXiv.2403.01469]"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.137","article-title":"M3-embedding: multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation","author":"Chen","journal-title":"Assoc Comput Linguist"},{"issue":"11","key":"R32","doi-asserted-by":"publisher","first-page":"11","DOI":"10.53469\/jrse.2022.04(11).07","article-title":"A comprehensive study of elastic search","volume":"4","author":"Prabhu","journal-title":"JRSE"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-67429-4","article-title":"A chatbot based question and answer system for the auxiliary diagnosis of chronic diseases based on large language model","volume":"14","author":"Zhang","journal-title":"Sci Rep"},{"key":"R34","unstructured":"Enforcement Rule of the Bioethics and Safety Act [Article in Korean]. Korea Ministry of Government Legislation (National Law Information Center). Jun 2, 2025. URL: https:\/\/www.law.go.kr\/LSW\/lsLinkCommonInfo.do?lspttninfSeq=75929&chrClsCd=010202 [Accessed 18-04-2026]"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e72604","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:02:57Z","timestamp":1782842577000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e72604"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,30]]},"references-count":34,"URL":"https:\/\/doi.org\/10.2196\/72604","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,30]]},"article-number":"v10i8e72604"}}