{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:08:20Z","timestamp":1782846500440,"version":"3.54.5"},"reference-count":29,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Antimicrobial resistance (AMR) poses a critical global health threat, undermining the efficacy of antibiotics and complicating clinical decision-making. Although scientific literature on AMR is extensive, retrieving and synthesizing relevant evidence remains time-consuming for clinicians and researchers. Recent advances in large language models (LLMs) offer opportunities to enhance access to domain-specific knowledge. However, the diversity of available models, ranging from open-source to commercial, necessitates a systematic comparison of their performance, cost, and scalability in real-world biomedical applications.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to describe the development of a retrieval-augmented generation (RAG) chatbot for AMR literature analysis and compare multiple commercial and open-source LLMs in terms of accuracy, faithfulness, response time, and cost-efficiency.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n A corpus of 164 peer-reviewed AMR-related articles was compiled from Google Scholar and embedded into a ChromaDB vector database using OpenAI\u2019s text-embedding-ada-002. The RAG chatbot was implemented to operate with 5 LLM backbones: GPT-4, GPT-4o, GPT-4o-mini, Claude 3.7 Sonnet, and LLaMA 4 Maverick. For each model, a temperature ablation study was performed to determine optimal performance. Evaluation metrics included correctness (pass rate and score), faithfulness, relevancy, computational cost, and latency, using a synthetic ground truth dataset generated with GPT-4.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n All models generated scientifically grounded responses when integrated into the RAG framework. GPT-4 achieved the highest correctness score (94.7%) but incurred the highest cost, while GPT-4o delivered nearly identical accuracy at a 9-fold lower cost and the fastest response time (3.88 s). LLaMA 4 Maverick and GPT-4o-mini offered lower accuracy but substantially reduced operational costs. Claude 3.7 Sonnet showed competitive accuracy, but the least favorable cost-performance ratio. Qualitative analysis revealed differences in response style, detail, and structure among models.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n A RAG-based chatbot can effectively support AMR research by delivering accurate, context-grounded, and scalable access to scientific literature. The comparative evaluation highlights trade-offs between performance, cost, and speed, guiding the selection of LLM architectures for clinical and research settings. Future work will focus on integrating language-specific embeddings and specialized domain agents to further enhance accuracy, adaptability, and clinical use.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/83206","type":"journal-article","created":{"date-parts":[[2025,10,24]],"date-time":"2025-10-24T14:35:05Z","timestamp":1761316505000},"page":"e83206-e83206","source":"Crossref","is-referenced-by-count":0,"title":["Evaluation of a Retrieval-Augmented Generation Chatbot for Antimicrobial Resistance Research: Comparative Analysis of Large Language Models"],"prefix":"10.2196","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5632-1212","authenticated-orcid":false,"given":"Oscar","family":"Escudero-Arnanz","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-4649-2985","authenticated-orcid":false,"given":"Manuel Eduardo","family":"Valero-M\u00e9ndez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6237-724X","authenticated-orcid":false,"given":"Noelia","family":"S\u00e1nchez-Ramos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5817-989X","authenticated-orcid":false,"given":"Cristina","family":"Soguero-Ru\u00edz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,3,24]]},"reference":[{"key":"R1","unstructured":"Antimicrobial resistance. World Health Organization. URL: https:\/\/www.who.int\/news-room\/fact-sheets\/detail\/antimicrobial-resistance [Accessed 08-03-2026]"},{"issue":"10","key":"R2","doi-asserted-by":"publisher","first-page":"726","DOI":"10.1038\/s41570-021-00313-1","article-title":"Towards the sustainable discovery and development of new antibiotics","volume":"5","author":"Miethke","journal-title":"Nat Rev Chem"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1016\/j.csbj.2021.05.034","article-title":"Antibiotic resistance: time of synthesis in a post-genomic age","volume":"19","author":"Gil-Gil","journal-title":"Comput Struct Biotechnol J"},{"key":"R4","doi-asserted-by":"crossref","unstructured":"Mart\u00edn-Palomeque P Escudero-Arnanz \u00d3 \u00c1lvarez-Rodr\u00edguez J Soguero-Ruiz C . Irregular temporal classification of multidrug resistance development in intensive care unit patients. Presented at: 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM); 5016-5023; Lisbon, Portugal. 2024.[doi: 10.1109\/BIBM62325.2024.10822248]","DOI":"10.1109\/BIBM62325.2024.10822248"},{"key":"R5","unstructured":"Pier B Romary L . Excessive growth in the number of scientific publications. Ouvrir la Science. URL: https:\/\/www.ouvrirlascience.fr\/excessive-growth-in-the-number-of-scientific-publications\/ [Accessed 22-05-2025]"},{"key":"R6","unstructured":"Zhu Y Yuan H Wang S et al. Large language models for information retrieval: a survey. arXiv. Preprint posted online on Aug 14, 2023. [doi: 10.48550\/arXiv.2308.07107]"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.nlp.2025.100128","article-title":"A survey on chatbots and large language models: testing and evaluation techniques","volume":"10","author":"Singh","journal-title":"Nat Lang Process J"},{"issue":"6","key":"R8","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare13060603","article-title":"A review of large language models in medical education, clinical decision support, and healthcare administration","volume":"13","author":"Vrdoljak","journal-title":"Healthcare (Basel)"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01083-y","article-title":"Evaluating large language models as agents in the clinic","volume":"7","author":"Mehandru","journal-title":"NPJ Digit Med"},{"key":"R10","unstructured":"Centulani G . Understanding hallucination in LLMs: causes, consequences and mitigation strategies. Medium. URL: https:\/\/medium.com\/@gcentulani\/understanding-hallucination-in-llms-causes-consequences-and-mitigation-strategies-b5e1d0268069 [Accessed 22-05-2025]"},{"issue":"4","key":"R11","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1093\/jamia\/ocaf008","article-title":"Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines","volume":"32","author":"Liu","journal-title":"J Am Med Inform Assoc"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1016\/j.crbiot.2024.100197","article-title":"Chatting with artificial intelligence to combat antibiotic resistance: opportunities and challenges","volume":"7","author":"Abavisani","journal-title":"Current Research in Biotechnology"},{"issue":"2","key":"R13","doi-asserted-by":"publisher","first-page":"1243","DOI":"10.1021\/acs.est.4c07842","article-title":"Using large language models to assist antimicrobial resistance policy development: integrating the environment into health protection planning","volume":"59","author":"Chen","journal-title":"Environ Sci Technol"},{"key":"R14","unstructured":"OpenAI text-embedding-ada-002. OpenAI. URL: https:\/\/platform.openai.com\/docs\/guides\/embeddings\/what-are-embeddings [Accessed 22-05-2025]"},{"key":"R15","unstructured":"Chroma: the AI-native open-source embedding database. GitHub. URL: https:\/\/github.com\/chroma-core\/chroma [Accessed 22-05-2025]"},{"key":"R16","unstructured":"OpenAI. GPT-4 technical report. arXiv. Preprint posted online on Mar 15, 2023. [doi: 10.48550\/arXiv.2303.08774]"},{"key":"R17","unstructured":"Hello, GPT-4o. OpenAI. URL: https:\/\/openai.com\/es-ES\/index\/hello-gpt-4o\/ [Accessed 26-06-2025]"},{"key":"R18","unstructured":"GPT-4o mini: advancing cost-efficient intelligence. OpenAI. URL: https:\/\/openai.com\/en\/index\/gpt-4o-mini-advancing-cost-efficient-intelligence\/ [Accessed 22-05-2025]"},{"key":"R19","unstructured":"Claude 3.7 sonnet and claude code. Anthropic. URL: https:\/\/www.anthropic.com\/news\/claude-3-7-sonnet [Accessed 16-06-2025]"},{"key":"R20","unstructured":"Unmatched performance and efficiency. Llama 4. URL: https:\/\/www.llama.com\/models\/llama-4\/ [Accessed 16-06-2025]"},{"key":"R21","unstructured":"AI agents for document OCR + workflows. LlamaIndex. URL: https:\/\/www.llamaindex.ai [Accessed 22-05-2025]"},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Goodrich B Rao V Liu PJ Saleh M . Assessing the factual accuracy of generated text. Presented at: KDD \u201919: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining; Aug 4-8, 2019; Anchorage AK, USA. [doi: 10.1145\/3292500.3330955]","DOI":"10.1145\/3292500.3330955"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00667","article-title":"Evaluating correctness and faithfulness of instruction-following models for question answering","volume":"12","author":"Adlakha","journal-title":"Trans Assoc Comput Linguist"},{"key":"R24","unstructured":"Document 32016R0679: regulation (EU) 2016\/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95\/46\/EC (general data protection regulation) (text with EEA relevance). EUR-Lex. URL: https:\/\/eur-lex.europa.eu\/eli\/reg\/2016\/679\/oj [Accessed 06-02-2026]"},{"key":"R25","unstructured":"Zou A Yu W Zhang H et al. DOCBENCH: a benchmark for evaluating LLM-based document reading systems. arXiv. Preprint posted online on Jul 15, 2024. [doi: 10.48550\/arXiv.2407.10701]"},{"key":"R26","unstructured":"Proprietary foundation model serving. Databricks. URL: https:\/\/www.databricks.com\/product\/pricing\/anthropic-foundation-model-serving [Accessed 26-06-2025]"},{"key":"R27","unstructured":"Mosaic AI foundation model serving. Databricks. URL: https:\/\/www.databricks.com\/product\/pricing\/foundation-model-serving [Accessed 26-06-2025]"},{"key":"R28","unstructured":"Calculadora de precios [Web page in Spanish]. Microsoft Azure. URL: https:\/\/azure.microsoft.com\/es-es\/pricing\/calculator\/?cdn=disable [Accessed 26-06-2025]"},{"key":"R29","unstructured":"Escudero-Arnanz O Valero-M\u00e9ndez M S\u00e1nchez-Ramos N Soguero-Ruiz C . RAG-based chatbot for antimicrobial resistance literature analysis. GitHub repository. 2025. URL: https:\/\/github.com\/valerovalero\/rag_chatbot_bacterias [Accessed 26-02-2026]"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e83206","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:17:15Z","timestamp":1782843435000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e83206"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,24]]},"references-count":29,"URL":"https:\/\/doi.org\/10.2196\/83206","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,24]]},"article-number":"v5i10e83206"}}