{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,19]],"date-time":"2026-10-19T18:55:01Z","timestamp":1792436101670,"version":"4.5.0"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2025,7,19]],"date-time":"2025-07-19T00:00:00Z","timestamp":1752883200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,7,19]],"date-time":"2025-07-19T00:00:00Z","timestamp":1752883200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Radiol Phys Technol"],"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Recent advances in large language models (LLMs) enable domain-specific question answering using external knowledge. However, addressing information that is not included in training data remains a challenge, particularly in nuclear medicine, where examination protocols are frequently updated and vary across institutions. In this study, we developed a retrieval-augmented generation (RAG) system using 40 internal manuals from a single Japanese hospital, each corresponding to a different examination in nuclear medicine. These institution-specific documents were segmented and indexed using a hybrid retrieval strategy combining dense vector search (text-embedding-3-small) and sparse keyword search (BM25). GPT-3.5 and GPT-4o were used with the OpenAI application programming interface (API) for response generation. The quality of the generated answers was assessed using a four-point Likert scale by three certified radiological technologists, of which one held an additional certification in nuclear medicine and another held an additional certification in medical physics. Automated evaluation was conducted using RAGAS metrics, including factual correctness and context recall. The GPT-4o model combined with hybrid retrieval achieved the highest performance, as per expert evaluations. Although traditional string-based metrics such as ROUGE and the Levenshtein distance poorly align with human ratings, RAGAS provided consistent rankings across system configurations, despite showing only a modest correlation with manual scores. These findings demonstrate that integrating examination-specific institutional manuals into RAG frameworks can effectively support domain-specific question answering in nuclear medicine. Moreover, LLM-based evaluation methods such as RAGAS may serve as practical tools to complement expert reviews in developing healthcare-oriented artificial intelligence systems.<\/jats:p>","DOI":"10.1007\/s12194-025-00941-y","type":"journal-article","created":{"date-parts":[[2025,7,19]],"date-time":"2025-07-19T17:01:06Z","timestamp":1752944466000},"page":"861-876","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Evaluation of a retrieval-augmented generation system using a Japanese Institutional Nuclear Medicine Manual and large language model-automated scoring"],"prefix":"10.1007","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0267-6675","authenticated-orcid":false,"given":"Yusuke","family":"Fukui","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuhei","family":"Kawata","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kazumasa","family":"Kobashi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1110-963X","authenticated-orcid":false,"given":"Yukihiro","family":"Nagatani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Harumi","family":"Iguchi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,7,19]]},"reference":[{"key":"941_CR1","unstructured":"Kaddour J, Harris J, Mozes M, et al. Challenges and Applications of Large Language Models. arXiv. 2023; arXiv:2307.10169."},{"key":"941_CR2","unstructured":"Chen ZZ, Ma J, Zhang X, et al. A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law. arXiv. 2024; arXiv:2405.01769."},{"key":"941_CR3","doi-asserted-by":"publisher","unstructured":"Bruscia M, Manduzio GA, Galatolo FA, et al. An overview on large language models across key domains: a systematic review. In: 2024 IEEE international conference on metrology for eXtended reality, artificial intelligence and neural engineering. 2024. p. 125\u201330. https:\/\/doi.org\/10.1109\/MetroXRAINE62247.2024.10797032.","DOI":"10.1109\/MetroXRAINE62247.2024.10797032"},{"key":"941_CR4","unstructured":"Chen Z, Hern\u00e1ndez Cano A, Romanou A, et al. MEDITRON-70B: scaling medical pretraining for large language models. arXiv. 2023; arXiv:2311.16079."},{"key":"941_CR5","unstructured":"Saab K, Tu T, Weng W-H, et al. Capabilities of gemini models in medicine. arXiv. 2024; arXiv:2404.18416."},{"issue":"6","key":"941_CR6","doi-asserted-by":"publisher","first-page":"603","DOI":"10.3390\/healthcare13060603","volume":"13","author":"J Vrdoljak","year":"2025","unstructured":"Vrdoljak J, Boban Z, Vilovi\u0107 M, Kumri\u0107 M, Bo\u017ei\u0107 J. A review of large language models in medical education, clinical decision support, and healthcare administration. Healthcare. 2025;13(6):603. https:\/\/doi.org\/10.3390\/healthcare13060603.","journal-title":"Healthcare"},{"key":"941_CR7","doi-asserted-by":"publisher","DOI":"10.3389\/fnume.2023.1213714","author":"M Djekidel","year":"2023","unstructured":"Djekidel M. The changing landscape of nuclear medicine and a new era: the \u201cNEW (Nu) CLEAR Medicine\u201d: a framework for the future. Front Nucl Med. 2023. https:\/\/doi.org\/10.3389\/fnume.2023.1213714.","journal-title":"Front Nucl Med"},{"issue":"Suppl 2","key":"941_CR8","doi-asserted-by":"publisher","first-page":"263S","DOI":"10.2967\/jnumed.120.254532","volume":"61","author":"WA Weber","year":"2020","unstructured":"Weber WA, Czernin J, Anderson CJ, et al. The future of nuclear medicine, molecular imaging, and theranostics. J Nucl Med. 2020;61(Suppl 2):263S-272S. https:\/\/doi.org\/10.2967\/jnumed.120.254532.","journal-title":"J Nucl Med"},{"key":"941_CR9","doi-asserted-by":"publisher","first-page":"9459","DOI":"10.5555\/3495724.3496517","volume":"33","author":"P Lewis","year":"2020","unstructured":"Lewis P, Perez E, Piktus A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv Neural Inf Process Syst. 2020;33:9459\u201374. https:\/\/doi.org\/10.5555\/3495724.3496517.","journal-title":"Adv Neural Inf Process Syst"},{"key":"941_CR10","doi-asserted-by":"publisher","unstructured":"Fan W, Ding Y, Ning L, et al. A survey on RAG meeting LLMs: towards retrieval-augmented large language models. In: Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining. 2024. p. 6491\u2013500. https:\/\/doi.org\/10.1145\/3637528.3671470.","DOI":"10.1145\/3637528.3671470"},{"issue":"3","key":"941_CR11","doi-asserted-by":"publisher","first-page":"445","DOI":"10.3390\/medicina60030445","volume":"60","author":"J Miao","year":"2024","unstructured":"Miao J, Thongprayoon C, Suppadungsuk S, Valencia OAG, Cheungpasitporn W. Integrating retrieval-augmented generation with large language models in nephrology: advancing practical applications. Medicina. 2024;60(3):445. https:\/\/doi.org\/10.3390\/medicina60030445.","journal-title":"Medicina"},{"key":"941_CR12","doi-asserted-by":"publisher","DOI":"10.2196\/58041","volume":"26","author":"D Wang","year":"2024","unstructured":"Wang D, Liang J, Ye J, et al. Enhancement of the performance of large language models in diabetes education through retrieval-augmented generation: comparative study. J Med Internet Res. 2024;26: e58041. https:\/\/doi.org\/10.2196\/58041.","journal-title":"J Med Internet Res"},{"key":"941_CR13","doi-asserted-by":"crossref","unstructured":"Ke Y, Jin L, Elangovan K, et al. Development and testing of retrieval augmented generation in large language models \u2013 a case study report. arXiv. 2024; arXiv:2402.01733.","DOI":"10.2139\/ssrn.4719185"},{"key":"941_CR14","unstructured":"Gumma V, Raghunath A, Jain M, Sitaram S. Health-Pariksha: assessing RAG models for health chatbots in real-world multilingual settings. arXiv. 2024; arXiv:2410.13671."},{"key":"941_CR15","doi-asserted-by":"publisher","DOI":"10.1253\/circj.CJ-23-0308","author":"K Kusunose","year":"2023","unstructured":"Kusunose K, Kashima S, Sata M. Evaluation of the accuracy of ChatGPT in answering clinical questions on the Japanese Society of Hypertension Guidelines. Circ J. 2023. https:\/\/doi.org\/10.1253\/circj.CJ-23-0308.","journal-title":"Circ J"},{"key":"941_CR16","doi-asserted-by":"publisher","DOI":"10.5551\/jat.65240","author":"T Hisamatsu","year":"2024","unstructured":"Hisamatsu T, Fukuda M, Kinuta M, Kanda H. ChatGPT responses to clinical questions in the Japan Atherosclerosis Society Guidelines for prevention of atherosclerotic cardiovascular disease 2022. J Atheroscler Thromb. 2024. https:\/\/doi.org\/10.5551\/jat.65240.","journal-title":"J Atheroscler Thromb"},{"key":"941_CR17","unstructured":"Sukeda I, Suzuki M, Sakaji H, Kodera S. JMedLoRA: medical domain adaptation on Japanese large language models using instruction-tuning. In: Deep generative models for health workshop, NeurIPS 2023. 2023. https:\/\/openreview.net\/forum?id=BfHX0hKRSe."},{"key":"941_CR18","unstructured":"Chase, H. LangChain: a framework for developing applications with LLMs. GitHub repository. https:\/\/github.com\/langchain-ai\/langchain. Accessed 31 May 2025."},{"key":"941_CR19","unstructured":"eiennohito, hayashi_mas, worksapplications. SudachiPy: Python version of Sudachi, a Japanese tokenizer. GitHub repository. https:\/\/github.com\/WorksApplications\/SudachiPy. Accessed 31 May 2025."},{"key":"941_CR20","doi-asserted-by":"crossref","unstructured":"Es S, James J, Espinosa Anke L, Schockaert S. RAGAS: automated evaluation of retrieval augmented generation. In: Proceedings of the 18th conference of the european chapter of the association for computational linguistics: system demonstrations. 2024. p. 150\u20138. https:\/\/aclanthology.org\/2024.eacl-demo.16\/.","DOI":"10.18653\/v1\/2024.eacl-demo.16"},{"key":"941_CR21","doi-asserted-by":"publisher","first-page":"1500","DOI":"10.1162\/tacl_a_00615","volume":"11","author":"NM Guerreiro","year":"2023","unstructured":"Guerreiro NM, Alves DM, Waldendorf J. Hallucinations in large multilingual translation models. Trans Assoc Comput Linguist. 2023;11:1500\u201317. https:\/\/doi.org\/10.1162\/tacl_a_00615.","journal-title":"Trans Assoc Comput Linguist"},{"key":"941_CR22","doi-asserted-by":"publisher","unstructured":"Ren R, Qu Y, Liu J, Zhao X. A thorough examination on zero-shot dense retrieval. In: Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. p. 15783\u201396. https:\/\/doi.org\/10.18653\/v1\/2023.findings-emnlp.1057.","DOI":"10.18653\/v1\/2023.findings-emnlp.1057"},{"key":"941_CR23","unstructured":"Desetty RT. Evaluating the ideal chunk size for a RAG system using LlamaIndex [Internet]. LlamaIndex Blog; 2023 Oct 5 [cited 2025 May 31]. Available from: https:\/\/www.llamaindex.ai\/blog\/evaluating-the-ideal-chunk-size-for-a-rag-system-using-llamaindex-6207e5d3fec5"},{"key":"941_CR24","unstructured":"Berntson A. Azure AI search: outperforming vector search with hybrid retrieval and reranking [Internet]. Microsoft Tech Community; 2023 Sep 18 [cited 2025 May 31]. Available from: https:\/\/techcommunity.microsoft.com\/blog\/azure-ai-services-blog\/azure-ai-search-outperforming-vector-search-with-hybrid-retrieval-and-reranking\/3929167"},{"key":"941_CR25","doi-asserted-by":"publisher","unstructured":"Kim Y, Jeong H, Chen S, et al. Medical hallucination in foundation models and their impact on healthcare. medRxiv. 2025. https:\/\/doi.org\/10.1101\/2025.02.28.25323115.","DOI":"10.1101\/2025.02.28.25323115"},{"key":"941_CR26","doi-asserted-by":"publisher","first-page":"205520762513371","DOI":"10.1177\/20552076251337177","volume":"11","author":"OK Gargari","year":"2025","unstructured":"Gargari OK, Habibi G. Enhancing medical AI with retrieval-augmented generation: a mini narrative review. Digit Health. 2025;11:20552076251337176. https:\/\/doi.org\/10.1177\/20552076251337177.","journal-title":"Digit Health"},{"issue":"1","key":"941_CR27","first-page":"19","volume":"21","author":"M Tomczak","year":"2014","unstructured":"Tomczak M, Tomczak E. The need to report effect size estimates revisited: An overview of some recommended measures of effect size. Trends Sport Sci. 2014;21(1):19\u201325.","journal-title":"Trends Sport Sci"}],"container-title":["Radiological Physics and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12194-025-00941-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s12194-025-00941-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12194-025-00941-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,11]],"date-time":"2025-08-11T12:10:08Z","timestamp":1754914208000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s12194-025-00941-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,19]]},"references-count":27,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,9]]}},"alternative-id":["941"],"URL":"https:\/\/doi.org\/10.1007\/s12194-025-00941-y","relation":{},"ISSN":["1865-0333","1865-0341"],"issn-type":[{"value":"1865-0333","type":"print"},{"value":"1865-0341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,19]]},"assertion":[{"value":"30 May 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 July 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 July 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 July 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"This study involved human expert evaluation of model outputs. Before conducting the study, the Ethics Committee of Shiga University of Medical Science was consulted and confirmed that formal ethical approval was not required for this type of research.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}}]}}