{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T21:01:00Z","timestamp":1792789260136,"version":"4.5.0"},"reference-count":41,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2026,9,27]],"date-time":"2026-09-27T00:00:00Z","timestamp":1790467200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Excellence in Production Research Framework"},{"name":"RESILIENT Competence Center"},{"name":"Swedish Energy Agency","id":[{"id":"https:\/\/ror.org\/0359z7n90","id-type":"ROR","asserted-by":"crossref"}]},{"id":[{"id":"https:\/\/ror.org\/0359z7n90","id-type":"ROR","asserted-by":"publisher"}]},{"name":"M\u00e4lardalen University","id":[{"id":"https:\/\/ror.org\/033vfbz75","id-type":"ROR","asserted-by":"crossref"}]},{"id":[{"id":"https:\/\/ror.org\/033vfbz75","id-type":"ROR","asserted-by":"publisher"}]},{"name":"Turkish Health Institutes","award":["33987"],"award-info":[{"award-number":["33987"]}]},{"name":"Kocaeli University","award":["FKA-2026-5195"],"award-info":[{"award-number":["FKA-2026-5195"]}],"id":[{"id":"https:\/\/ror.org\/0411seq30","id-type":"ROR","asserted-by":"crossref"}]},{"name":"Kocaeli University","award":["FBA-2026-4982"],"award-info":[{"award-number":["FBA-2026-4982"]}],"id":[{"id":"https:\/\/ror.org\/0411seq30","id-type":"ROR","asserted-by":"crossref"}]},{"award":["FKA-2026-5195"],"award-info":[{"award-number":["FKA-2026-5195"]}],"id":[{"id":"https:\/\/ror.org\/0411seq30","id-type":"ROR","asserted-by":"publisher"}]},{"award":["FBA-2026-4982"],"award-info":[{"award-number":["FBA-2026-4982"]}],"id":[{"id":"https:\/\/ror.org\/0411seq30","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background\/Objectives: Large language models (LLMs) are increasingly explored for clinical decision support and digital health applications. However, reliable diagnostic assistance remains challenging for low-resource medical languages such as Turkish due to limited source grounding, transparency, clinical safety, and localized medical knowledge. This study presents TurkishMedLLM, a source-grounded and safety-aware Turkish medical LLM designed for clinician-supervised diagnostic decision support, symptom interpretation, and patient care. Methods: The methodology integrates multi-source Turkish medical data ingestion, schema standardization, duplicate removal, quality filtering, supervised fine-tuning, embedding generation, vector indexing, Qwen3-8B fine-tuning, retrieval-augmented generation (RAG), and multi-layer evaluation. The system was evaluated using retrieval metrics, ROUGE, RAGAS, DeepEval, and clinical safety assessments. As a use case, TurkishMedLLM was integrated into the AI-based Diabetes Care (AIDCare) mHealth platform, which supports patient queries related to lifestyle management, symptoms, diagnosis, and treatment of diabetes. The system generates safety-aware responses with clinician-in-the-loop validation before delivery through the mobile application. Results: After pre-processing, the final dataset comprised 232,926 unique documents, including 210,791 Turkish medical question\u2013answer pairs and 22,135 hospital medical articles. The retrieval module achieved Hit Rate@1, Hit Rate@3, and Hit Rate@5 of 94.67%, 98.67%, and 100.00%, respectively, indicating consistent retrieval of clinically relevant evidence. QLoRA fine-tuning achieved a validation loss of 0.9373 and ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Lsum scores of 0.8338, 0.5476, 0.7836, and 0.7372, respectively. The fine-tuned RAG system achieved RAGAS faithfulness and response relevancy scores of 0.91 and 0.88, respectively, while DeepEval achieved an answer relevancy score of 0.90. Clinical safety assessment achieved a caution score of 0.93, indicating generally evidence-grounded and clinically cautious responses for symptom interpretation and diagnosis-related patient support. Conclusions: Combining retrieval grounding, parameter-efficient fine-tuning, and multi-layer safety evaluation provides a promising approach for clinician-supervised medical AI in Turkish. TurkishMedLLM demonstrates potential for symptom interpretation, differential diagnostic support, and trustworthy digital health applications.<\/jats:p>","DOI":"10.3390\/diagnostics16193142","type":"journal-article","created":{"date-parts":[[2026,9,28]],"date-time":"2026-09-28T11:35:59Z","timestamp":1790595359000},"page":"3142","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Design and Evaluation of a Source-Grounded Medical LLM for Clinical Decision Support and Patient Care in Trustworthy Diagnostic Systems"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0934-1069","authenticated-orcid":false,"given":"Muhammad","family":"Jamil","sequence":"first","affiliation":[{"name":"Department of Computer Engineering, Kocaeli University, Kocaeli 41001, T\u00fcrkiye"},{"name":"Wireless Information and Intelligent Systems (WINS) Research Center, Kocaeli University, Kocaeli 41001, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"visualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"software","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"formal-analysis","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5694-8042","authenticated-orcid":false,"given":"Adnan","family":"Kavak","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kocaeli University, Kocaeli 41001, T\u00fcrkiye"},{"name":"Wireless Information and Intelligent Systems (WINS) Research Center, Kocaeli University, Kocaeli 41001, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"investigation","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"supervision","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"formal-analysis","vocabulary":"credit"},{"role":"validation","vocabulary":"credit"},{"role":"project-administration","vocabulary":"credit"},{"role":"resources","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1214-9235","authenticated-orcid":false,"given":"Sevin\u00e7 \u0130lhan","family":"Omurca","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Kocaeli University, Kocaeli 41001, T\u00fcrkiye"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"project-administration","vocabulary":"credit"},{"role":"supervision","vocabulary":"credit"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5590-0784","authenticated-orcid":false,"given":"Hossein","family":"Fotouhi","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, M\u00e4lardalen University (MDU), 72123 V\u00e4ster\u00e5s, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"},{"role":"conceptualization","vocabulary":"credit"},{"role":"methodology","vocabulary":"credit"},{"role":"data-curation","vocabulary":"credit"},{"role":"writing-original-draft","vocabulary":"credit"},{"role":"writing-review-editing","vocabulary":"credit"},{"role":"funding-acquisition","vocabulary":"credit"},{"role":"project-administration","vocabulary":"credit"},{"role":"supervision","vocabulary":"credit"}]}],"member":"1968","published-online":{"date-parts":[[2026,9,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Stafie, C.S., Sufaru, I.G., Ghiciuc, C.M., Stafie, I.I., Sufaru, E.C., Solomon, S.M., and Hancianu, M. (2023). Exploring the intersection of artificial intelligence and clinical healthcare: A multidisciplinary review. Diagnostics, 13.","DOI":"10.3390\/diagnostics13121995"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Maleki Varnosfaderani, S., and Forouzanfar, M. (2024). The role of AI in hospitals and clinics: Transforming healthcare in the 21st century. Bioengineering, 11.","DOI":"10.3390\/bioengineering11040337"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"e66905","DOI":"10.2196\/66905","article-title":"Using large language models for chronic disease management tasks: Scoping review","volume":"13","author":"Serugunda","year":"2025","journal-title":"JMIR Med. Inform."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kwan, H.Y., Shell, J., Fahy, C., Yang, S., and Xing, Y. (2025). Integrating large language models into medication management in remote healthcare: Current applications, challenges, and future prospects. Systems, 13.","DOI":"10.3390\/systems13040281"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Liu, L., Yang, X., Lei, J., Shen, Y., Wang, J., Wei, P., Chu, Z., Qin, Z., and Ren, K. (IEEE Trans. Knowl. Data Eng., 2026). A survey on medical large language models: Technology, application, trustworthiness, and future directions, IEEE Trans. Knowl. Data Eng., Early Access.","DOI":"10.1109\/TKDE.2026.3709941"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Maity, S., and Saikia, M.J. (2025). Large language models in healthcare and medical applications: A review. Bioengineering, 12.","DOI":"10.3390\/bioengineering12060631"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Saidu, F., and Wall, J. (2026). Retrieval-augmented large language model for clinical decision support with a medical knowledge graph. Electronics, 15.","DOI":"10.3390\/electronics15030555"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Gou, F., Liu, J., Xiao, C., and Wu, J. (2024). Research on artificial-intelligence-assisted medicine: A survey on medical artificial intelligence. Diagnostics, 14.","DOI":"10.3390\/diagnostics14141472"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"\u0130ncidelen, M., and Aydo\u011fan, M. (2024). Developing question-answering models in low-resource languages: A case study on Turkish medical texts using transformer-based approaches. Proceedings of the 2024 8th International Artificial Intelligence and Data Processing Symposium (IDAP), IEEE.","DOI":"10.1109\/IDAP64064.2024.10711128"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1348","DOI":"10.1111\/all.15976","article-title":"Human-in-the-loop: Human involvement in enhancing medical inquiry performance in large language models","volume":"79","author":"Shu","year":"2024","journal-title":"Allergy"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Geng, H., Huang, Y., Lai, L., Du, Q., Chu, H., He, Z., Hu, J., and Tao, X. (2026). ProMedical: Hierarchical Fine-Grained Criteria Modeling for Medical LLM Alignment via Explicit Injection. Proceedings of the Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics.","DOI":"10.18653\/v1\/2026.acl-long.1714"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1007\/s44196-026-01402-z","article-title":"Quantized Low-Rank Adaptation in Large Language Models for Clinical Text Simplification","volume":"19","author":"Srinivasu","year":"2026","journal-title":"Int. J. Comput. Intell. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Aljohani, B., and Alsanoosy, T. (2026). Enhancing Medical Question Answering with LLMs via a Hybrid Retrieval-Augmented Generation Framework. Information, 17.","DOI":"10.3390\/info17020133"},{"key":"ref_14","unstructured":"Raza, M., Salem, S., Habib, A., Abdulmahmod, O., Kwon, H., Hussain, J., and Al-antari, M.A. MedXpert-CAD: A multimodal multi-agentic system for clinical imaging analysis via model context protocol LLM-driven agentic workflows. Proceedings of the International Workshop on Agentic AI for Medicine, Springer."},{"key":"ref_15","unstructured":"Lim, E., He, Y.V., Joselowitz, J., Preston, K., Chowdhury, M., Williams, L., Higham, A., Mason, K., Melo, M., and Lawton, T. (2025). MATRIX: Multi-Agent simulaTion fRamework for safe Interactions and conteXtual clinical conversational evaluation. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"e82997","DOI":"10.2196\/82997","article-title":"Improving Retrieval Augmented Generation for Health Care by Fine-Tuning Clinical Embedding Models: Development and Evaluation Study","volume":"28","author":"Arzideh","year":"2026","journal-title":"J. Med. Internet Res."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Pingua, B., Sahoo, A., Kandpal, M., Murmu, D., Rautaray, J., Barik, R.K., and Saikia, M.J. (2025). Medical LLMs: Fine-tuning vs. retrieval-augmented generation. Bioengineering, 12.","DOI":"10.3390\/bioengineering12070687"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","year":"2025","journal-title":"Nat. Med."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Urda-C\u00eempean, A.E., Leucu\u021ba, D.C., Drugan, C., Du\u021bu, A.G., C\u0103linici, T., and Drugan, T. (2025). Assessing the accuracy of diagnostic capabilities of large language models. Diagnostics, 15.","DOI":"10.3390\/diagnostics15131657"},{"key":"ref_20","first-page":"1","article-title":"Pre-trained language models in biomedical domain: A systematic survey","volume":"56","author":"Wang","year":"2023","journal-title":"ACM Comput. Surv."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1038\/s44222-025-00279-5","article-title":"Application of large language models in medicine","volume":"3","author":"Liu","year":"2025","journal-title":"Nat. Rev. Bioeng."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2114","DOI":"10.1093\/jamia\/ocae074","article-title":"Large language models for biomedicine: Foundations, opportunities, challenges, and best practices","volume":"31","author":"Sahoo","year":"2024","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_23","first-page":"e39305","article-title":"Embracing large language models for medical applications: Opportunities and challenges","volume":"15","author":"Karabacak","year":"2023","journal-title":"Cureus"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2630001","DOI":"10.1142\/S2737599426300011","article-title":"Large language models in healthcare and biomedical informatics: A comprehensive review","volume":"13","author":"Hornback","year":"2026","journal-title":"Innov. Emerg. Technol."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"e2662","DOI":"10.7717\/peerj-cs.2662","article-title":"TurkMedNLI: A Turkish medical natural language inference dataset through large language model based translation","volume":"11","author":"Soygazi","year":"2025","journal-title":"PeerJ Comput. Sci."},{"key":"ref_26","unstructured":"D\u00f6nmez, T.B. (2026, May 25). TUSGPT-TR-Medical-Dataset-v1: A Turkish Medical SFT Dataset. Available online: https:\/\/huggingface.co\/datasets\/turkerberkdonmez\/TUSGPT-TR-Medical-Dataset-v1."},{"key":"ref_27","unstructured":"Bayram, M.A. (2026, May 25). T\u00fcrk\u00e7e T\u0131bbi Soru-Cevap Veri Seti: 167 Bin Sa\u011fl\u0131k Sorusu ve Cevab\u0131. Available online: https:\/\/zenodo.org\/records\/12770916."},{"key":"ref_28","unstructured":"Bayram, A. (2026, May 25). Turkish Hospital Medical Articles Dataset. Available online: https:\/\/huggingface.co\/datasets\/alibayram\/turkish-hospital-medical-articles."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1038\/s41746-025-01519-z","article-title":"Retrieval augmented generation for 10 large language models and its generalizability in assessing medical fitness","volume":"8","author":"Ke","year":"2025","journal-title":"npj Digit. Med."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1015","DOI":"10.1007\/s13042-024-02318-w","article-title":"Large language models for medicine: A survey","volume":"16","author":"Zheng","year":"2025","journal-title":"Int. J. Mach. Learn. Cybern."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Nazi, Z.A., and Peng, W. (2024). Large Language Models in Healthcare and Medical Domain: A Review. Informatics, 11.","DOI":"10.3390\/informatics11030057"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Neha, F., Bhati, D., and Shukla, D.K. (2025). Retrieval-augmented generation (rag) in healthcare: A comprehensive review. AI, 6.","DOI":"10.20944\/preprints202508.1022.v1"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Patil, R., Abbidi, M., and Fannon, S. (2025). RAGMed: A RAG-based medical AI assistant for improving healthcare delivery. AI, 6.","DOI":"10.3390\/ai6100240"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Chen, J., Xiao, S., Zhang, P., Luo, K., and Lian, D. (2024). BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv.","DOI":"10.18653\/v1\/2024.findings-acl.137"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Fan, Y., Wang, N., Xue, K., Liu, J., and Ruan, T. (2025). Medeureka: A medical domain benchmark for multi-granularity and multi-data-type embedding-based retrieval. Proceedings of the Findings of the Association for Computational Linguistics: NAACL 2025, Association for Computational Linguistics.","DOI":"10.18653\/v1\/2025.findings-naacl.154"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"100184","DOI":"10.1016\/j.mcpdig.2024.11.005","article-title":"Fine-tuning large language models for specialized use cases","volume":"3","author":"Anisuzzaman","year":"2025","journal-title":"Mayo Clin. Proc. Digit. Health"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"10088","DOI":"10.52202\/075280-0441","article-title":"Qlora: Efficient finetuning of quantized llms","volume":"36","author":"Dettmers","year":"2023","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_38","unstructured":"Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., and Lv, C. (2025). Qwen3 technical report. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1080\/17843286.2026.2613903","article-title":"Effective prompt design for large language models in clinical practice","volume":"81","author":"Callens","year":"2026","journal-title":"Acta Clin. Belg."},{"key":"ref_40","unstructured":"OpenAI (2026, May 11). OpenAI Evals: Framework for Evaluating LLMs and LLM Systems. Available online: https:\/\/github.com\/openai\/evals."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/19\/3142\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T04:16:09Z","timestamp":1790655369000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/16\/19\/3142"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,27]]},"references-count":41,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2026,10]]}},"alternative-id":["diagnostics16193142"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics16193142","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,27]]}}}