{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T23:31:46Z","timestamp":1774481506878,"version":"3.50.1"},"reference-count":36,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2025,11,3]],"date-time":"2025-11-03T00:00:00Z","timestamp":1762128000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002322","name":"Coordena\u00e7\u00e3o de Aperfei\u00e7oamento de Pessoal de N\u00edvel Superior\u2014Brasil","doi-asserted-by":"crossref","award":["Finance Code 001"],"award-info":[{"award-number":["Finance Code 001"]}],"id":[{"id":"10.13039\/501100002322","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100003593","name":"National Council for Scientific and Technological Development","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100003593","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Telemedicine Laboratory (LabTelemed) of Federal University of Santa Catarina"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Applied Sciences"],"abstract":"<jats:p>The development of new technologies, improved by advances in artificial intelligence, has enabled the creation of a new generation of applications in different scenarios. In medical systems, adopting AI-driven solutions has brought new possibilities, but their effective impacts still need further investigation. In this context, a chatbot prototype trained with large language models (LLMs) was developed using data from the Santa Catarina Telemedicine and Telehealth System (STT) Dermatology module. The system adapts Llama 3 8B via supervised Fine-tuning with QLoRA on a proprietary, domain-specific dataset (33 input-output pairs). Although it achieved 100% Fluency and 89.74% Coherence, Factual Correctness remained low (43.59%), highlighting the limitations of training LLMs on small datasets. In addition to G-Eval metrics, we conducted expert human validation, encompassing both quantitative and qualitative aspects. This low factual score indicates that a retrieval-augmented generation (RAG) mechanism is essential for robust information retrieval, which we outline as a primary direction for future work. This approach enabled a more in-depth analysis of a real-world telemedicine environment, highlighting both the practical challenges and the benefits of implementing LLMs in complex systems, such as those used in telemedicine.<\/jats:p>","DOI":"10.3390\/app152111732","type":"journal-article","created":{"date-parts":[[2025,11,3]],"date-time":"2025-11-03T16:18:42Z","timestamp":1762186722000},"page":"11732","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["From Knowledge Extraction to Assertive Response: An LLM Chatbot for Information Retrieval in Telemedicine Systems"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-4977-4463","authenticated-orcid":false,"given":"Bruna D.","family":"Pupo","sequence":"first","affiliation":[{"name":"Telemedicine Laboratory, Federal University of Santa Catarina (UFSC), Florian\u00f3polis 88040-900, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3988-8476","authenticated-orcid":false,"given":"Daniel G.","family":"Costa","sequence":"additional","affiliation":[{"name":"SYSTEC-ARISE, Faculty of Engineering, University of Porto, 4200-465 Porto, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2483-6382","authenticated-orcid":false,"given":"Roger","family":"Immich","sequence":"additional","affiliation":[{"name":"Instituto Metr\u00f3pole Digital (IMD\/UFRN), Natal 59078-900, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4532-1417","authenticated-orcid":false,"given":"Aldo von","family":"Wangenheim","sequence":"additional","affiliation":[{"name":"Telemedicine Laboratory, Federal University of Santa Catarina (UFSC), Florian\u00f3polis 88040-900, Brazil"},{"name":"Brazilian Institute for Digital Convergence (INCoD), Florian\u00f3polis 88040-900, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9144-1535","authenticated-orcid":false,"given":"Alex Sandro Roschildt","family":"Pinto","sequence":"additional","affiliation":[{"name":"Telemedicine Laboratory, Federal University of Santa Catarina (UFSC), Florian\u00f3polis 88040-900, Brazil"},{"name":"Brazilian Institute for Digital Convergence (INCoD), Florian\u00f3polis 88040-900, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3237-4168","authenticated-orcid":false,"given":"Douglas D. J.","family":"de Macedo","sequence":"additional","affiliation":[{"name":"Telemedicine Laboratory, Federal University of Santa Catarina (UFSC), Florian\u00f3polis 88040-900, Brazil"},{"name":"Brazilian Institute for Digital Convergence (INCoD), Florian\u00f3polis 88040-900, Brazil"}]}],"member":"1968","published-online":{"date-parts":[[2025,11,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1109\/ACCESS.2020.3045115","article-title":"Edge Intelligence and Internet of Things in Healthcare: A Survey","volume":"9","author":"Amin","year":"2021","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Al Khatib, I., Shamayleh, A., and Ndiaye, M. (2024). Healthcare and the Internet of Medical Things: Applications, Trends, Key Challenges, and Proposed Resolutions. Informatics, 11.","DOI":"10.3390\/informatics11030047"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"He, P., Huang, D., Wu, D., He, H., Wei, Y., Cui, Y., Wang, R., and Peng, L. (2024). A Survey of Internet of Medical Things: Technology, Application and Future Directions. Digit. Commun. Netw., in press.","DOI":"10.1016\/j.dcan.2024.11.013"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"104351","DOI":"10.1016\/j.tra.2024.104351","article-title":"Telemedicine Adoption before, during, and after COVID-19: The Role of Socioeconomic and Built Environment Variables","volume":"192","author":"Haddad","year":"2025","journal-title":"Transp. Res. Part A Policy Pract."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Alenoghena, C.O., Ohize, H.O., Adejo, A.O., Onumanyi, A.J., Ohihoin, E.E., Balarabe, A.I., Okoh, S.A., Kolo, E., and Alenoghena, B. (2023). Telemedicine: A Survey of Telecommunication Technologies, Developments, and Challenges. J. Sens. Actuator Netw., 12.","DOI":"10.3390\/jsan12020020"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1089\/15305620252933365","article-title":"Telemedicine and Health Care","volume":"8","author":"Bashshur","year":"2002","journal-title":"Telemed. J. E-Health"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s43162-023-00234-z","article-title":"The role of telemedicine in healthcare: An overview and update","volume":"35","author":"Stoltzfus","year":"2023","journal-title":"Egypt. J. Intern. Med."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Xiao, Y., Takahashi, D., and Hu, F. (2007, January 11\u201315). Telemedicine Usage and Potentials. Proceedings of the IEEE Wireless Communications and Networking Conference (WCNC), Hong Kong, China.","DOI":"10.1109\/WCNC.2007.508"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Shakor, M.Y., and Khaleel, M.I. (2024). Recent Advances in Big Medical Image Data Analysis Through Deep Learning and Cloud Computing. Electronics, 13.","DOI":"10.3390\/electronics13244860"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Wallauer, J., von Wangenheim, A., Andrade, R., and Macedo, D.D.J. (2008, January 17\u201319). A Telemedicine Network Using Secure Techniques and Intelligent User Access Control. Proceedings of the 21st IEEE International Symposium on Computer-Based Medical Systems (CBMS), Jyv\u00e4skyl\u00e4, Finland.","DOI":"10.1109\/CBMS.2008.124"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"5799","DOI":"10.1109\/OJCOMS.2024.3456549","article-title":"LLM-based edge intelligence: A comprehensive survey on architectures, applications, security and trustworthiness","volume":"5","author":"Friha","year":"2024","journal-title":"IEEE Open J. Commun. Soc."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1007\/s11019-021-10049-w","article-title":"Chatbot breakthrough in the 2020s? An ethical reflection on the trend of automated consultations in health care","volume":"25","author":"Parviainen","year":"2022","journal-title":"Med. Health Care Philos."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Nazi, Z.A., and Peng, W. (2024). Large Language Models in Healthcare and Medical Domain: A Review. Informatics, 11.","DOI":"10.3390\/informatics11030057"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Teixeira, B.N., Leit\u00e3o, A., Nascimento, G., Campos-Fernandes, A., and Cercas, F. (2024). Can ChatGPT Support Clinical Coding Using the ICD-10-CM\/PCS?. Informatics, 11.","DOI":"10.3390\/informatics11040084"},{"key":"ref_15","unstructured":"Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., and Gao, J. (2023). LLaVA\u2013Med: Training a Large Language\u2013and\u2013Vision Assistant for Biomedicine in One Day. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"De Souza In\u00e1cio, A., Andrade, R., von Wangenheim, A., and de Macedo, D.D.J. (2014, January 15\u201318). Designing an information retrieval system for the STT\/SC. Proceedings of the 2014 IEEE International Conference on e-Health Networking, Applications and Services (Healthcom), Natal, Brazil.","DOI":"10.1109\/HealthCom.2014.7001893"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"102848","DOI":"10.1016\/j.ijhcs.2022.102848","article-title":"Can We Trust a Chatbot like a Physician? A Qualitative Study on Understanding the Emergence of Trust toward Diagnostic Chatbots","volume":"165","author":"Seitz","year":"2022","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1111\/ans.18666","article-title":"Investigating the Impact of Innovative AI Chatbot on Post-Pandemic Medical Education and Clinical Assistance: A Comprehensive Analysis","volume":"94","author":"Xie","year":"2024","journal-title":"ANZ J. Surg."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"316","DOI":"10.58631\/ajemb.v2i8.55","article-title":"Managing healthcare product demand effectively in the post-COVID-19 environment: Navigating demand variability and forecasting complexities","volume":"2","author":"Niaz","year":"2023","journal-title":"Am. J. Econ. Manag. Bus. (AJEMB)"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"103158","DOI":"10.1016\/j.omega.2024.103158","article-title":"Sharing service in healthcare systems: A recent survey","volume":"129","author":"Wang","year":"2024","journal-title":"Omega"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1049\/iet-smc.2020.0044","article-title":"COVID-19 pandemic: A review of smart-city initiatives to face new outbreaks","volume":"2","author":"Costa","year":"2020","journal-title":"IET Smart Cities"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Pap, I.A., and Oniga, S. (2024). eHealth Assistant AI Chatbot Using a Large Language Model to Provide Personalized Answers through Secure Decentralized Communication. Sensors, 24.","DOI":"10.3390\/s24186140"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"14604582251315595","DOI":"10.1177\/14604582251315595","article-title":"A blueprint for large language model-augmented telehealth for HIV mitigation in Indonesia: A scoping review of a novel therapeutic modality","volume":"31","author":"Busch","year":"2024","journal-title":"Health Informatics J."},{"key":"ref_24","unstructured":"Bolton, E., Venigalla, A., Yasunaga, M., Hall, D., Xiong, B., Lee, T., Daneshjou, R., Frankle, J., Liang, P., and Carbin, M. (2024). BioMedLM: A 2.7B Parameter Language Model Trained on Biomedical Text. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","year":"2025","journal-title":"Nat. Med."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: A Pre-trained Biomedical Language Representation Model for Biomedical Text Mining","volume":"36","author":"Lee","year":"2019","journal-title":"Bioinformatics"},{"key":"ref_27","unstructured":"O\u2019Neil, M. (2025, September 01). The Inside Story of Babylon Health. Prospect Magazine. Available online: https:\/\/www.prospectmagazine.co.uk\/ideas\/technology\/40385\/the-inside-story-of-babylon-health."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Puel, A., Meurer, M.I., von Wangenheim, A., and Macedo, D.D.J. (2014, January 27\u201329). BUCOMAX: Collaborative Multimedia Platform for Real-Time Support of Diagnosis and Teaching Based on Bucomaxillofacial Diagnostic Images. Proceedings of the IEEE Symposium on Computer-Based Medical Systems (CBMS), New York, NY, USA.","DOI":"10.1109\/CBMS.2014.12"},{"key":"ref_29","unstructured":"Meta AI Team (2024, December 18). LLaMA 3: Advancing the State of Large Language Models with Efficiency and Scalability. Available online: https:\/\/ai.meta.com\/blog\/meta-llama-3\/."},{"key":"ref_30","unstructured":"Ollama Team (2024, December 18). Ollama: A Platform to Run and Customize Large Language Models Locally with Privacy and Performance. Available online: https:\/\/ollama.com\/."},{"key":"ref_31","unstructured":"Open WebUI Team (2024, December 18). Open WebUI is an Extensible, Self-Hosted AI Interface that Adapts to Your Workflow, All While Operating Entirely Offline. Available online: https:\/\/openwebui.com\/."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"114354","DOI":"10.1016\/j.dss.2024.114354","article-title":"Know where to go: Make LLM a relevant, responsible, and trustworthy searcher","volume":"188","author":"Shi","year":"2025","journal-title":"Decis. Support Syst."},{"key":"ref_33","unstructured":"(2025, October 20). Brazil. Lei n\u00b0 13.709, de 14 de Agosto de 2018\u2014Lei Geral de Prote\u00e7\u00e3o de Dados Pessoais (LGPD). Presid\u00eancia da Rep\u00fablica, Casa Civil, Subchefia para Assuntos Jur\u00eddicos, Available online: https:\/\/www.planalto.gov.br\/ccivil_03\/_ato2015-2018\/2018\/lei\/l13709.htm."},{"key":"ref_34","first-page":"485","article-title":"Significance of Epochs on Training a Neural Network","volume":"9","author":"Afaq","year":"2020","journal-title":"Int. J. Sci. Technol. Res."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. (2023). G-Eval: NLG Evaluation Using GPT-4 with Better Human Alignment. arXiv.","DOI":"10.18653\/v1\/2023.emnlp-main.153"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1001\/jama.2023.9651","article-title":"The challenges for regulating medical use of ChatGPT and other large language models","volume":"330","author":"Minssen","year":"2023","journal-title":"JAMA"}],"container-title":["Applied Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-3417\/15\/21\/11732\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,5]],"date-time":"2025-11-05T05:32:40Z","timestamp":1762320760000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-3417\/15\/21\/11732"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,3]]},"references-count":36,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2025,11]]}},"alternative-id":["app152111732"],"URL":"https:\/\/doi.org\/10.3390\/app152111732","relation":{},"ISSN":["2076-3417"],"issn-type":[{"value":"2076-3417","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,3]]}}}