{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T16:00:58Z","timestamp":1773072058304,"version":"3.50.1"},"reference-count":22,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T00:00:00Z","timestamp":1773014400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Accessing and understanding medication information can be challenging for many people, especially when patient information leaflets (PILs) are long, complex, and printed in small font. This study presents MediChat, an open-source, locally executable chatbot designed to provide reliable, easy-to-read answers to medication-related questions based exclusively on official PILs. MediChat follows a retrieval-augmented generation (RAG) architecture: PILs from the Austrian Medicinal Product Index are received via API, converted to text, split into overlapping chunks, embedded, and stored in a Chroma vector database. From there the top-k relevant chunks are retrieved, and Llama 3.1 generates German responses based on this evidence. The system was evaluated using a hybrid framework. Quantitatively, 200 yes\/no questions across ten drugs were answered with 80% accuracy, overall precision 0.977, recall 0.686, F1-score 0.806, and a mean response time of 727 ms. Qualitatively, two personas were used in eight simulated dialogues. Response times were around 1.1\u20131.3 s, and task completion exceeded 85% with high ratings for relevance and quantity. These results indicate that an open-source RAG chatbot can deliver leaflet-grounded, user-friendly medication information and provide a reproducible template for future healthcare chatbot evaluations.<\/jats:p>","DOI":"10.3390\/fi18030139","type":"journal-article","created":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T10:16:12Z","timestamp":1773051372000},"page":"139","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Implementation and Evaluation of an Open-Source Chatbot for Patient Information Leaflets"],"prefix":"10.3390","volume":"18","author":[{"given":"Lisa","family":"Heiler","sequence":"first","affiliation":[{"name":"Institute of eHealth, University of Applied Sciences\u2014FH Joanneum, 8020 Graz, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Katharina","family":"Kirchsteiger","sequence":"additional","affiliation":[{"name":"Institute of eHealth, University of Applied Sciences\u2014FH Joanneum, 8020 Graz, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3833-4252","authenticated-orcid":false,"given":"Sten","family":"Hanke","sequence":"additional","affiliation":[{"name":"Institute of eHealth, University of Applied Sciences\u2014FH Joanneum, 8020 Graz, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6018-7821","authenticated-orcid":false,"given":"Markus","family":"B\u00f6denler","sequence":"additional","affiliation":[{"name":"Institute of eHealth, University of Applied Sciences\u2014FH Joanneum, 8020 Graz, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,3,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1053","DOI":"10.1093\/eurpub\/ckv043","article-title":"Health Literacy in Europe: Comparative Results of the European Health Literacy Survey (HLS-EU)","volume":"25","author":"Pelikan","year":"2015","journal-title":"Eur. J. Public Health"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1372","DOI":"10.1007\/s11096-016-0376-4","article-title":"Low Literacy and Written Drug Information: Information-Seeking, Leaflet Evaluation and Preferences, and Roles for Images","volume":"38","author":"Bos","year":"2016","journal-title":"Int. J. Clin. Pharm."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"e2200005","DOI":"10.1200\/CCI.22.00005","article-title":"Health Literacy and EHealth: Challenges and Strategies","volume":"6","author":"Hasannejadasl","year":"2022","journal-title":"JCO Clin. Cancer Inform."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1145\/3641289","article-title":"A Survey on Evaluation of Large Language Models","volume":"15","author":"Chang","year":"2024","journal-title":"ACM Trans. Intell. Syst. Technol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1850","DOI":"10.1039\/D3DD00112A","article-title":"Domain-Specific Chatbots for Science Using Embeddings","volume":"2","author":"Yager","year":"2023","journal-title":"Digit. Discov."},{"key":"ref_6","first-page":"111","article-title":"Open Data Chatbot","volume":"Volume 11762 LNCS","author":"Keyner","year":"2019","journal-title":"Proceedings of the Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Fern\u00e1ndez-Mart\u00ednez, F., Luna-Jim\u00e9nez, C., Kleinlein, R., Griol, D., Callejas, Z., and Montero, J.M. (2022). Fine-Tuning BERT Models for Intent Recognition Using a Frequency Cut-Off Strategy for Domain-Specific Vocabulary Extension. Appl. Sci., 12.","DOI":"10.3390\/app12031610"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"ElSayed, H.S., Hussien, M.T., Salah, E.M., Farouk, A.K., Ahmed, A.T., El-Hafez, M.M.A., and Salah, K. (2023). Chatbot as a Virtual Assistant to Retrieve Information from Datasheets Using Memory Controllers Domain Knowledge. Proceedings of the 2023 30th IEEE International Conference on Electronics, Circuits and Systems (ICECS), IEEE.","DOI":"10.1109\/ICECS58634.2023.10382771"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"da Silveira Colombo, C., Badue, C., and de Oliveira, E. (2023). PharmaBulaBot: A Chatbot to Answer Drug Questions Based on Information from Pharmaceutical Package Inserts. ACM International Conference Proceeding Series, ACM.","DOI":"10.1145\/3592813.3592890"},{"key":"ref_10","first-page":"9459","article-title":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"631","DOI":"10.4103\/ija.ija_204_24","article-title":"Comparing Patient Education Tools for Chronic Pain Medications: Artificial Intelligence Chatbot versus Traditional Patient Information Leaflets","volume":"68","author":"Gondode","year":"2024","journal-title":"Indian J. Anaesth."},{"key":"ref_12","unstructured":"Austrian Federal Office for Safety in Health Care (BASG) (2024, November 09). Austrian Medicinal Product Index. Available online: https:\/\/aspregister.basg.gv.at\/aspregister\/faces\/aspregister.jspx."},{"key":"ref_13","unstructured":"PHARMIG\u2014Verband der Pharmazeutischen Industrie \u00d6sterreichs (2026, January 31). PHARMIG Facts & Figures. Available online: https:\/\/www.pharmig.at\/mediathek\/publikationen\/."},{"key":"ref_14","first-page":"22","article-title":"Large Language Model-Based Evaluation of Medical Question Answering Systems: Algorithm Development and Case Study","volume":"313","author":"Reichenpfader","year":"2024","journal-title":"Stud. Health Technol. Inform."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"102110","DOI":"10.1016\/j.japh.2024.102110","article-title":"Accuracy of a Chatbot in Answering Questions That Patients Should Ask before Taking a New Medication","volume":"64","author":"Cornelison","year":"2024","journal-title":"J. Am. Pharm. Assoc."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1055\/s-0041-1736664","article-title":"Evaluation Metrics for Health Chatbots: A Delphi Study","volume":"60","author":"Denecke","year":"2021","journal-title":"Methods Inf. Med."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"e18301","DOI":"10.2196\/18301","article-title":"Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review","volume":"22","author":"Safi","year":"2020","journal-title":"J. Med. Internet Res."},{"key":"ref_18","unstructured":"Peras, D. (2018). Chatbot Evaluation Metrics: Review Paper. Proceedings of the Economic and Social Development, Varazdin Development and Entrepreneurship Agency."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"105004","DOI":"10.1016\/j.compbiomed.2021.105004","article-title":"A Conversational Agent for Querying Italian Patient Information Leaflets and Improving Health Literacy","volume":"141","author":"Minutolo","year":"2022","journal-title":"Comput. Biol. Med."},{"key":"ref_20","unstructured":"Corcoran, P.F. (2023). An Open Source Chatbot for a Complex Domain. [Master\u2019s Thesis, University College Dublin]."},{"key":"ref_21","first-page":"761","article-title":"Integrating AI-Generated Summaries of Medication Leaflets into a Mobile Application for Medication Management","volume":"316","author":"Heiler","year":"2024","journal-title":"Stud. Health Technol. Inform."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"117310","DOI":"10.1016\/j.measurement.2025.117310","article-title":"A New Method for Long-Term Temperature Compensation of Structural Health Monitoring by Ultrasonic Guided Wave","volume":"252","author":"Liu","year":"2025","journal-title":"Measurement"}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/18\/3\/139\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T10:32:23Z","timestamp":1773052343000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/18\/3\/139"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,9]]},"references-count":22,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2026,3]]}},"alternative-id":["fi18030139"],"URL":"https:\/\/doi.org\/10.3390\/fi18030139","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,9]]}}}