{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T16:39:15Z","timestamp":1775839155231,"version":"3.50.1"},"reference-count":34,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T00:00:00Z","timestamp":1731974400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T00:00:00Z","timestamp":1731974400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Hum-Cent Intell Syst"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The increasing adoption of large language models (LLMs) in healthcare presents both opportunities and challenges. While LLM-powered applications are being utilized for various medical tasks, concerns persist regarding their accuracy and reliability, particularly when not specifically trained on medical data. Using open-source models without proper fine-tuning for medical applications can lead to inaccurate or potentially harmful advice, underscoring the need for domain-specific adaptation. Therefore, this study addresses these issues by developing PharmaLLM, a fine-tuned version of the open-source Llama 2 model, designed to provide accurate medicine prescription information. PharmaLLM incorporates a multi-modal input\/output mechanism, supporting both text and speech, to enhance accessibility. The fine-tuning process utilized LoRA (Low-Rank Adaptation) with a rank of 16 for parameter-efficient fine-tuning. The learning rate was maintained at 2e-4 for stable adjustments, and a batch size of 12 was chosen to balance computational efficiency and learning effectiveness. The system demonstrated strong performance metrics, achieving 87% accuracy, 92.16% F1 score, 94% sensitivity, 66% specificity, and 90% precision. A usability study involving 33 participants was conducted to evaluate the system using the Chatbot Usability Questionnaire, focusing on error handling, response generation, navigation, and personality. Results from the questionnaire indicated that participants found the system easy to navigate and the responses useful and relevant. PharmaLLM aims to facilitate improved patient-physician interactions, particularly in areas with limited healthcare resources and low literacy rates. This research contributes to the advancement of medical informatics by offering a reliable, accessible web-based tool that benefits both patients and healthcare providers.<\/jats:p>","DOI":"10.1007\/s44230-024-00085-z","type":"journal-article","created":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T16:56:54Z","timestamp":1732035414000},"page":"527-544","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["PharmaLLM: A Medicine Prescriber Chatbot Exploiting Open-Source Large Language Models"],"prefix":"10.1007","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5853-8506","authenticated-orcid":false,"given":"Ayesha","family":"Azam","sequence":"first","affiliation":[]},{"given":"Zubaira","family":"Naz","sequence":"additional","affiliation":[]},{"given":"Muhammad Usman Ghani","family":"Khan","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,11,19]]},"reference":[{"key":"85_CR1","doi-asserted-by":"publisher","unstructured":"Al Nazi Z, Peng W, Large Language models in healthcare and medical domain: a review, Informatics 2024, 11(3): 57; 2024, https:\/\/doi.org\/10.3390\/INFORMATICS11030057.","DOI":"10.3390\/INFORMATICS11030057"},{"key":"85_CR2","doi-asserted-by":"publisher","DOI":"10.1016\/J.MLWA.2020.100006","volume":"2","author":"E Adamopoulou","year":"2020","unstructured":"Adamopoulou E, Moussiades L. Chatbots: History, technology, and applications. Mach Learn Appl. 2020;2: 100006. https:\/\/doi.org\/10.1016\/J.MLWA.2020.100006.","journal-title":"Mach Learn Appl."},{"key":"85_CR3","doi-asserted-by":"publisher","unstructured":"Prof. Jadhav P, Samnani A, Alachiya A, Shah V, Selvam A Intelligent Chatbot, International Journal of Advanced Research in Science, Communication and Technology, pp. 679\u2013683, May 2022, https:\/\/doi.org\/10.48175\/IJARSCT-3996.","DOI":"10.48175\/IJARSCT-3996"},{"key":"85_CR4","unstructured":"Vaswani A et al. Attention Is All You Need, Adv Neural Inf Process Syst, vol. 2017-December, pp. 5999\u20136009, Jun. 2017, Accessed: Oct. 11, 2024. [Online]. https:\/\/arxiv.org\/abs\/1706.03762v7"},{"key":"85_CR5","doi-asserted-by":"publisher","unstructured":"Borsci S, Schmettow M, Malizia A, Chamberlain A, Frank Van Der Velde, A confirmatory factorial analysis of the Chatbot Usability Scale: a multilanguage validation, Pers Ubiquitous Comput 1: 3, https:\/\/doi.org\/10.1007\/s00779-022-01690-0.","DOI":"10.1007\/s00779-022-01690-0"},{"issue":"1","key":"85_CR6","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/S12966-021-01224-6\/TABLES\/4","volume":"18","author":"YJ Oh","year":"2021","unstructured":"Oh YJ, Zhang J, Fang ML, Fukuoka Y. A systematic review of artificial intelligence chatbots for promoting physical activity, healthy diet, and weight loss. Int J Behav Nutr Phys Act. 2021;18(1):1\u201325. https:\/\/doi.org\/10.1186\/S12966-021-01224-6\/TABLES\/4.","journal-title":"Int J Behav Nutr Phys Act"},{"key":"85_CR7","doi-asserted-by":"publisher","DOI":"10.1016\/J.IJMEDINF.2022.104827","volume":"165","author":"IC Chang","year":"2022","unstructured":"Chang IC, Shih YS, Kuo KM. Why would you use medical chatbots? interview and survey. Int J Med Inform. 2022;165: 104827. https:\/\/doi.org\/10.1016\/J.IJMEDINF.2022.104827.","journal-title":"Int J Med Inform"},{"key":"85_CR8","doi-asserted-by":"publisher","unstructured":"Eleonor Comendador BV, Michael Francisco BB, Medenilla JS, Mae Nacion ST, Bryle Serac TE Pharmabot: a pediatric generic medicine consultant chatbot, https:\/\/doi.org\/10.12720\/joace.3.2.137-140.","DOI":"10.12720\/joace.3.2.137-140"},{"issue":"1","key":"85_CR9","doi-asserted-by":"publisher","first-page":"41","DOI":"10.14445\/22312803\/IJCTT-V60P106","volume":"60","author":"R Dharwadkar","year":"2018","unstructured":"Dharwadkar R, Deshpande NA. A Medical ChatBot. Int J Comput Trends Technol. 2018;60(1):41\u20135. https:\/\/doi.org\/10.14445\/22312803\/IJCTT-V60P106.","journal-title":"Int J Comput Trends Technol"},{"key":"85_CR10","unstructured":"Prasad GK Satvik Ranjan C Assistant Professor UG Student and T. Ankit Vivek Kumar, A Personalized Medical Assistant Chatbot: MediBot, IJSTE-Int J Sci Technol Eng 5, 2019, Accessed: Jun. 27, 2024. [Online]. www.ijste.org"},{"key":"85_CR11","doi-asserted-by":"crossref","unstructured":"Kandpal P, Jasnani K, Raut R, Bhorge S Contextual chatbot for healthcare purposes (using deep learning), Proceedings of the World Conference on Smart Trends in Systems, Security and Sustainability, WS4 2020, pp. 625\u2013634, 2020, 10.1109\/WORLDS450073.2020.9210351.","DOI":"10.1109\/WorldS450073.2020.9210351"},{"key":"85_CR12","doi-asserted-by":"publisher","unstructured":"Vamsi GK, Rasool A, Hajela G Chatbot: a deep neural network based human to machine conversation model, 2020 11th International Conference on Computing, Communication and Networking Technologies, ICCCNT 2020, Jul. 2020, https:\/\/doi.org\/10.1109\/ICCCNT49239.2020.9225395.","DOI":"10.1109\/ICCCNT49239.2020.9225395"},{"key":"85_CR13","doi-asserted-by":"publisher","unstructured":"Shinde NV, Akhade A, Bagad P, Bhavsar H, Wagh SK, Kamble A, Healthcare Chatbot System using Artificial Intelligence, Proceedings of the 5th International Conference on Trends in Electronics and Informatics, ICOEI 2021, pp. 1174\u20131181, Jun. 2021, https:\/\/doi.org\/10.1109\/ICOEI51242.2021.9452902.","DOI":"10.1109\/ICOEI51242.2021.9452902"},{"key":"85_CR14","doi-asserted-by":"publisher","unstructured":"Badlani S, Aditya T, Dave M, Chaudhari S, Multilingual healthcare chatbot using machine learning, 2021 2nd International Conference for Emerging Technology, INCET 2021, May 2021, https:\/\/doi.org\/10.1109\/INCET51464.2021.9456304.","DOI":"10.1109\/INCET51464.2021.9456304"},{"issue":"17","key":"85_CR15","doi-asserted-by":"publisher","first-page":"23777","DOI":"10.1007\/S11042-022-12820-4\/TABLES\/4","volume":"81","author":"IC Hsu","year":"2022","unstructured":"Hsu IC, De Yu J. A medical Chatbot using machine learning and natural language understanding. Multimed Tools Appl. 2022;81(17):23777\u201399. https:\/\/doi.org\/10.1007\/S11042-022-12820-4\/TABLES\/4.","journal-title":"Multimed Tools Appl"},{"key":"85_CR16","doi-asserted-by":"publisher","first-page":"128469","DOI":"10.1109\/ACCESS.2022.3227208","volume":"10","author":"S Chakraborty","year":"2022","unstructured":"Chakraborty S, et al. An ai-based medical chatbot model for infectious disease prediction. IEEE Access. 2022;10:128469\u201383. https:\/\/doi.org\/10.1109\/ACCESS.2022.3227208.","journal-title":"IEEE Access"},{"key":"85_CR17","doi-asserted-by":"publisher","unstructured":"Rajani G, Ruparel K Deep learning based chatbot architecture for medical diagnosis and treatment recommendation, Proceedings of 3rd International Conference on Advanced Computing Technologies and Applications, ICACTA 2023, 2023, https:\/\/doi.org\/10.1109\/ICACTA58201.2023.10392702.","DOI":"10.1109\/ICACTA58201.2023.10392702"},{"key":"85_CR18","doi-asserted-by":"publisher","unstructured":"Dohare P, Johri S, Priya S, Singh S, Upadhyay S, Good Fellow\u202f: A Healthcare Chatbot System, 2023 14th International Conference on Computing Communication and Networking Technologies, ICCCNT 2023, 2023, https:\/\/doi.org\/10.1109\/ICCCNT56998.2023.10308009.","DOI":"10.1109\/ICCCNT56998.2023.10308009"},{"key":"85_CR19","doi-asserted-by":"publisher","unstructured":"Rao DR, Thottempudi K, Surla BK, Satapathy A, Design and evaluation of a medical chatbot built on BERT language model for remote health assistance, 2024 3rd International Conference for Innovation in Technology, INOCON 2024, 2024, https:\/\/doi.org\/10.1109\/INOCON60754.2024.10511493.","DOI":"10.1109\/INOCON60754.2024.10511493"},{"key":"85_CR20","doi-asserted-by":"publisher","DOI":"10.1016\/J.RCSOP.2024.100419","volume":"13","author":"A Babu","year":"2024","unstructured":"Babu A, Boddu SB. BERT-based medical Chatbot: enhancing healthcare communication through natural language understanding. Exploratory Res Clin Soc Pharmacy. 2024;13: 100419. https:\/\/doi.org\/10.1016\/J.RCSOP.2024.100419.","journal-title":"Exploratory Res Clin Soc Pharmacy"},{"key":"85_CR21","unstructured":"EDA | 11,000 Medicines. Accessed: Jun. 27, 2024. [Online]. https:\/\/www.kaggle.com\/code\/sasakitetsuya\/eda-11-000-medicines"},{"key":"85_CR22","unstructured":"Zhang P, Zeng G, Wang T, Lu W TinyLlama: an open-source small language model, Jan. 2024, Accessed: Oct. 11, 2024. [Online]. https:\/\/arxiv.org\/abs\/2401.02385v2"},{"key":"85_CR23","doi-asserted-by":"publisher","unstructured":"Souza JD, A. Intelligence, A review of transformer models, https:\/\/doi.org\/10.48366\/r640001.","DOI":"10.48366\/r640001"},{"key":"85_CR24","unstructured":"Touvron H et al. LLaMA: Open and Efficient Foundation Language Models, 2023, Accessed: Oct. 11, 2024. [Online]. https:\/\/arxiv.org\/abs\/2302.13971v1"},{"key":"85_CR25","unstructured":"Fang Tan T et al. Fine-tuning Large Language Model (LLM) Artificial Intelligence Chatbots in Ophthalmology and LLM-based evaluation using GPT-4, Feb. 2024, Accessed: Jun. 27, 2024. [Online]. https:\/\/arxiv.org\/abs\/2402.10083v1"},{"key":"85_CR26","unstructured":"N. S. Google, GLU Variants Improve Transformer, Feb. 2020, Accessed: Jun. 27, 2024. [Online]. https:\/\/arxiv.org\/abs\/2002.05202v1"},{"key":"85_CR27","doi-asserted-by":"publisher","DOI":"10.1016\/J.NEUCOM.2023.127063","volume":"568","author":"J Su","year":"2024","unstructured":"Su J, Ahmed M, Lu Y, Pan S, Bo W, Liu Y. RoFormer: enhanced transformer with rotary position embedding. Neurocomputing. 2024;568: 127063. https:\/\/doi.org\/10.1016\/J.NEUCOM.2023.127063.","journal-title":"Neurocomputing"},{"key":"85_CR28","doi-asserted-by":"publisher","unstructured":"Ainslie J, Lee-Thorp J, de Jong M, Zemlyanskiy Y, Lebr\u00f3n F, Sanghai S GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, EMNLP 2023 - 2023 Conference on Empirical Methods in Natural Language Processing, Proceedings, pp. 4895\u20134901, May 2023, https:\/\/doi.org\/10.18653\/v1\/2023.emnlp-main.298.","DOI":"10.18653\/v1\/2023.emnlp-main.298"},{"key":"85_CR29","unstructured":"Zhang B, Sennrich R Root mean square layer normalization, Adv Neural Inf Process Syst, 32, Oct. 2019, Accessed: Jun. 27, 2024. [Online]. https:\/\/arxiv.org\/abs\/1910.07467v1"},{"key":"85_CR30","doi-asserted-by":"publisher","unstructured":"Holmes S, Moorhead A, Bond R, Zheng H, Coates V, McTear M Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces?, ECCE 2019 - Proceedings of the 31st European Conference on Cognitive Ergonomics: \u201c\u2018Design for Cognition,\u2019\u201d pp. 207\u2013214, Sep. 2019, https:\/\/doi.org\/10.1145\/3335082.3335094.","DOI":"10.1145\/3335082.3335094"},{"key":"85_CR31","unstructured":"Zheng L et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Jun. 2023, Accessed: Jun. 27, 2024. [Online]. https:\/\/arxiv.org\/abs\/2306.05685v4"},{"key":"85_CR32","unstructured":"Devlin J, Chang MW, Lee K, Toutanova K BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, vol. 1, pp. 4171\u20134186, Oct. 2018, Accessed: Jun. 27, 2024. [Online]. https:\/\/arxiv.org\/abs\/1810.04805v2"},{"key":"85_CR33","unstructured":"Jia F et al. OncoGPT: A Medical Conversational Model Tailored with Oncology Domain Expertise on a Large Language Model Meta-AI (LLaMA), Feb. 2024, Accessed: Jun. 27, 2024. [Online]. https:\/\/arxiv.org\/abs\/2402.16810v1"},{"key":"85_CR34","unstructured":"Abbasian M, Azimi I, Rahmani AM, Jain R, Conversational Health Agents: A Personalized LLM-Powered Agent Framework, Oct. 2023, Accessed: Jun. 27, 2024. [Online]. https:\/\/arxiv.org\/abs\/2310.02374v4"}],"container-title":["Human-Centric Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44230-024-00085-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44230-024-00085-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44230-024-00085-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,8]],"date-time":"2025-01-08T15:15:07Z","timestamp":1736349307000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44230-024-00085-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,19]]},"references-count":34,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["85"],"URL":"https:\/\/doi.org\/10.1007\/s44230-024-00085-z","relation":{},"ISSN":["2667-1336"],"issn-type":[{"value":"2667-1336","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,19]]},"assertion":[{"value":"2 August 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 October 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 November 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}]}}