{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T04:09:36Z","timestamp":1747282176071,"version":"3.40.5"},"publisher-location":"Cham","reference-count":43,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031897030","type":"print"},{"value":"9783031897047","type":"electronic"}],"license":[{"start":{"date-parts":[[2025,1,1]],"date-time":"2025-01-01T00:00:00Z","timestamp":1735689600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T00:00:00Z","timestamp":1747267200000},"content-version":"vor","delay-in-days":134,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>This research examines how five free of charge Large Language Models (LLMs)-Zephyr, Mistral 7B, LLAMA 2 7B, ChatGPT 3.5 and Copilot Precise-perform when faced with a nursing clinical scenario involving a neuropsychiatric emergency. Their responses were evaluated based on established guidelines by a Delphi consensus using a 5-point Likert scale to rate safety, accuracy, reliability and the potential for improvement. The findings underscore the greatest importance of safety and accuracy metrics. LLAMA 2 7B exhibits balanced but poor performance, scoring 3 out of 5 in Safety, Accuracy, and References, and 4 out of 5 in providing Improvement suggestions. ChatGPT 3.5 demonstrates adequate performance in Safety, Accuracy, and References, each with a score of 4 out of 5, indicating its proficiency in generating accurate, reliable content and ensuring patient safety, though there is room for improvement in enhancement suggestions (3 out of 5). Copilot Precise shows a unique profile, with balanced scores of 3 out of 5 in Safety, Accuracy, and Improvements, and a perfect score of 5 out of 5 only in References, highlighting its high accuracy in generating references. Reliability was reported in terms of both reference precision criteria and consistency over time computed through automated assessment. These preliminary results underscore the importance of developing language models that focus on ensuring safety and precision, in clinical decision-making scenarios. Further studies should aim to improve the accuracy and dependability of these models by examining a range of situations and incorporating real-time feedback mechanisms from experts. This will enhance their usefulness in clinical environments.\n<\/jats:p>","DOI":"10.1007\/978-3-031-89704-7_11","type":"book-chapter","created":{"date-parts":[[2025,5,14]],"date-time":"2025-05-14T11:28:12Z","timestamp":1747222092000},"page":"134-149","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Assessing and\u00a0Comparing Free Large Language Models\u2019 Responses to\u00a0a\u00a0Clinical Case: Accuracy, Safety, and\u00a0Reliability"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0331-7815","authenticated-orcid":false,"given":"Elena","family":"Sblendorio","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5581-4114","authenticated-orcid":false,"given":"Alessio","family":"Lo Cascio","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2481-5354","authenticated-orcid":false,"given":"Daniele","family":"Napolitano","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7120-7533","authenticated-orcid":false,"given":"Francesco","family":"Germini","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1148-332X","authenticated-orcid":false,"given":"Vincenzo","family":"Dentamaro","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5998-476X","authenticated-orcid":false,"given":"Michela","family":"Piredda","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2736-1792","authenticated-orcid":false,"given":"Giancarlo","family":"Cicolini","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,5,15]]},"reference":[{"key":"11_CR1","volume-title":"Deep Learning","author":"I Goodfellow","year":"2016","unstructured":"Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)"},{"key":"11_CR2","first-page":"1877","volume":"33","author":"T Brown","year":"2020","unstructured":"Brown, T., et al.: Language models are few-shot learners. Adv. Neural. Inf. Process. Syst. 33, 1877\u20131901 (2020)","journal-title":"Adv. Neural. Inf. Process. Syst."},{"key":"11_CR3","unstructured":"Topol, E.: Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again. Basic Books, New York (2019)"},{"key":"11_CR4","unstructured":"World Health Organization: Ethics and Governance of Artificial Intelligence for Health: WHO Guidance. World Health Organization (2022)"},{"issue":"3","key":"11_CR5","first-page":"146045822211123","volume":"28","author":"S Patel","year":"2022","unstructured":"Patel, S., Thakar, S., Esposito, A.: Leveraging AI for medical translation: challenges and opportunities. Health Inform. J. 28(3), 14604582221112364 (2022)","journal-title":"Health Inform. J."},{"key":"11_CR6","unstructured":"Miotto, R., Wang, F., Wang, S., Jiang, X., Dudley, J.T.: Deep learning for healthcare: review, opportunities and challenges. Brief. Bioinform. 24(2), bbx044 (2023)"},{"key":"11_CR7","first-page":"344","volume":"146","author":"Y Wang","year":"2023","unstructured":"Wang, Y., Kung, L., Byrd, T.A.: Big data analytics: understanding its capabilities and potential benefits for healthcare organizations. Technol. Forecast. Soc. Change 146, 344\u2013352 (2023)","journal-title":"Technol. Forecast. Soc. Change"},{"issue":"6","key":"11_CR8","volume":"24","author":"AB Kocaballi","year":"2022","unstructured":"Kocaballi, A.B., Laranjo, L., Coiera, E.: Measuring the impact of conversational interfaces on health communication: a systematic review. J. Med. Internet Res. 24(6), e37908 (2022)","journal-title":"J. Med. Internet Res."},{"issue":"2","key":"11_CR9","first-page":"12","volume":"47","author":"E Steinberg","year":"2023","unstructured":"Steinberg, E., Lee, K.S., Saleh, M.N.: The role of AI in clinical decision-making: challenges and future directions. J. Med. Syst. 47(2), 12 (2023)","journal-title":"J. Med. Syst."},{"issue":"4","key":"11_CR10","doi-asserted-by":"publisher","first-page":"689","DOI":"10.1007\/s11023-018-9482-5","volume":"28","author":"L Floridi","year":"2022","unstructured":"Floridi, L., et al.: AI4People-an ethical framework for a good AI society: opportunities, risks, principles, and recommendations. Minds Mach. 28(4), 689\u2013707 (2022)","journal-title":"Minds Mach."},{"issue":"1","key":"11_CR11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12910-021-00737-w","volume":"23","author":"D Leslie","year":"2022","unstructured":"Leslie, D., Holmes, D., Hitrova, C., Floridi, L.: Ethics of AI in health care: a mapping review. BMC Med. Ethics 23(1), 1\u201317 (2022)","journal-title":"BMC Med. Ethics"},{"issue":"10377","key":"11_CR12","doi-asserted-by":"publisher","first-page":"641","DOI":"10.1016\/S0140-6736(23)00216-7","volume":"401","author":"A Arora","year":"2023","unstructured":"Arora, A., Arora, A.: The promise of large language models in health care. Lancet 401(10377), 641 (2023)","journal-title":"Lancet"},{"key":"11_CR13","unstructured":"Wang, Y., Zhao, Y., Petzold, L.: Are large language models ready for healthcare? A comparative study on clinical language understanding. In: Machine Learning Healthcare Conference, pp. 804\u2013823 (2023)"},{"key":"11_CR14","unstructured":"Umerenkov, D., Zubkova, G., Nesterov, A.: Deciphering diagnoses: how large language models explanations influence clinical decision making. arXiv preprint arXiv:2310.01708 (2023)"},{"issue":"9","key":"11_CR15","doi-asserted-by":"publisher","DOI":"10.1016\/j.jacadv.2023.100658","volume":"2","author":"PC Lee","year":"2023","unstructured":"Lee, P.C., Sharma, S.K., Motaganahalli, S., Huang, A.: Evaluating the clinical decision-making ability of large language models using MKSAP-19 cardiology questions. JACC Adv. 2(9), 100658 (2023)","journal-title":"JACC Adv."},{"key":"11_CR16","doi-asserted-by":"crossref","unstructured":"Gottlieb, S., Silvis, L.: How to safely integrate large language models into health care. JAMA Health Forum 4(9), e233909\u2013e233909 (2023)","DOI":"10.1001\/jamahealthforum.2023.3909"},{"key":"11_CR17","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1016\/j.jbi.2018.10.005","volume":"88","author":"S Velupillai","year":"2018","unstructured":"Velupillai, S., et al.: Using clinical natural language processing for health outcomes research: overview and actionable suggestions for future advances. J. Biomed. Inform. 88, 11\u201319 (2018)","journal-title":"J. Biomed. Inform."},{"issue":"10","key":"11_CR18","doi-asserted-by":"publisher","first-page":"e2335924","DOI":"10.1001\/jamanetworkopen.2023.35924","volume":"6","author":"RH Perlis","year":"2023","unstructured":"Perlis, R.H., Fihn, S.D.: Evaluating the application of large language models in clinical research contexts. JAMA Netw. Open 6(10), e2335924\u2013e2335924 (2023)","journal-title":"JAMA Netw. Open"},{"issue":"2","key":"11_CR19","doi-asserted-by":"publisher","first-page":"51","DOI":"10.35680\/2372-0247.1276","volume":"6","author":"T Turpen","year":"2019","unstructured":"Turpen, T., Matthews, L., Guney, C.: Beneath the surface of talking about physicians: a statistical model of language for patient experience comments. Patient Exp. J. 6(2), 51\u201358 (2019)","journal-title":"Patient Exp. J."},{"issue":"1","key":"11_CR20","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1177\/13670069211022851","volume":"26","author":"S Hayakawa","year":"2022","unstructured":"Hayakawa, S., Pan, Y., Marian, V.: Language changes medical judgments and beliefs. Int. J. Biling. 26(1), 104\u2013121 (2022)","journal-title":"Int. J. Biling."},{"key":"11_CR21","doi-asserted-by":"crossref","unstructured":"Hochberg, L., et al.: Towards automatic annotation of clinical decision-making style. In: Proceedings of the LAW VIII-The 8th Linguistic Annotation Workshop, pp. 129\u2013138 (2014)","DOI":"10.3115\/v1\/W14-4919"},{"key":"11_CR22","unstructured":"U.S. Food and Drug Administration: Valium (diazepam) medication guide (2020). https:\/\/www.accessdata.fda.gov\/drugsatfda_docs\/label\/2020\/013263s094lbl.pdf. Accessed 14 May 2024"},{"key":"11_CR23","unstructured":"Hikma Pharmaceuticals USA: Flumazenil Injection, USP. MedLibrary.org. https:\/\/medlibrary.org\/lib\/rx\/meds\/flumazenil-19\/. Accessed 15 May 2024"},{"key":"11_CR24","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1007\/s00228-020-03031-7","volume":"77","author":"AS Razavizadeh","year":"2021","unstructured":"Razavizadeh, A.S., Zamani, N., Ziaeefar, P., Ebrahimi, S., Hassanian-Moghaddam, H.: Protective effect of flumazenil infusion in severe acute benzodiazepine toxicity: a pilot randomized trial. Eur. J. Clin. Pharmacol. 77, 547\u2013554 (2021)","journal-title":"Eur. J. Clin. Pharmacol."},{"key":"11_CR25","doi-asserted-by":"publisher","DOI":"10.1016\/j.drugalcdep.2022.109501","volume":"236","author":"T MacDonald","year":"2022","unstructured":"MacDonald, T., Gallo, A., Basso-Hulse, G., Bennett, K., Hulse, G.K.: A double-blind randomised crossover trial of low-dose flumazenil for benzodiazepine withdrawal: a proof of concept. Drug Alcohol Depend. 236, 109501 (2022)","journal-title":"Drug Alcohol Depend."},{"issue":"3","key":"11_CR26","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1177\/0269881120981390","volume":"35","author":"AT Gallo","year":"2021","unstructured":"Gallo, A.T., Hulse, G.: Pharmacological uses of flumazenil in benzodiazepine use disorders: a systematic review of limited data. J. Psychopharmacol. 35(3), 211\u2013220 (2021)","journal-title":"J. Psychopharmacol."},{"issue":"1","key":"11_CR27","doi-asserted-by":"publisher","first-page":"17","DOI":"10.5811\/westjem.2011.9.6864","volume":"13","author":"JS Richmond","year":"2012","unstructured":"Richmond, J.S., et al.: Verbal de-escalation of the agitated patient: consensus statement of the American association for emergency psychiatry project BETA de-escalation workgroup. West. J. Emerg. Med. 13(1), 17\u201325 (2012)","journal-title":"West. J. Emerg. Med."},{"issue":"3","key":"11_CR28","doi-asserted-by":"publisher","first-page":"314","DOI":"10.1016\/j.annemergmed.2017.05.021","volume":"71","author":"RC Dart","year":"2018","unstructured":"Dart, R.C., et al.: Expert consensus guidelines for stocking of antidotes in hospitals that provide emergency care. Ann. Emerg. Med. 71(3), 314\u2013325 (2018)","journal-title":"Ann. Emerg. Med."},{"key":"11_CR29","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12913-021-07097-6","volume":"21","author":"M Hussein","year":"2021","unstructured":"Hussein, M., Pavlova, M., Ghalwash, M., Groot, W.: The impact of hospital accreditation on the quality of healthcare: a systematic literature review. BMC Health Serv. Res. 21, 1\u201312 (2021)","journal-title":"BMC Health Serv. Res."},{"key":"11_CR30","volume":"16","author":"S Pohl","year":"2022","unstructured":"Pohl, S., Battistelli, A., Djediat, A., Andela, M.: Emotional support at work: a key component for nurses\u2019 work engagement, their quality of care and their organizational citizenship behaviour. Int. J. Afr. Nurs. Sci. 16, 100424 (2022)","journal-title":"Int. J. Afr. Nurs. Sci."},{"issue":"3","key":"11_CR31","first-page":"120","volume":"41","author":"S Chiappinotto","year":"2022","unstructured":"Chiappinotto, S., Palese, A., Longhini, J., et al.: Le videochiamate tra pazienti e familiari: Una revisione narrativa. Assist. Inferm. Ric. 41(3), 120\u2013128 (2022)","journal-title":"Assist. Inferm. Ric."},{"issue":"6","key":"11_CR32","doi-asserted-by":"publisher","first-page":"262","DOI":"10.3928\/00220124-20071101-03","volume":"38","author":"CA LaSala","year":"2007","unstructured":"LaSala, C.A., Connors, P.M., Pedro, J.T., Phipps, M.: The role of the clinical nurse specialist in promoting evidence-based practice and effecting positive patient outcomes. J. Contin. Educ. Nurs. 38(6), 262\u2013270 (2007)","journal-title":"J. Contin. Educ. Nurs."},{"issue":"3","key":"11_CR33","doi-asserted-by":"publisher","first-page":"172","DOI":"10.12968\/bjon.2017.26.3.172","volume":"26","author":"P Copanitsanou","year":"2017","unstructured":"Copanitsanou, P., Fotos, N., Brokalaki, H.: Effects of work environment on patient and nurse outcomes. Br. J. Nurs. 26(3), 172\u2013176 (2017)","journal-title":"Br. J. Nurs."},{"issue":"4","key":"11_CR34","first-page":"306","volume":"111","author":"F Zaghini","year":"2020","unstructured":"Zaghini, F.: The influence of work context and organizational well-being on psychophysical health of healthcare providers. Med. Lav. 111(4), 306 (2020)","journal-title":"Med. Lav."},{"key":"11_CR35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12913-019-4667-z","volume":"19","author":"A Palese","year":"2019","unstructured":"Palese, A., et al.: A path analysis on the direct and indirect effects of the unit environment on eating dependence among cognitively impaired nursing home residents. BMC Health Serv. Res. 19, 1\u201314 (2019)","journal-title":"BMC Health Serv. Res."},{"key":"11_CR36","first-page":"5","volume":"4","author":"E Sblendorio","year":"2023","unstructured":"Sblendorio, E., et al.: Assessment of stress levels using technological tools: a review and prospective analysis of heart rate variability and sleep quality parameters. Neurodegener Dis 4, 5 (2023)","journal-title":"Neurodegener Dis"},{"issue":"8","key":"11_CR37","doi-asserted-by":"publisher","first-page":"639","DOI":"10.9734\/BJMMR\/2015\/17192","volume":"8","author":"W Verrusio","year":"2015","unstructured":"Verrusio, W., Moscucci, F., Cacciafesta, M., Gueli, N.: Mozart effect and its clinical applications: a review. Br. J. Med. Med. Res. 8(8), 639\u2013650 (2015)","journal-title":"Br. J. Med. Med. Res."},{"key":"11_CR38","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12913-021-06341-3","volume":"21","author":"R Gualandi","year":"2021","unstructured":"Gualandi, R., Masella, C., Piredda, M., Ercoli, M., Tartaglini, D.: What does the patient have to say? Valuing the patient experience to improve the patient journey. BMC Health Serv. Res. 21, 1\u201312 (2021)","journal-title":"BMC Health Serv. Res."},{"key":"11_CR39","unstructured":"Tunstall, L., et al.: Zephyr: direct distillation of LM alignment. arXiv preprint arXiv:2310.16944 (2023)"},{"key":"11_CR40","unstructured":"Jiang, A.Q., et al.: Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)"},{"key":"11_CR41","unstructured":"Touvron, H., et al.: Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)"},{"key":"11_CR42","doi-asserted-by":"crossref","unstructured":"Abdullah, M., Madain, A., Jararweh, Y.: ChatGPT: fundamentals, applications and social impacts. In: Proceedings of the SNAMS, pp. 1\u20136 (2022)","DOI":"10.1109\/SNAMS58071.2022.10062688"},{"key":"11_CR43","doi-asserted-by":"crossref","unstructured":"Ormerod, M., Mart\u00ednez del Rinc\u00f3n, J., Devereux, B.: Predicting semantic similarity between clinical sentence pairs using transformer models: evaluation and representational analysis. JMIR Med. Inform. 9(5) (2021)","DOI":"10.2196\/23099"}],"container-title":["Lecture Notes in Computer Science","Computational Intelligence Methods for Bioinformatics and Biostatistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-89704-7_11","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,14]],"date-time":"2025-05-14T11:28:19Z","timestamp":1747222099000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-89704-7_11"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025]]},"ISBN":["9783031897030","9783031897047"],"references-count":43,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-89704-7_11","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025]]},"assertion":[{"value":"15 May 2025","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"The authors declare no conflicts of interest.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interests"}},{"value":"CIBB","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Meeting on Computational Intelligence Methods for Bioinformatics and Biostatistics","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Benevento","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Italy","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"4 September 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"6 September 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"19","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"cibb2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"http:\/\/cibb2024.unisannio.it","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}