{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,1]],"date-time":"2026-09-01T20:19:19Z","timestamp":1788293959836,"version":"build-2803163510"},"reference-count":44,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T00:00:00Z","timestamp":1781568000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JCM"],"abstract":"<jats:p>Background: Chatbots are becoming increasingly valuable in clinical settings, offering rapid access to medical information, aiding documentation, and improving perioperative patient education. Their adaptability makes them promising tools for personalized perioperative risk stratification (PRS) and anesthesia planning, but their definitive role remains uncertain. We aimed to evaluate chatbot performance in PRS compared to standard clinical judgment and to assess the certainty of the evidence supporting their use. Methods: This systematic review (PROSPERO ID: CRD42025642357) followed PRISMA extended and PRISMA-S guidelines. The population was defined according to the PICO framework: we included adult surgical patients undergoing anesthesia assessment (P), evaluated with LLM-based chatbots for perioperative risk stratification and anesthesia planning (I), compared with traditional clinician assessment (C), and extracted performance metrics (O). Comprehensive searches of PubMed, MEDLINE, Scopus, Embase, Google Scholar, Open Gray, ClinicalTrials.gov, WHO ICTRP, and Cochrane Library Central were conducted through January 2026. Risk of bias and study quality were assessed using PROBAST-AI, RoB-2, and ROBINS-I. Certainty of the evidence was assessed using GRADE system. A random-effects meta-analysis of pooled chatbot accuracy was performed, with subgroup analyses by ASA status and perioperative risk stratification. A sensitivity analysis was performed with a leave-one-out exclusion test. Results: Eleven studies published between 2023 and January 2026 were included (N = 227,059 patients). Five prospective cohorts, two large retrospective cohorts, one randomized non-inferiority trial, and three non-clinical or mixed-methods studies were found. Meta-analysis showed that the pooled accuracy of LLM-based chatbots for AI\u2013clinician concordance in perioperative risk stratification and ASA classification was 0.90 [95% CI: 0.42\u20130.99; 95% prediction interval 0.03\u20131.00]. Subgroup analyses indicated that the ASA status prediction subgroup reached a pooled accuracy of 0.91 (95% CI: 0.46 to 0.99), whereas the exploratory perioperative risk stratification subgroup showed an accuracy of 0.73 (95% CI: 0.10 to 0.98). Performance decreased with increasing patient complexity. Evidence is limited by small sample sizes, extreme sample size skew toward a single center, geographic bias, inconsistent outcome definitions and performance metrics, and incomplete reporting of adverse events. Most studies lacked prospective trial registration or robust control for confounding, and publication bias cannot be excluded. Conclusions: LLM-based chatbots show promising performance in routine perioperative risk stratification but remain unreliable in complex cases, with potential safety concerns. Given the overall very low GRADE certainty of evidence, these tools should be used as clinician-supervised decision support aids for routine ASA assessment, and should not be relied upon for autonomous use in complex cases or for general perioperative risk stratification. Other: This research received no external funding. PROSPERO ID: CRD42025642357.<\/jats:p>","DOI":"10.3390\/jcm15124670","type":"journal-article","created":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T03:27:21Z","timestamp":1781666841000},"page":"4670","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Perioperative Risk Stratification with AI-Powered Chatbots: A Systematic Review and Meta-Analysis"],"prefix":"10.3390","volume":"15","author":[{"given":"Valentina","family":"Bellini","sequence":"first","affiliation":[{"name":"Anesthesiology, Critical Care and Pain Medicine Division, Department of Medicine and Surgery, University of Parma, Viale Gramsci 14, 43126 Parma, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-8581-7085","authenticated-orcid":false,"given":"Matteo","family":"Panizzi","sequence":"additional","affiliation":[{"name":"Anesthesiology, Critical Care and Pain Medicine Division, Department of Medicine and Surgery, University of Parma, Viale Gramsci 14, 43126 Parma, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stefano","family":"Delrio","sequence":"additional","affiliation":[{"name":"Anesthesiology, Critical Care and Pain Medicine Division, Department of Medicine and Surgery, University of Parma, Viale Gramsci 14, 43126 Parma, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michele","family":"Berdini","sequence":"additional","affiliation":[{"name":"Anesthesiology, Critical Care and Pain Medicine Division, Department of Medicine and Surgery, University of Parma, Viale Gramsci 14, 43126 Parma, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Victor","family":"Sapountzakis","sequence":"additional","affiliation":[{"name":"Anesthesiology and Perioperative Medicine Division, Servicos Medicos de Anestesia (SMA), Rua Conselheiro Brotero 1486, S\u00e3o Paulo 01232-910, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luis Antonio","family":"dos Santos Diego","sequence":"additional","affiliation":[{"name":"Departamento de Anestesiologia, Universidade Federal Fluminense (UFF), Rio de Janeiro 24070-090, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9985-4126","authenticated-orcid":false,"given":"Elena Giovanna","family":"Bignami","sequence":"additional","affiliation":[{"name":"Anesthesiology, Critical Care and Pain Medicine Division, Department of Medicine and Surgery, University of Parma, Viale Gramsci 14, 43126 Parma, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,6,16]]},"reference":[{"key":"ref_1","first-page":"100419","article-title":"BERT-Based Medical Chatbot: Enhancing Healthcare Communication through Natural Language Understanding","volume":"13","author":"Babu","year":"2024","journal-title":"Explor. Res. Clin. Soc. Pharm."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"110447","DOI":"10.1016\/j.resuscitation.2024.110447","article-title":"Enhancing cardiac arrest response: Evaluating GPT-4o\u2019s advanced voice interaction system","volume":"205","author":"Semeraro","year":"2024","journal-title":"Resuscitation"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Chun-Hung, L., Guan-Hsiung, L., Wu-Chuan, Y., and Yu-Hsin, L. (2023). Chatbot-assisted therapy for patients with methamphetamine use disorder: A preliminary randomized controlled trial. Front. Psychiatry, 14.","DOI":"10.3389\/fpsyt.2023.1159399"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"e12887","DOI":"10.2196\/12887","article-title":"Physicians\u2019 Perceptions of Chatbots in Health Care: Cross-Sectional Web-Based Survey","volume":"21","author":"Palanica","year":"2019","journal-title":"J. Med. Internet Res."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"866","DOI":"10.1001\/jama.2023.14217","article-title":"Creation and Adoption of Large Language Models in Medicine","volume":"330","author":"Shah","year":"2023","journal-title":"JAMA"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1237","DOI":"10.1093\/jamia\/ocad072","article-title":"Using AI-generated suggestions from ChatGPT to optimize clinical decision support","volume":"30","author":"Liu","year":"2023","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Bhardwaz, S., and Kumar, J. (2023, January 4\u20136). An extensive comparative analysis of chatbot technologies\u2014ChatGPT, Google BARD and Micro-soft Bing. Proceedings of the 2023 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC), Salem, India.","DOI":"10.1109\/ICAAIC56838.2023.10140214"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"e49368","DOI":"10.2196\/49368","article-title":"A SWOT (Strengths, Weaknesses, Opportunities, and Threats) Analysis of ChatGPT in the Medical Literature: Concise Review","volume":"25","author":"Wolf","year":"2023","journal-title":"J. Med. Internet Res."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Sallam, M. (2023). ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare, 11.","DOI":"10.3390\/healthcare11060887"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"244","DOI":"10.17085\/apm.23056","article-title":"Utilizing ChatGPT in clinical research related to anesthesiology: A comprehensive review of opportunities and limitations","volume":"18","author":"Lee","year":"2023","journal-title":"Anesth. Pain Med."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"111475","DOI":"10.1016\/j.jclinane.2024.111475","article-title":"Evaluating the accuracy of ChatGPT-4 in predicting ASA scores: A prospective multicentric study ChatGPT-4 in ASA score prediction","volume":"96","author":"Turan","year":"2024","journal-title":"J. Clin. Anesth."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"599","DOI":"10.1097\/ALN.0000000000004841","article-title":"Understanding New Machine Learning Architectures: Practical Generative Artificial Intelligence for Anesthesiologists","volume":"140","author":"Connor","year":"2024","journal-title":"Anesthesiology"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"202","DOI":"10.4097\/kja.22157","article-title":"Artificial intelligence in perioperative medicine: A narrative review","volume":"75","author":"Yoon","year":"2022","journal-title":"Korean J. Anesthesiol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"e100985","DOI":"10.1136\/bmjhci-2023-100985","article-title":"Perioperative application of chatbots: A systematic review and meta-analysis","volume":"31","author":"Lin","year":"2024","journal-title":"BMJ Health Care Inform."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"e36","DOI":"10.1016\/j.bja.2023.04.033","article-title":"Artificial intelligence and anesthesia examinations: Exploring ChatGPT as a prelude to the future","volume":"131","author":"Aldridge","year":"2023","journal-title":"Br. J. Anaesth."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"e48633","DOI":"10.2196\/48633","article-title":"Barriers to and Facilitators of Artificial Intelligence Adoption in Health Care: Scoping Review","volume":"11","author":"Hassan","year":"2024","journal-title":"JMIR Hum. Factors"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"100172","DOI":"10.1016\/j.fhj.2024.100172","article-title":"Quality of interaction between clinicians and artificial intelligence systems. A systematic review","volume":"11","author":"Perivolaris","year":"2024","journal-title":"Future Healthc. J."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1136\/bmj.n71","article-title":"The PRISMA 2020 statement: An updated guideline for reporting systematic reviews","volume":"372","author":"Page","year":"2021","journal-title":"BMJ"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1186\/s13643-020-01542-z","article-title":"PRISMA-S: An extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews","volume":"10","author":"Rethlefsen","year":"2021","journal-title":"Syst. Rev."},{"key":"ref_20","unstructured":"Li, T., Higgins, J.P.T., and Deeks, J.J. (2024). Collecting data. Cochrane Handbook, Cochrane. Chapter 5."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"924","DOI":"10.1136\/bmj.39489.470347.AD","article-title":"GRADE: An emerging consensus on rating quality of evidence and strength of recommendations","volume":"336","author":"Guyatt","year":"2008","journal-title":"BMJ"},{"key":"ref_22","unstructured":"Sch\u00fcnemann, H.J., Higgins, J.P.T., Vist, G.E., Glasziou, P., Akl, E.A., Skoetz, N., and Guyatt, G.H. (2024). Completing \u2018Summary of findings\u2019 tables and grading the certainty of the evidence. Cochrane Handbooks and Manuals, Cochrane. Chapter 14."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"928","DOI":"10.1001\/jamasurg.2024.1621","article-title":"Large Language Model Capabilities in Periopera-tive Risk Prediction and Prognostication","volume":"159","author":"Chung","year":"2024","journal-title":"JAMA Surg."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1038\/s41746-025-01489-2","article-title":"The foundational capabilities of large language models in predicting postoperative risks using clinical notes","volume":"8","author":"Alba","year":"2025","journal-title":"npj Digit. Med."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Qi, J., Shen, J., Wu, Y., Zhang, S., Xu, D., and Wang, Y. (2025). Efficacy and Efficiency of Artificial Intelligence-based Preoperative Anesthesia Evaluation: A Randomized Controlled Non-Inferiority Study. Preprint.","DOI":"10.21203\/rs.3.rs-7364932\/v1"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"e39044","DOI":"10.2196\/39044","article-title":"Perioperative Risk Assessment of Patients Using the MyRISK Digital Score Completed Before the Preanesthetic Consultation: Prospective Observational Study","volume":"6","author":"Laurent","year":"2023","journal-title":"JMIR Perioper. Med."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Fujimoto, M., Kuroda, H., Katayama, T., Yamaguchi, A., Katagiri, N., Kagawa, K., Tsukimoto, S., Nakano, A., Imaizumi, U., and Sato-Boku, A. (2024). Evaluating Large Language Models in Dental Anesthesiology: A Comparative Analysis of ChatGPT-4, Claude 3 Opus, and Gemini 1.0 on the Japanese Dental Society of Anesthesiology Board Certification Exam. Cureus, 16.","DOI":"10.7759\/cureus.70302"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"62","DOI":"10.1111\/anae.16755","article-title":"Real-world deployment and evaluation of PEri-operative AI CHatbot (PEACH): A large language model chatbot for peri-operative medicine","volume":"81","author":"Ke","year":"2026","journal-title":"Anaesthesia"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1186\/s13741-024-00469-6","article-title":"The performance of ChatGPT in day surgery and pre-anesthesia risk assessment: A case-control study of 150 simulated patient presentations","volume":"13","author":"Cheng","year":"2024","journal-title":"Perioper. Med."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"\u00c7elik, E., Turgut, M.A., Aydo\u011fan, M., K\u0131l\u0131n\u00e7, M., Tokta\u015f, \u0130., and Akelma, H. (2025). Comparison of AI applications and anesthesiologist\u2019s anesthesia method choices. BMC Anesthesiol., 25.","DOI":"10.1186\/s12871-024-02882-2"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1007\/s10916-025-02247-3","article-title":"Performance of Large Language Models in Complex Anesthesia Decision-Making: A Comparative Study of Four LLMs in High-Risk Patients","volume":"49","author":"Ruan","year":"2025","journal-title":"J. Med. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1333","DOI":"10.1016\/j.bja.2024.08.038","article-title":"Generation of preoperative anaesthetic plans by ChatGPT-4.0: A mixed-method study","volume":"134","author":"Dahan","year":"2025","journal-title":"Br. J. Anaesth."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"462","DOI":"10.1038\/s41746-025-01858-x","article-title":"Clinical and economic impact of a large language model in perioperative medicine: A randomized crossover trial","volume":"8","author":"Ke","year":"2025","journal-title":"npj Digit. Med."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"904","DOI":"10.1007\/s12630-025-02973-9","article-title":"The applications of ChatGPT and other large language models in anesthesiology and critical care: A systematic review","volume":"72","author":"Daccache","year":"2025","journal-title":"Can. J. Anaesth.\/J. Can. D\u2019anesth\u00e9sie"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1213\/ANE.0000000000004728","article-title":"Science Without Conscience Is but the Ruin of the Soul: The Ethics of Big Data and Artificial Intelligence in Perioperative Medicine","volume":"130","author":"Canales","year":"2020","journal-title":"Anesth. Analg."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1898","DOI":"10.1007\/s10439-023-03234-w","article-title":"Artificial Intelligence in Intensive Care Medicine: Toward a ChatGPT\/GPT-4 Way?","volume":"51","author":"Lu","year":"2023","journal-title":"Ann. Biomed. Eng."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"e73","DOI":"10.1016\/j.bja.2023.06.052","article-title":"Large language models in anaesthesiology: Use of ChatGPT for American Society of Anesthesiologists physical status classification","volume":"131","author":"Lim","year":"2023","journal-title":"Br. J. Anaesth."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"111103","DOI":"10.1016\/j.jclinane.2023.111103","article-title":"Development and validation of a machine learning ASA-score to identify candidates for comprehensive preoperative screening and risk stratification","volume":"87","author":"Wongtangman","year":"2023","journal-title":"J. Clin. Anesth."},{"key":"ref_39","first-page":"e47155","article-title":"Machine Learning Algorithm to Perform the American Society of Anesthesiologists Physical Status Classification","volume":"15","author":"Lew","year":"2023","journal-title":"Cureus"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"e50638","DOI":"10.2196\/50638","article-title":"Prompt Engineering as an Important Emerging Skill for Medical Professionals: Tutorial","volume":"25","year":"2023","journal-title":"J. Med. Internet Res."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"691","DOI":"10.1097\/ACO.0000000000001318","article-title":"Decision-making in anesthesiology: Will artificial intelligence make intraoperative care safer?","volume":"36","author":"Duran","year":"2023","journal-title":"Curr. Opin. Anaesthesiol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"100280","DOI":"10.1016\/j.bjao.2024.100280","article-title":"Comparison of artificial intelligence large language model chatbots in answering frequently asked questions in anesthesia","volume":"10","author":"Nguyen","year":"2024","journal-title":"BJA Open"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Chung, P., Fong, C.T., Walters, A.M., Yetisgen, M., and O\u2019Reilly-Shah, V.N. (2023). Prediction of American Society of Anesthesiologists Physical Status Classification from preoperative clinical text narratives using natural language processing. BMC Anesthesiol., 23.","DOI":"10.1186\/s12871-023-02248-0"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2544331.","DOI":"10.1080\/08989621.2025.2544331","article-title":"GAIDeT (Generative AI Delegation Taxonomy): A taxonomy for humans to delegate tasks to generative artificial intelligence in scientific research and publishing","volume":"33","author":"Suchikova","year":"2026","journal-title":"Account. Res."}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/12\/4670\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,22]],"date-time":"2026-06-22T04:21:29Z","timestamp":1782102089000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/15\/12\/4670"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,16]]},"references-count":44,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2026,6]]}},"alternative-id":["jcm15124670"],"URL":"https:\/\/doi.org\/10.3390\/jcm15124670","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,16]]}}}