{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T20:59:21Z","timestamp":1791233961673,"version":"4.1.0"},"reference-count":46,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Nursing"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Clinical handover is the process during which responsibility and accountability for care are transferred between clinicians. AI has the potential to improve the reliability and completeness of clinical handover by helping clinicians detect predefined content areas that have been communicated, identify explicit information gaps, and prompt clarification before responsibility is transferred.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study evaluated the performance of several large language models and prompt optimization strategies for structured information extraction of synthetic nursing handover transcripts.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Two registered nurses independently annotated a dataset of 203 synthetic handover transcripts to produce consensus labels for information extraction tasks. Tasks included (1) labeling spans of text into SBAR (Situation, Background, Assessment, Recommendation) categories, (2) content detection to determine if specific pieces of information were communicated, and (3) labeling spans of text that communicated information using uncertain terms that included a subtask for identifying unknown facts. Baseline and Genetic-Pareto (GEPA)\u2013optimized prompts were compared for the GPT-5.2, GPT-5-nano, and MedGemma 27B large language models. Additionally, the LangExtract framework was evaluated for span-extraction tasks.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The GPT-5.2\u2013optimized model achieved a micro\n \u2013F<\/jats:italic>\n 1<\/jats:sub>\n -score of 0.85 (95% CI 0.83\u20100.88) for content detection, an absolute improvement of +0.08 compared with the matched baseline. GPT-5-nano also performed better after optimization for content detection (micro\n \u2013F<\/jats:italic>\n 1<\/jats:sub>\n -score 0.81, 95% CI 0.78\u20100.84), suggesting that this structured task was not limited to the highest-capacity model. For SBAR span extraction, GPT-5.2 with prompt optimization achieved a micro\n \u2013F<\/jats:italic>\n 1<\/jats:sub>\n -score of 0.76 (95% CI 0.72\u20100.79), improving by +0.24 compared with baseline and exceeding LangExtract; GPT-5-nano also improved to a micro\n \u2013F<\/jats:italic>\n 1<\/jats:sub>\n -score of 0.69 (95% CI 0.66\u20100.72). Broad uncertainty-span extraction remained comparatively weak despite prompt optimization (micro\n \u2013F<\/jats:italic>\n 1<\/jats:sub>\n -score 0.41, 95% CI 0.33\u20100.48; absolute improvement +0.06). In contrast, explicit unknown-fact extraction was more accurate with GPT-5.2 (micro\n \u2013F<\/jats:italic>\n 1<\/jats:sub>\n -score 0.84, 95% CI 0.63\u20101.00), GPT-5-nano (micro\n \u2013F<\/jats:italic>\n 1<\/jats:sub>\n -score, 0.84 95% CI 0.63\u20101.00), and MedGemma 27B (micro\n \u2013F<\/jats:italic>\n 1<\/jats:sub>\n -score 0.80, 95% CI 0.63\u20101.00). Genetic-Pareto\u2013optimized prompts outperformed the LangExtract approach across each span-extraction task.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Prompt optimization improved matched-model point estimates, with the highest performance observed for predefined content detection and SBAR span extraction. Broad uncertainty extraction remained less accurate than the narrower unknown-fact task. These technical results do not establish clinical effectiveness, safety, or readiness for real-time use. Validation using authentic nursing handover communication and prospective evaluation in clinical workflows are required before clinical application.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/106133","type":"journal-article","created":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T20:31:56Z","timestamp":1791232316000},"page":"e106133-e106133","source":"Crossref","is-referenced-by-count":0,"title":["AI-Based Structured Information Extraction From Synthetic Nursing Handover Transcripts: Comparative Evaluation of Large Language Models"],"prefix":"10.2196","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9583-8636","authenticated-orcid":false,"given":"Aaron","family":"Conway","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9794-6502","authenticated-orcid":false,"given":"Adriana","family":"Hada","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-7381-7107","authenticated-orcid":false,"given":"Jessica","family":"Schluter","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3421-4176","authenticated-orcid":false,"given":"Hui (Grace)","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-2932-952X","authenticated-orcid":false,"given":"Dan","family":"Lowden","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4908-6063","authenticated-orcid":false,"given":"Tim","family":"Miller","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7652-6322","authenticated-orcid":false,"given":"Ken","family":"Donald","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0802-718X","authenticated-orcid":false,"given":"Andrew","family":"Teodorczuk","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,10,5]]},"reference":[{"key":"R1","unstructured":"Further information on communicating for safety. Australian Commission on Safety and Quality in Health Care. 2026. URL: https:\/\/www.safetyandquality.gov.au\/national-standards\/nsqhs-standards\/communicating-safety-standard\/further-information-communicating-safety [Accessed 22-09-2026]"},{"issue":"6","key":"R2","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1016\/s1553-7250(11)37035-3","article-title":"A systematic review of failures in handoff communication during intrahospital transfers","volume":"37","author":"Ong","journal-title":"Jt Comm J Qual Patient Saf"},{"issue":"1-2","key":"R3","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1111\/jocn.12986","article-title":"Perspectives of clinical handover processes: a multi-site survey across different health professionals","volume":"25","author":"Manias","journal-title":"J Clin Nurs"},{"key":"R4","unstructured":"The Joint Commission, and Joint Commission International. Communication during patient hand-overs. World Health Organization; 2007. URL: https:\/\/cdn.who.int\/media\/docs\/default-source\/patient-safety\/patient-safety-solutions\/ps-solution3-communication-during-patient-handovers.pdf [Accessed 22-09-2026]"},{"issue":"2","key":"R5","doi-asserted-by":"publisher","first-page":"466","DOI":"10.1111\/nhs.12836","article-title":"Translating evidence-based nursing clinical handover practice in an acute care setting: a quasi-experimental study","volume":"23","author":"Hada","journal-title":"Nurs Health Sci"},{"issue":"3","key":"R6","doi-asserted-by":"publisher","DOI":"10.1093\/intqhc\/mzad065","article-title":"A scoping review of clinical handover mnemonic devices","volume":"35","author":"Yung","journal-title":"Int J Qual Health Care"},{"issue":"10","key":"R7","doi-asserted-by":"publisher","first-page":"680","DOI":"10.1136\/bmjqs-2024-018385","article-title":"Use of structured handoff protocols for within-hospital unit transitions: a systematic review from Making Healthcare Safer IV","volume":"34","author":"McCarthy","journal-title":"BMJ Qual Saf"},{"issue":"11","key":"R8","doi-asserted-by":"publisher","first-page":"4297","DOI":"10.1111\/jocn.17430","article-title":"Exploring the experiences and perceptions of the utilisation of structured clinical handover frameworks by nurses working in acute care settings: a scoping review","volume":"33","author":"Moyo","journal-title":"J Clin Nurs"},{"issue":"7","key":"R9","doi-asserted-by":"publisher","first-page":"2688","DOI":"10.1111\/jocn.17107","article-title":"Implementing a ward-level intervention to improve nursing handover communication with a focus on bedside handover-a qualitative study","volume":"33","author":"Chien","journal-title":"J Clin Nurs"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijnurstu.2017.10.014","article-title":"Patient participation in nursing bedside handover: a systematic mixed-methods review","volume":"77","author":"Tobiano","journal-title":"Int J Nurs Stud"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.1155\/2024\/3208747","article-title":"Perceptions of patients and nurses about bedside nursing handover: a qualitative systematic review and meta-synthesis","volume":"2024","author":"Anshasi","journal-title":"Nurs Res Pract"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.130","article-title":"Large language models are few-shot clinical information extractors","author":"Agrawal","journal-title":"Proc 2022 Conf Empir Methods Nat Lang Process"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.2196\/55318","article-title":"An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: algorithm development and validation study","volume":"12","author":"Sivarajkumar","journal-title":"JMIR Med Inform"},{"issue":"July","key":"R14","doi-asserted-by":"publisher","DOI":"10.2196\/70358","article-title":"Technological solutions to improve inpatient handover in the era of artificial intelligence: scoping review","volume":"27","author":"Agha-Mir-Salim","journal-title":"J Med Internet Res"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/81604","article-title":"Integrating a large language model to streamline nursing handover documentation across multiple hospitals in Taiwan: development and implementation study","volume":"28","author":"Chen","journal-title":"J Med Internet Res"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","article-title":"The TRIPOD-LLM reporting guideline for studies using large language models","volume":"31","author":"Gallifant","journal-title":"Nat Med"},{"issue":"2","key":"R17","doi-asserted-by":"publisher","DOI":"10.2196\/medinform.4321","article-title":"Benchmarking clinical speech recognition and information extraction: new data, methods, and evaluations","volume":"3","author":"Suominen","journal-title":"JMIR Med Inform"},{"key":"R18","unstructured":"Khattab O Singhvi A Maheshwari P et al. DSPy: compiling declarative language model calls into self-improving pipelines. Presented at: The 12th International Conference on Learning Representations (ICLR 2024); May 7-11, 2024; Vienna, Austria. URL: https:\/\/openreview.net\/pdf?id=sY5N0zY5Od [Accessed 22-09-2026]"},{"key":"R19","unstructured":"Australian Research Council (ARC); Universities Australia. National statement on ethical conduct in human research 2025. National Health and Medical Research Council (NHMRC); 2025. URL: https:\/\/www.nhmrc.gov.au\/sites\/default\/files\/documents\/attachments\/publications\/National-Statement-on-Ethical-Conduct-Human-Research-2025.pdf [Accessed 22-09-2026]"},{"key":"R20","unstructured":"Prodigy. URL: https:\/\/prodi.gy [Accessed 22-09-2026]"},{"key":"R21","unstructured":"Singh A Fry A Perelman A et al. OpenAI GPT-5 system card. arXiv. Preprint posted online on Dec 19, 2025. [doi: 10.48550\/arXiv.2601.03267]"},{"key":"R22","unstructured":"Sellergren A Kazemzadeh S Jaroensri T et al. MedGemma technical report. arXiv. Preprint posted online on Jul 7, 2025. [doi: 10.48550\/arXiv.2507.05201]"},{"key":"R23","unstructured":"Agrawal LA Tan S Soylu D et al. GEPA: reflective prompt evolution can outperform reinforcement learning. Presented at: 14th International Conference on Learning Representations (ICLR 2026); Apr 23-27, 2026; Rio de Janeiro, Brazil. URL: https:\/\/openreview.net\/pdf?id=RQm2KQTM5r [Accessed 22-09-2026]"},{"key":"R24","unstructured":"Goel A . LangExtract v1.7.0. Zenodo. 2026. URL: https:\/\/doi.org\/10.5281\/zenodo.17015089 [Accessed 22-09-2026]"},{"issue":"3","key":"R25","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1016\/s1553-7250(06)32022-3","article-title":"SBAR: a shared mental model for improving communication between clinicians","volume":"32","author":"Haig","journal-title":"Jt Comm J Qual Patient Saf"},{"issue":"4","key":"R26","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1097\/01.NAJ.0000370154.79857.09","article-title":"Nursing handoffs: a systematic review of the literature","volume":"110","author":"Riesenberg","journal-title":"Am J Nurs"},{"issue":"3","key":"R27","doi-asserted-by":"publisher","first-page":"744","DOI":"10.1111\/jonm.12936","article-title":"A systematic review on the structured handover interventions between nurses in improving patient safety outcomes","volume":"28","author":"Bukoh","journal-title":"J Nurs Manag"},{"issue":"2","key":"R28","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1016\/j.bpa.2011.02.006","article-title":"Effective handover communication: an overview of research and improvement efforts","volume":"25","author":"Manser","journal-title":"Best Pract Res Clin Anaesthesiol"},{"issue":"7494","key":"R29","doi-asserted-by":"publisher","first-page":"765","DOI":"10.1136\/bmj.38398.500764.8F","article-title":"Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success","volume":"330","author":"Kawamoto","journal-title":"BMJ"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1136\/amiajnl-2011-000089","article-title":"Automation bias: a systematic review of frequency, effect mediators, and mitigators","volume":"19","author":"Goddard","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"R31","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1136\/bmjqs-2018-008370","article-title":"Artificial intelligence, bias and clinical safety","volume":"28","author":"Challen","journal-title":"BMJ Qual Saf"},{"issue":"2","key":"R32","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1136\/jamia.2010.007237","article-title":"Data from clinical notes: a perspective on the tension between structure and flexible documentation","volume":"18","author":"Rosenbloom","journal-title":"J Am Med Inform Assoc"},{"issue":"6","key":"R33","doi-asserted-by":"publisher","first-page":"493","DOI":"10.1136\/qshc.2009.033480","article-title":"The published literature on handoffs in hospitals: deficiencies identified in an extensive review","volume":"19","author":"Cohen","journal-title":"Qual Saf Health Care"},{"issue":"April","key":"R34","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1016\/j.apnr.2017.12.013","article-title":"Mixed methods evaluation of a quality improvement and audit tool for nurse-to-nurse bedside clinical handover in ward settings","volume":"40","author":"Redley","journal-title":"Appl Nurs Res"},{"issue":"2","key":"R35","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1093\/intqhc\/mzh026","article-title":"Handoff strategies in settings with high consequences for failure: lessons for health care operations","volume":"16","author":"Patterson","journal-title":"Int J Qual Health Care"},{"issue":"Suppl 1","key":"R36","doi-asserted-by":"publisher","first-page":"i85","DOI":"10.1136\/qhc.13.suppl_1.i85","article-title":"The human factor: the critical importance of effective teamwork and communication in providing safe care","volume":"13","author":"Leonard","journal-title":"Qual Saf Health Care"},{"issue":"19","key":"R37","doi-asserted-by":"publisher","first-page":"1803","DOI":"10.1056\/NEJMsa1405556","article-title":"Changes in medical errors after implementation of a handoff program","volume":"371","author":"Starmer","journal-title":"N Engl J Med"},{"issue":"5","key":"R38","doi-asserted-by":"publisher","first-page":"809","DOI":"10.1136\/amiajnl-2011-000648","article-title":"Active learning for clinical text classification: is it better than random sampling?","volume":"19","author":"Figueroa","journal-title":"J Am Med Inform Assoc"},{"key":"R39","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2015.09.010","article-title":"A study of active learning methods for named entity recognition in clinical text","volume":"58","author":"Chen","journal-title":"J Biomed Inform"},{"issue":"1","key":"R40","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-022-00611-y","article-title":"Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare","volume":"5","author":"Feng","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R41","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-019-1426-2","article-title":"Key challenges for delivering clinical impact with artificial intelligence","volume":"17","author":"Kelly","journal-title":"BMC Med"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.2196\/101190","article-title":"Beyond time saved: implementation, equity, and the utility threshold for nursing AI scribes","volume":"28","author":"Ronquillo","journal-title":"J Med Internet Res"},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.2196\/72674","article-title":"Opportunities, challenges, and future directions for the integration of automation in nursing practice: discursive study","volume":"8","author":"Pepito","journal-title":"JMIR Nurs"},{"issue":"Suppl 3","key":"R44","doi-asserted-by":"publisher","first-page":"i68","DOI":"10.1136\/qshc.2010.042085","article-title":"A new sociotechnical model for studying health information technology in complex adaptive healthcare systems","volume":"19 Suppl 3","author":"Sittig","journal-title":"Qual Saf Health Care"},{"key":"R45","doi-asserted-by":"publisher","DOI":"10.2196\/84148","article-title":"Readiness assessment for AI in nursing care projects: multimethods study","volume":"9","author":"Seibert","journal-title":"JMIR Nurs"},{"key":"R46","unstructured":"Large language models for structured information extraction in artificial intelligence-assisted clinical handover: evaluation study. Figshare. URL: https:\/\/figshare.com\/articles\/dataset\/Large_Language_Models_for_Structured_Information_Extraction_in_Artificial_Intelligence-Assisted_Clinical_Handover_Evaluation_Study\/32658084?file=65497158 [Accessed 22-09-2026]"}],"container-title":["JMIR Nursing"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T20:31:59Z","timestamp":1791232319000},"score":1,"resource":{"primary":{"URL":"https:\/\/nursing.jmir.org\/2026\/1\/e106133"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,5]]},"references-count":46,"URL":"https:\/\/doi.org\/10.2196\/106133","relation":{},"ISSN":["2562-7600"],"issn-type":[{"value":"2562-7600","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,5]]},"article-number":"v9i6e106133"}}