{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T16:30:21Z","timestamp":1787329821895,"version":"build-2736575974"},"reference-count":38,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>The standardized extraction of postoperative complications from unstructured routine clinical documentation remains a major unresolved challenge in digital surgery and health informatics. Although the Clavien-Dindo classification is the established standard for grading postoperative complications, its application in routine clinical documentation is largely implicit and unstructured, limiting scalable quality assessment in surgical care.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aimed to assess the capability of open-weight and proprietary large language models (LLMs) to classify postoperative complications according to the Clavien-Dindo system using discharge letters, benchmarked against expert annotation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We analyzed discharge letters from 650 surgical cases of 649 patients (median 67, IQR 58-73 y; 229\/649, 35% female) who underwent hepatobiliary surgery between 2010 and 2024. The cohort included grade I-II complications in 24% (153\/650), grade III-IV in 19% (121\/650), and grade V (death) in 6% (42\/650) of patients. A total of 4 open-weight (Qwen3-235B [Alibaba Cloud], Llama-3.3-70B [Meta AI], GPT-OSS-120B [OpenAI], Ministral-3-8B [Mistral AI]) and 2 proprietary (GPT 5.1 [OpenAI], Gemini 3 Pro [Google]) LLMs were prompted to infer complication grades directly from the discharge letters in a zero-shot setting. Model performance was evaluated against expert assessment using accuracy, F1-scores, and Cohen \u03ba. To assess interrater reliability and establish a human benchmark, a stratified 10% (n=65) subset was independently annotated by a second clinician, and Cohen \u03ba was calculated between annotators and between each model and the primary expert.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Interrater agreement between the 2 clinical annotators yielded a Cohen \u03ba of 0.75, providing a human benchmark for model performance interpretation. On the full 650-case dataset, open-weight models achieved accuracies ranging from 0.75 to 0.78 for fine-grained prediction, with weighted F1-scores of 0.76-0.78 and macroaveraged F1-scores of 0.50-0.63. For binary classification, accuracies ranged from 0.93 to 0.94, with weighted F1-scores of 0.93-0.95 and Cohen \u03ba of 0.76-0.79, approaching the human interrater benchmark. On a balanced 50-case subset, used as the sole basis for direct cross-model comparison, proprietary models achieved accuracies of 0.78 for fine-grained and 0.94-0.98 for binary classification. An ensemble approach yielded additional gains in classification performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>LLMs demonstrated promising accuracy in classifying postoperative complications from discharge letters in a zero-shot setting, with performance approaching the upper bound of human interrater agreement. Open-weight models offer a particularly attractive trade-off between accuracy and computational efficiency, while ensemble strategies further enhance robustness. These results support the potential of LLMs to standardize complication assessment at scale and enable data-driven quality monitoring in surgical care.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/92407","type":"journal-article","created":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T14:50:07Z","timestamp":1784213407000},"page":"e92407","source":"Crossref","is-referenced-by-count":0,"title":["Zero-Shot Classification of Postoperative Complications From Real-World Discharge Letters According to the Clavien-Dindo System Using Large Language Models in Liver Surgery: Comparative Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-2262-2655","authenticated-orcid":false,"given":"Sina","family":"Warmer","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6074-804X","authenticated-orcid":false,"given":"Kamyar","family":"Arzideh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-4528-2322","authenticated-orcid":false,"given":"Marie","family":"Morys","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1507-9690","authenticated-orcid":false,"given":"Ahmad","family":"Idrissi-Yaghir","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8143-6452","authenticated-orcid":false,"given":"Jan","family":"Bednarsch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6923-0849","authenticated-orcid":false,"given":"Daniel","family":"Heise","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8353-9532","authenticated-orcid":false,"given":"Arzu","family":"Oezcelik","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-3113-4741","authenticated-orcid":false,"given":"Marc","family":"Reschke","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-5793-1959","authenticated-orcid":false,"given":"Tom F","family":"Ulmer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3831-8917","authenticated-orcid":false,"given":"Ulf P","family":"Neumann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5811-7100","authenticated-orcid":false,"given":"Felix","family":"Nensa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6987-6041","authenticated-orcid":false,"given":"Katarzyna","family":"Borys","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1760-2342","authenticated-orcid":false,"given":"Ren\u00e9","family":"Hosch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6732-1595","authenticated-orcid":false,"given":"Sophia M","family":"Schmitz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,21]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artmed.2021.102224"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2021.103984"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2024.09.062"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI190228"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1097\/SLA.0000000000006471"},{"key":"ref6","doi-asserted-by":"publisher","DOI":"10.1097\/01.sla.0000133083.54934.ae"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-024-02855-5"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02448-8"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2025.102963"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1145\/3744746"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.2196\/68776"},{"key":"ref12","doi-asserted-by":"publisher","DOI":"10.1007\/s11704-026-60308-3"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2405.19327"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2407.21783"},{"key":"ref16","unstructured":"Introducing Mistral 3Mistral AI20252026-07-18https:\/\/mistral.ai\/news\/mistral-3\/"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.16609"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2508.10925"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01488-3"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.06825"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.13971"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.1109\/ISPASS61541.2024.00034"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1109\/cbms.2013.6627810"},{"key":"ref24","doi-asserted-by":"publisher","DOI":"10.2196\/64486"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.3390\/dj13060261"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.2196\/76947"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.1007\/s12630-025-02973-9"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1055\/a-2424-1989"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.5489\/cuaj.505"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.1097\/SLA.0000000000006399"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.1007\/s00270-025-04287-1"},{"key":"ref32","doi-asserted-by":"publisher","DOI":"10.1097\/SLA.0b013e3181b13ca2"},{"key":"ref33","doi-asserted-by":"publisher","DOI":"10.1111\/iju.12576"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1001\/jamaoto.2023.0169"},{"key":"ref35","doi-asserted-by":"publisher","DOI":"10.4254\/wjh.v6.i10.745"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaa296"},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-023-02101-x"},{"key":"ref38","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-019-0935-4"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e92407","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,21]],"date-time":"2026-08-21T15:35:28Z","timestamp":1787326528000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e92407"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,21]]},"references-count":38,"URL":"https:\/\/doi.org\/10.2196\/92407","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.92407","asserted-by":"object"}]},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,21]]}}}