{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:43:49Z","timestamp":1785545029029,"version":"3.56.0"},"reference-count":50,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Molecular Tumor Boards (MTBs) generate highly technical recommendations. The language used in their protocols is rarely accessible to patients. Lay-language patient protocols could support patient-clinician communication, yet manual production is difficult to sustain in high-volume oncology settings. Large language models (LLMs) may offer scalable drafting assistance, yet clinical usability remains largely uninvestigated under real-world deployment constraints. Existing evaluations rely predominantly on synthetic data or closed-source models that are incompatible with strict data protection requirements.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study evaluated whether open-weight LLMs can provide clinically usable drafting support for German MTB patient protocols under real-world deployment constraints and developed a transferable evaluation framework for patient-facing text generation.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Eight open-weight LLMs were evaluated under zero-shot (A1) and one-shot (A2) prompting with constrained decoding, which ensures section-schema compliance. Automatic evaluation used ROUGE-1 (Recall-Oriented Understudy for Gisting Evaluation), BERTScore-F1 (Bidirectional Encoder Representations From Transformers Score), Wiener Sachtextformel version 4, and DistilBERT (Distilled Version of Bidirectional Encoder Representations From Transformers)\u2013based complexity using a corpus of 316 MTB protocols and 47 expert-written patient protocols. For expert evaluation, 7 medical oncologists evaluated 50 protocols from the best-performing model across 3 International Organization for Standardization 9241\u201011 usability dimensions using fine-grained error annotation, perceived postediting effort (PPEE), and net promoter score. Critical errors were defined as bearing the risk of patient harm.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Llama-3.3-70B-Instruct achieved the strongest automatic performance. Across models, A2 significantly improved most automatic metrics compared to A1. However, expert usability evaluation of Llama-3.3-70B-Instruct showed the opposite picture: the proportion of protocols containing at least 1 critical error doubled under A2 (10\/25, 40% vs 5\/25, 20%) compared with A1, and the dominant error type shifted from language (40\/108, 37%) errors to factual errors (69\/145, 48%). Overall, 16% (230\/1420) of the annotated paragraphs contained errors. Median PPEE was 2 (IQR 2.0\u20103.0; low), and median net promoter score was 7 (IQR 5.0-9.0). Detractors (46\/100, 46%) outweighed promoters (29\/100, 29%), which suggests hesitation toward routine adoption. These differences in expert evaluation between A2 and A1 were directionally consistent but did not reach individual statistical significance for the paired samples (n=25).<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Prompting strategies that improve automatic metrics can simultaneously increase the number of critical errors. Surface-level metric gains were, therefore, insufficient proxies for clinical safety. This was observed as a consistent directional pattern for a single model, but generalization to other models remains to be investigated. Nonetheless, the low paragraph-level error rate and favorable PPEE suggest that structured open-weight LLM generation may be a useful drafting support in a clinician-supervised setting. The proposed evaluation framework provides a text-quality-focused basis for future assessment of patient-facing LLM applications in real-world clinical settings.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/99136","type":"journal-article","created":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T09:50:05Z","timestamp":1784886605000},"page":"e99136-e99136","source":"Crossref","is-referenced-by-count":0,"title":["LLM-Generated Lay-Language Protocols for Molecular Tumor Board Patients: Evaluation of Quality and Clinical Usability"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-9802-7167","authenticated-orcid":false,"given":"Tabea Margareta Grace","family":"Pakull","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5552-4061","authenticated-orcid":false,"given":"No\u00eblle","family":"Bender","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4487-4258","authenticated-orcid":false,"given":"Sven","family":"Benson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9377-1159","authenticated-orcid":false,"given":"Anke","family":"Fleischhauer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-2853-0269","authenticated-orcid":false,"given":"Mohammad","family":"Alsara","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-1373-0840","authenticated-orcid":false,"given":"Tanja","family":"Gromke","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-5814-8044","authenticated-orcid":false,"given":"Thomas","family":"Hilser","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-9884-7114","authenticated-orcid":false,"given":"Katharina","family":"Kaminski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7854-9931","authenticated-orcid":false,"given":"Michael","family":"Pogorzelski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-4375-8769","authenticated-orcid":false,"given":"Nicola","family":"Prasuhn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5133-6776","authenticated-orcid":false,"given":"Vivian","family":"Rosery","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3524-7858","authenticated-orcid":false,"given":"Dirk","family":"Schadendorf","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2166-3394","authenticated-orcid":false,"given":"Martin","family":"Schuler","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9698-9559","authenticated-orcid":false,"given":"Marcel","family":"Wiesweg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3813-6961","authenticated-orcid":false,"given":"Gregor","family":"Zaun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0901-3484","authenticated-orcid":false,"given":"Peter Alexander","family":"Horn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7906-0038","authenticated-orcid":false,"given":"Christoph Matthias","family":"Friedrich","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-6725-9829","authenticated-orcid":false,"given":"Ina","family":"Pretzell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,7,23]]},"reference":[{"issue":"9","key":"R1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0107828","article-title":"An integrative model of patient-centeredness - a systematic review and concept analysis","volume":"9","author":"Scholl","journal-title":"PLOS ONE"},{"issue":"3","key":"R2","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1016\/j.pec.2008.11.015","article-title":"How does communication heal? Pathways linking clinician-patient communication to health outcomes","volume":"74","author":"Street","journal-title":"Patient Educ Couns"},{"issue":"6","key":"R3","doi-asserted-by":"publisher","first-page":"1298","DOI":"10.1038\/s41591-023-02354-z","article-title":"The German Network for Personalized Medicine to enhance patient care and translational research","volume":"29","author":"Illert","journal-title":"Nat Med"},{"issue":"4","key":"R4","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","journal-title":"JAMA"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-024-00717-2","article-title":"Current applications and challenges in large language models for patient care: a systematic review","volume":"5","author":"Busch","journal-title":"Commun Med (Lond)"},{"key":"R6","unstructured":"Brown TB Mann B Ryder N et al. Language models are few-shot learners. Presented at: Proceedings of the 34th international conference on neural information processing systems Vancouver; Dec 6-12, 2020; Vancouver, British Columbia, Canada. URL: https:\/\/proceedings.neurips.cc\/paper\/2020\/file\/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf [Accessed 04-07-2026]"},{"issue":"7972","key":"R7","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature"},{"key":"R8","unstructured":"OpenAI AJ Adler S Agarwal S et al. GPT-4 technical report. arXiv. Preprint posted online on Mar 4, 2024. [doi: 10.48550\/arXiv.2303.08774]"},{"key":"R9","unstructured":"Willard BT Louf R . Efficient guided generation for large language models. arXiv. Preprint posted online on Aug 19, 2023. [doi: 10.48550\/arXiv.2307.09702]"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01670-7","article-title":"A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation","volume":"8","author":"Asgari","journal-title":"NPJ Digit Med"},{"key":"R11","doi-asserted-by":"crossref","unstructured":"Pakull T Dada A Damm H et al. Preliminary evaluation of an open-source LLM for lay translation of german clinical documents. In: Ananiadou S Demner-Fushman D Gupta D Thompson P , editors. Presented at: Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health); May 3-4, 2025:180-192; Albuquerque, NM. [doi: 10.18653\/v1\/2025.cl4health-1.15]","DOI":"10.18653\/v1\/2025.cl4health-1.15"},{"key":"R12","unstructured":"Ergonomics of human-system interaction \u2014 part 11: usability: definitions and concepts. ISO. 2018. URL: https:\/\/www.iso.org\/standard\/63500.html [Accessed 04-07-2026]"},{"issue":"5","key":"R13","doi-asserted-by":"publisher","DOI":"10.1093\/oncolo\/oyag119","article-title":"MyCODE: a prospective evaluation of lay-language molecular tumor board protocols in precision oncology","volume":"31","author":"Pretzell","journal-title":"Oncologist"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Kwon W Li Z Zhuang S et al. Efficient memory management for large language model serving with pagedattention. Presented at: SOSP \u201923; Oct 23-26, 2023:611-626; Koblenz, Germany. [doi: 10.1145\/3600006.3613165]","DOI":"10.1145\/3600006.3613165"},{"key":"R15","unstructured":"Openai\/openai-python. OpenAI. URL: https:\/\/github.com\/openai\/openai-python [Accessed 04-07-2026]"},{"issue":"4","key":"R16","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1145\/3714983.3714987","article-title":"AWQ: activation-aware weight quantization for on-device LLM compression and acceleration","volume":"28","author":"Lin","journal-title":"GetMobile: Mobile Comp Commun"},{"key":"R17","unstructured":"Rouhani BD Garegrat N Savell T et al. OCP microscaling formats (MX) specification - version 1.0. Open Compute Project; Sep 2023. URL: https:\/\/www.opencompute.org\/documents\/ocp-microscaling-formats-mx-v1-0-spec-final-pdf [Accessed 04-07-2026]"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","article-title":"The TRIPOD-LLM reporting guideline for studies using large language models","volume":"31","author":"Gallifant","journal-title":"Nat Med"},{"key":"R19","unstructured":"OpenAIAgarwal S Ahmad L Ai J et al. Gpt-oss-120b & gpt-oss-20b model card. arXiv. Preprint posted online on Aug 8, 2025. [doi: 10.48550\/arXiv.2508.10925]"},{"key":"R20","unstructured":"Grattafiori A Dubey A Jauhri A Pandey A Kadian A Al-Dahle A . The llama 3 herd of models. arXiv. Preprint posted online on Nov 23, 2024. [doi: 10.48550\/ARXIV.2407.21783]"},{"key":"R21","unstructured":"Mistralai\/mistral-large-instruct-2411. Hugging Face. URL: https:\/\/huggingface.co\/mistralai\/Mistral-Large-Instruct-2411 [Accessed 04-07-2026]"},{"key":"R22","unstructured":"Mistral AI Rastogi A Jiang AQ et al. Magistral. arXiv. Preprint posted online on Jun 12, 2025. [doi: 10.48550\/ARXIV.2506.10910]"},{"key":"R23","unstructured":"Yang A Li A Yang B et al. Qwen3 technical report. arXiv. Preprint posted online on May 14, 2025. [doi: 10.48550\/ARXIV.2505.09388]"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.2196\/58329","article-title":"Evaluation framework of large language models in medical documentation: development and usability study","volume":"26","author":"Seo","journal-title":"J Med Internet Res"},{"key":"R25","unstructured":"Guidance-ai\/llguidance. GitHub. URL: https:\/\/github.com\/guidance-ai\/llguidance [Accessed 04-07-2026]"},{"key":"R26","unstructured":"Pallets\/jinja. GitHub. URL: https:\/\/github.com\/pallets\/jinja [Accessed 04-07-2026]"},{"key":"R27","unstructured":"Lin CY . ROUGE: a package for automatic evaluation of summaries. Presented at: Workshop on Text Summarization Branches Out, Post-Conference Workshop of ACL; Jul 25-26, 2004:74-81; Barcelona, Spain. URL: https:\/\/www.microsoft.com\/en-us\/research\/publication\/rouge-a-package-for-automatic-evaluation-of-summaries\/ [Accessed 04-07-2026]"},{"key":"R28","unstructured":"Zhang T Kishore V Wu F Weinberger KQ Artzi Y . BERTScore: evaluating text generation with BERT. Presented at: 8th International Conference on Learning Representations Addis Ababa; Apr 26-30, 2020; Addis Ababa, Ethiopia. URL: https:\/\/openreview.net\/forum?id=SkeHuCVFDr [Accessed 04-07-2026]"},{"key":"R29","unstructured":"Bamberger R Vanecek E . Lesen-Verstehen-Lernen-Schreiben: Die Schwierigkeitsstufen von Texten in Deutscher Sprache [Book in German]. Jugend und Volk; 1984. ISBN: 978-3-425-01903-1"},{"key":"R30","unstructured":"Ansch\u00fctz M Groh G . TUM social computing at GermEval 2022: towards the significance of text statistics and neural embeddings in text complexity prediction. Presented at: Proceedings of the GermEval 2022 workshop on text complexity assessment of german text Potsdam; Sep 12-15, 2022:21-26; Potsdam. URL: https:\/\/aclanthology.org\/anthology-files\/anthology-files\/pdf\/germeval\/2022.germeval-1.pdf [Accessed 04-07-2026]"},{"issue":"2","key":"R31","doi-asserted-by":"publisher","first-page":"164","DOI":"10.4338\/aci-2011-11-ra-0070","article-title":"Assessing electronic note quality using the Physician Documentation Quality Instrument (PDQI-9)","volume":"3","author":"Stetson","journal-title":"Appl Clin Inform"},{"key":"R32","doi-asserted-by":"crossref","unstructured":"Fabbri AR Kry\u015bci\u0144ski W McCann B Xiong C Socher R Radev D . SummEval: re-evaluating summarization evaluation. In: Roark B Nenkova A , editors. Transactions of the Association for Computational Linguistics. Vol 9. MIT Press; 2021:391-409. [doi: 10.1162\/tacl_a_00373]","DOI":"10.1162\/tacl_a_00373"},{"key":"R33","doi-asserted-by":"crossref","unstructured":"Maynez J Narayan S Bohnet B McDonald R . On faithfulness and factuality in abstractive summarization. In: Jurafsky D Chai J Schluter N Tetreault J , editors. Presented at: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; Jul 5-10, 2020:1906-1919; [doi: 10.18653\/v1\/2020.acl-main.173]","DOI":"10.18653\/v1\/2020.acl-main.173"},{"key":"R34","unstructured":"Lommel AR Burchardt A Uszkoreit H . Multidimensional quality metrics: a flexible system for assessing translation quality. Presented at: Proceedings of Translating and the Computer 35; Nov 28-29, 2013; Paddington, London, UK. URL: https:\/\/aclanthology.org\/2013.tc-1.6.pdf [Accessed 04-07-2026]"},{"issue":"3","key":"R35","doi-asserted-by":"publisher","first-page":"425","DOI":"10.2307\/30036540","article-title":"User acceptance of information technology: toward a unified view","volume":"27","author":"Venkatesh","journal-title":"MIS Q"},{"key":"R36","unstructured":"Guerberof A . Productivity and quality in MT post-editing. Presented at: Beyond Translation Memories: New Tools for Translators Workshop; Aug 26-30, 2009; Ottawa, Canada. URL: https:\/\/aclanthology.org\/2009.mtsummit-btm.7\/ [Accessed 04-07-2026]"},{"key":"R37","unstructured":"Scarton S Forcada ML Espl\u00e0-Gomis M Specia L . Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT quality. In: Niehues J Cattoni R St\u00fcker S Negri M Turchi M Ha TL Salesky E Sanabria R Barrault L Specia L Federico M , editors. Presented at: Proceedings of the 16th International Conference on Spoken Language Translation; Nov 2-3, 2019; Hong Kong. 2019.URL: https:\/\/aclanthology.org\/2019.iwslt-1.23.pdf [Accessed 04-07-2026]"},{"issue":"12","key":"R38","first-page":"46","volume":"81","author":"Reichheld","journal-title":"Harv Bus Rev"},{"key":"R39","unstructured":"Streamlit\/streamlit. GitHub. URL: https:\/\/github.com\/streamlit\/streamlit [Accessed 04-07-2026]"},{"issue":"3","key":"R40","doi-asserted-by":"publisher","DOI":"10.2307\/3001946","article-title":"Probability tables for individual comparisons by ranking methods","volume":"3","author":"Wilcoxon","journal-title":"Biometrics"},{"issue":"1","key":"R41","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1111\/j.2517-6161.1995.tb02031.x","article-title":"Controlling the false discovery rate: a practical and powerful approach to multiple testing","volume":"57","author":"Benjamini","journal-title":"J R Stat Soc Ser B"},{"key":"R42","unstructured":"Conover WJ Conover WJ . Practical Nonparametric Statistics. 3rd ed. Wiley; 1999. ISBN: 978-0-471-16068-7"},{"issue":"Pt 1","key":"R43","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1348\/000711006X126600","article-title":"Computing inter-rater reliability and its variance in the presence of high agreement","volume":"61","author":"Gwet","journal-title":"Br J Math Stat Psychol"},{"key":"R44","doi-asserted-by":"publisher","DOI":"10.1177\/23821205221093205","article-title":"Assessing interrater reliability of a faculty-provided feedback rating instrument","volume":"9","author":"Walsh","journal-title":"J Med Educ Curric Dev"},{"key":"R45","doi-asserted-by":"crossref","unstructured":"Kryscinski W McCann B Xiong C Socher R . Evaluating the factual consistency of abstractive text summarization. In: Webber B Cohn T He Y Liu Y , editors. Presented at: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP); Nov 16-20, 2020:9332-9346; [doi: 10.18653\/v1\/2020.emnlp-main.750]","DOI":"10.18653\/v1\/2020.emnlp-main.750"},{"key":"R46","doi-asserted-by":"crossref","unstructured":"Min S Lyu X Holtzman A et al. Rethinking the role of demonstrations: what makes in-context learning work? In: Goldberg Y Kozareva Z Zhang Y , editors. Presented at: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing; Dec 7-11, 2022:11048-11064; Abu Dhabi, United Arab Emirates. [doi: 10.18653\/v1\/2022.emnlp-main.759]","DOI":"10.18653\/v1\/2022.emnlp-main.759"},{"key":"R47","doi-asserted-by":"publisher","DOI":"10.1016\/j.clinimag.2026.110741","article-title":"Comment on \u201cclassifying the clinical significance of common breast pain symptoms using a large language model, ChatGPT (GPT-4)\u201d","volume":"132","author":"Hu","journal-title":"Clin Imaging"},{"key":"R48","unstructured":"Subramanian S Elango V Gungor M . Small language models (SLMs) can still pack a punch: a survey. arXiv. Preprint posted online on May 14, 2026. [doi: 10.48550\/arXiv.2501.05465]"},{"issue":"1","key":"R49","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1136\/amiajnl-2011-000089","article-title":"Automation bias: a systematic review of frequency, effect mediators, and mitigators","volume":"19","author":"Goddard","journal-title":"J Am Med Inf Assoc"},{"key":"R50","doi-asserted-by":"publisher","DOI":"10.2196\/90759","article-title":"Toward retrieval-grounded evaluation for conversational large language model-based risk assessment","volume":"5","author":"Hu","journal-title":"JMIR AI"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e99136","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:23:06Z","timestamp":1785536586000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e99136"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,23]]},"references-count":50,"URL":"https:\/\/doi.org\/10.2196\/99136","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,23]]},"article-number":"v28i9e99136"}}