{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T18:15:21Z","timestamp":1782929721717,"version":"3.54.5"},"reference-count":36,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T00:00:00Z","timestamp":1778198400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>The automation of medical report generation using large language models (LLMs) could significantly reduce physicians\u2019 documentation burden while enhancing healthcare efficiency. However, the misuse of generative artificial intelligence in medical reporting can lead to important safety risks for patients. We addressed 2 questions: (1) What is the quality of medical reports generated by LLMs in English and French? and (2) Can we distinguish between human-written and LLM-generated medical reports?<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and methods<\/jats:title>\n                    <jats:p>We evaluated the quality of reports generated by several multilingual, open-weight LLMs using text similarity metrics on 4212 medical reports in English and French across multiple specialties. A bilingual expert panel of certified physicians (n\u2009=\u20094) and medical residents (n\u2009=\u20095) scored accuracy, fluency, and completeness of generated reports using a 1-5 Likert scale. Experts also completed a Turing-like test, blindly identifying reports as human or machine-generated.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Phi-4 achieved the best overall performance (ROUGE-1: 0.70, BERTScore: 0.83). Expert evaluation confirmed high-quality reports in both languages (overall 4.6\/5.0). Medical experts performed better than chance but struggled to differentiate human versus machine reports (accuracy: 0.60). Automatic classifiers showed strong performance (accuracy: 0.98).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>The high quality of LLM-generated reports supports their potential to enhance healthcare efficiency in multilingual settings. However, the discrepancy between human detection difficulty and automated detection success reveals inherent limitations in relying solely on human oversight for quality assurance and misuse prevention.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Deployment of LLMs for medical reporting requires combining automated detection tools with human expertise to ensure patient safety. Dataset and code: https:\/\/github.com\/ds4dh\/medical_report_generation.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocag070","type":"journal-article","created":{"date-parts":[[2026,4,20]],"date-time":"2026-04-20T11:24:23Z","timestamp":1776684263000},"page":"1303-1313","source":"Crossref","is-referenced-by-count":1,"title":["The detectability paradox: bilingual medical report generation with open-weight models and the limits of human oversight"],"prefix":"10.1093","volume":"33","author":[{"given":"Hossein","family":"Rouhizadeh","sequence":"first","affiliation":[{"name":"University of Geneva Department of Radiology and Medical Informatics, Faculty of Medicine, , Geneva, 1202,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abiram","family":"Sandralegar","sequence":"additional","affiliation":[{"name":"Geneva University Hospitals Division of Neurosurgery, , Geneva, 1211,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anthony","family":"Yazdani","sequence":"additional","affiliation":[{"name":"University of Geneva Department of Radiology and Medical Informatics, Faculty of Medicine, , Geneva, 1202,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weibo","family":"Feng","sequence":"additional","affiliation":[{"name":"University of Geneva Department of Radiology and Medical Informatics, Faculty of Medicine, , Geneva, 1202,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oren","family":"Schreier","sequence":"additional","affiliation":[{"name":"University of Geneva Department of Radiology and Medical Informatics, Faculty of Medicine, , Geneva, 1202,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yonnou","family":"Ahn-Kim","sequence":"additional","affiliation":[{"name":"University of Geneva Faculty of Medicine, , Geneva, 1211,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Assiya","family":"Sirbal","sequence":"additional","affiliation":[{"name":"University of Geneva Faculty of Medicine, , Geneva, 1211,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Valentino","family":"Pirelli","sequence":"additional","affiliation":[{"name":"University of Geneva Faculty of Medicine, , Geneva, 1211,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Yang","sequence":"additional","affiliation":[{"name":"Duke-NUS Medical School Centre for Quantitative Medicine, , Singapore, 169857,","place":["Singapore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lukas","family":"Sveikata","sequence":"additional","affiliation":[{"name":"Geneva University Hospitals and Faculty of Medicine Division of Neurology, Department of Clinical Neurosciences, , Geneva, 1211,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elena","family":"Tessitore","sequence":"additional","affiliation":[{"name":"Geneva University Hospitals Cardiology Division, Internal Medicine Department, , Geneva, 1211,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3610-4883","authenticated-orcid":false,"given":"Nan","family":"Liu","sequence":"additional","affiliation":[{"name":"Duke-NUS Medical School Centre for Quantitative Medicine, , Singapore, 169857,","place":["Singapore"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Philippe","family":"Bijlenga","sequence":"additional","affiliation":[{"name":"Geneva University Hospitals Division of Neurosurgery, , Geneva, 1211,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6238-4503","authenticated-orcid":false,"given":"Douglas","family":"Teodoro","sequence":"additional","affiliation":[{"name":"University of Geneva Department of Radiology and Medical Informatics, Faculty of Medicine, , Geneva, 1202,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,5,8]]},"reference":[{"key":"2026063015352667700_ocag070-B1","doi-asserted-by":"crossref","first-page":"102888","DOI":"10.1016\/j.inffus.2024.102888","article-title":"A comprehensive survey of large language models and multimodal large language models in medicine","volume":"117","author":"Xiao","year":"2025","journal-title":"Inform Fusion"},{"key":"2026063015352667700_ocag070-B2","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1016\/j.mayocp.2018.02.016","article-title":"Reimagining clinical documentation with artificial intelligence","volume":"93","author":"Lin","year":"2018","journal-title":"Mayo Clin Proc."},{"key":"2026063015352667700_ocag070-B3","first-page":"16266","article-title":"Auto-encoding knowledge graph for unsupervised medical report generation","volume":"34","author":"Liu","year":"2021","journal-title":"Adv Neural Inf Process Syst."},{"key":"2026063015352667700_ocag070-B4","doi-asserted-by":"publisher","first-page":"7597","DOI":"10.18653\/v1\/2024.emnlp-main.433","author":"Zhou","year":"2024"},{"key":"2026063015352667700_ocag070-B5","first-page":"565","author":"Yuan","year":"2024"},{"key":"2026063015352667700_ocag070-B6","author":"Jung"},{"key":"2026063015352667700_ocag070-B7","first-page":"2952"},{"key":"2026063015352667700_ocag070-B8"},{"key":"2026063015352667700_ocag070-B9","doi-asserted-by":"crossref","first-page":"5066","DOI":"10.1109\/JBHI.2023.3304376","article-title":"Leveraging summary guidance on medical report summarization","volume":"27","author":"Zhu","year":"2023","journal-title":"IEEE J Biomed Health Inform"},{"key":"2026063015352667700_ocag070-B10"},{"key":"2026063015352667700_ocag070-B11","author":"Gostin"},{"key":"2026063015352667700_ocag070-B12","doi-asserted-by":"publisher","author":"Rouhizadeh","year":"2025","DOI":"10.1101\/2025.01.15.25320579"},{"key":"2026063015352667700_ocag070-B13","first-page":"74","article-title":"Integrating structured and unstructured EHR data using an FHIR-based type system: a case study with medication data","volume":"2017","author":"Hong","year":"2018","journal-title":"AMIA Jt Summits Transl Sci Proc"},{"key":"2026063015352667700_ocag070-B14","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1093\/bmb\/ldy038","article-title":"The impact of the general data protection regulation on health research","volume":"128","author":"Chico","year":"2018","journal-title":"Br Med Bull."},{"key":"2026063015352667700_ocag070-B15","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1038\/s41746-025-01429-0","article-title":"Privacy preserving strategies for electronic health records in the era of large language models","volume":"8","author":"Jonnagaddala","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2026063015352667700_ocag070-B16","first-page":"788","article-title":"Exploring zero-shot cross-lingual biomedical concept normalization via large language models","volume":"327","author":"Rouhizadeh","year":"2025","journal-title":"Stud Health Technol Inform."},{"key":"2026063015352667700_ocag070-B17"},{"key":"2026063015352667700_ocag070-B18","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1038\/s41597-024-03317-w","article-title":"A dataset for evaluating contextualized representation of biomedical concepts in language models","volume":"11","author":"Rouhizadeh","year":"2024","journal-title":"Sci Data"},{"key":"2026063015352667700_ocag070-B19","doi-asserted-by":"crossref","first-page":"AIra2400012","DOI":"10.1056\/AIra2400012","article-title":"Clinical text datasets for medical artificial intelligence and large language models\u2014a systematic review","volume":"1","author":"Wu","year":"2024","journal-title":"NEJM AI."},{"key":"2026063015352667700_ocag070-B20","doi-asserted-by":"publisher","first-page":"1513","DOI":"10.18653\/v1\/2025.emnlp-main.79","author":"Xuan","year":"2025"},{"key":"2026063015352667700_ocag070-B21","doi-asserted-by":"crossref","first-page":"909","DOI":"10.1038\/s41597-023-02814-8","article-title":"A large-scale dataset of patient summaries for retrieval-based clinical decision support systems","volume":"10","author":"Zhao","year":"2023","journal-title":"Sci Data."},{"key":"2026063015352667700_ocag070-B22"},{"key":"2026063015352667700_ocag070-B23"},{"key":"2026063015352667700_ocag070-B24","author":"AI@Meta","year":"2024"},{"key":"2026063015352667700_ocag070-B25"},{"key":"2026063015352667700_ocag070-B26"},{"key":"2026063015352667700_ocag070-B27","year":"2024"},{"key":"2026063015352667700_ocag070-B28","author":"Ankit Pal","year":"2024"},{"key":"2026063015352667700_ocag070-B29"},{"key":"2026063015352667700_ocag070-B30"},{"key":"2026063015352667700_ocag070-B31","first-page":"74","volume-title":"Text Summarization Branches Out","author":"Lin","year":"2004"},{"key":"2026063015352667700_ocag070-B32"},{"key":"2026063015352667700_ocag070-B33"},{"key":"2026063015352667700_ocag070-B34","doi-asserted-by":"crossref","first-page":"1015","DOI":"10.1093\/jamia\/ocaf045","article-title":"Evaluating the effectiveness of biomedical fine-tuning for large language models on clinical tasks","volume":"32","author":"Dorfner","year":"2025","journal-title":"J Am Med Inform Assoc."},{"key":"2026063015352667700_ocag070-B35","doi-asserted-by":"crossref","first-page":"38","DOI":"10.7453\/gahmj.2013.008","article-title":"The CARE guidelines: consensus-based clinical case reporting guideline development*","volume":"2","author":"Gagnier","year":"2013","journal-title":"Glob Adv Health Med"},{"key":"2026063015352667700_ocag070-B36","first-page":"1059","article-title":"Toward relieving clinician burden by automatically generating progress notes using interim hospital data","volume":"2024","author":"Soni","year":"2024","journal-title":"AMIA Annu Symp Proc."}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/advance-article-pdf\/doi\/10.1093\/jamia\/ocag070\/68247613\/ocag070.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/7\/1303\/68247613\/ocag070.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/7\/1303\/68247613\/ocag070.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T10:37:20Z","timestamp":1782902240000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/7\/1303\/8672665"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,8]]},"references-count":36,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2026,5,8]]},"published-print":{"date-parts":[[2026,7,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocag070","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,7]]},"published":{"date-parts":[[2026,5,8]]}}}