{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,27]],"date-time":"2026-04-27T16:26:16Z","timestamp":1777307176963,"version":"3.51.4"},"reference-count":32,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2025,10,27]],"date-time":"2025-10-27T00:00:00Z","timestamp":1761523200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["SCH-2205289"],"award-info":[{"award-number":["SCH-2205289"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["SCH-2014438"],"award-info":[{"award-number":["SCH-2014438"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["IIS-2034479"],"award-info":[{"award-number":["IIS-2034479"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>Large language models\u2019 (LLMs\u2019) performance in high-stakes, compliance-driven settings such as drafting clinical research documents remains underexplored. This study aims to build a benchmark and an evaluation framework for assessing LLMs\u2019 compliance and factuality in generating informed consent forms (ICFs) from clinical trial protocols.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>We introduce InformBench, a benchmark comprising 900 clinical trial documents, and propose an evaluation framework grounded in regulatory guidelines and site-specific consent templates. We assess LLM performance on transforming trial protocols, often hundreds of pages, into concise, patient-facing ICFs. Additionally, we design InformGen, a retrieval-augmented, human-in-the-loop pipeline aimed at improving generation quality.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Baseline LLMs such as GPT-4o achieved only 70%-80% compliance and exhibited factual errors in 18%-43% of cases. In contrast, InformGen substantially improved outputs, achieving nearly 100% regulatory compliance and over 90% factual accuracy, as validated by 5 domain-expert annotators.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>The study reveals critical limitations in current LLMs for clinical research document drafting, particularly in regulatory sensitivity and factual grounding. Our results highlight the need for domain-specific benchmarks and structured evaluations to support safe deployment in real-world clinical research workflows.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>LLMs offer value in clinical research document generation but must be adapted and rigorously evaluated for high-stakes applications. Our benchmark and framework provide a foundation for improving and assessing LLM-generated outputs in compliance-critical domains.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf174","type":"journal-article","created":{"date-parts":[[2025,9,26]],"date-time":"2025-09-26T12:06:54Z","timestamp":1758888414000},"page":"563-572","source":"Crossref","is-referenced-by-count":3,"title":["Compliance and factuality of large language models for clinical research document generation"],"prefix":"10.1093","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3026-9970","authenticated-orcid":false,"given":"Zifeng","family":"Wang","sequence":"first","affiliation":[{"name":"Keiji AI , Seattle, WA, 98115,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4951-8682","authenticated-orcid":false,"given":"Junyi","family":"Gao","sequence":"additional","affiliation":[{"name":"Centre for Medical Informatics, Usher Institute, University of Edinburgh , Edinburgh, EH3 9DR,","place":["United Kingdom"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Benjamin","family":"Danek","sequence":"additional","affiliation":[{"name":"Keiji AI , Seattle, WA, 98115,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Brandon","family":"Theodorou","sequence":"additional","affiliation":[{"name":"Keiji AI , Seattle, WA, 98115,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ruba","family":"Shaik","sequence":"additional","affiliation":[{"name":"Carle Illinois College of Medicine, University of Illinois Urbana-Champaign , Urbana, IL, 61801,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shivashankar","family":"Thati","sequence":"additional","affiliation":[{"name":"Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign , Urbana, IL, 61801,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Seunghyun","family":"Won","sequence":"additional","affiliation":[{"name":"Medical Research Collaborating Center, Seoul National University Bundang Hospital , Gyeonggi, 13620,","place":["Republic of Korea"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1512-6426","authenticated-orcid":false,"given":"Jimeng","family":"Sun","sequence":"additional","affiliation":[{"name":"Keiji AI , Seattle, WA, 98115,","place":["United States"]},{"name":"Carle Illinois College of Medicine, University of Illinois Urbana-Champaign , Urbana, IL, 61801,","place":["United States"]},{"name":"Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign , Urbana, IL, 61801,","place":["United States"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2025,10,27]]},"reference":[{"key":"2026031216464872600_ocaf174-B1","article-title":"E6(r2) good clinical practice: integrated addendum to ich e6(r1)"},{"key":"2026031216464872600_ocaf174-B2","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1186\/1472-6939-14-28","article-title":"Improving understanding in the research informed consent process: a systematic review of 54 interventions tested in randomized control trials","volume":"14","author":"Nishimura","year":"2013","journal-title":"BMC Med Ethics"},{"key":"2026031216464872600_ocaf174-B3","doi-asserted-by":"crossref","first-page":"117","DOI":"10.2149\/tmh.2014-36","article-title":"Understanding of essential elements required in informed consent form among researchers and institutional review board members","volume":"43","author":"Koonrungsesomboon","year":"2015","journal-title":"Trop Med Health"},{"key":"2026031216464872600_ocaf174-B4","doi-asserted-by":"crossref","first-page":"855","DOI":"10.1038\/s41586-023-06382-0","article-title":"Consent document translation expense hinders inclusive clinical trial enrolment","volume":"620","author":"Velez","year":"2023","journal-title":"Nature"},{"key":"2026031216464872600_ocaf174-B5","first-page":"841","author":"Yuan","year":"2022"},{"key":"2026031216464872600_ocaf174-B6","author":"Wang","year":"2024"},{"key":"2026031216464872600_ocaf174-B7","doi-asserted-by":"crossref","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","article-title":"Adapted large language models can outperform medical experts in clinical text summarization","volume":"30","author":"Van Veen","year":"2024","journal-title":"Nat Med"},{"key":"2026031216464872600_ocaf174-B8","doi-asserted-by":"crossref","first-page":"2878","DOI":"10.1038\/s41591-024-03148-7","article-title":"Outpatient reception via collaboration between nurses and a large language model: a randomized controlled trial","volume":"30","author":"Wan","year":"2024","journal-title":"Nat Med"},{"key":"2026031216464872600_ocaf174-B9","doi-asserted-by":"crossref","first-page":"9074","DOI":"10.1038\/s41467-024-53081-z","article-title":"Matching patients to clinical trials with large language models","volume":"15","author":"Jin","year":"2024","journal-title":"Nat Commun"},{"key":"2026031216464872600_ocaf174-B10","first-page":"12461","author":"Wang","year":"2023"},{"key":"2026031216464872600_ocaf174-B11","first-page":"2024","author":"Lin","year":"2024"},{"key":"2026031216464872600_ocaf174-B12","doi-asserted-by":"crossref","first-page":"AIcs2300145","DOI":"10.1056\/AIcs2300145","article-title":"Using ChatGPT to facilitate truly informed medical consent","volume":"1","author":"Mirza","year":"2024","journal-title":"Nejm Ai"},{"key":"2026031216464872600_ocaf174-B13","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1038\/s41746-024-01039-2","article-title":"Bridging the literacy gap for surgical consents: an ai-human expert collaborative approach","volume":"7","author":"Ali","year":"2024","journal-title":"NPJ Digit Med"},{"key":"2026031216464872600_ocaf174-B14","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026031216464872600_ocaf174-B15","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026031216464872600_ocaf174-B16","author":"Gao","year":"2023"},{"key":"2026031216464872600_ocaf174-B17","first-page":"1","article-title":"A survey on LLM-generated text detection: necessity, methods, and future directions","author":"Wu","year":"2025","journal-title":"Computational Linguistics"},{"key":"2026031216464872600_ocaf174-B18","author":"Wang","year":"2025"},{"key":"2026031216464872600_ocaf174-B19","first-page":"ZE10","article-title":"Protocol writing in clinical research","volume":"10","author":"Al-JunDi","year":"2016","journal-title":"J Clin Diagn Res JCDR"},{"key":"2026031216464872600_ocaf174-B20","volume-title":"Informed Consent Guidance for Irbs, Clinical Investigators, and Sponsors","author":"Food, Administration, D","year":"2023"},{"key":"2026031216464872600_ocaf174-B21","author":"Forms & consent templates"},{"key":"2026031216464872600_ocaf174-B22","author":"Consent and assent form templates"},{"key":"2026031216464872600_ocaf174-B23","first-page":"311","author":"Papineni","year":"2002"},{"key":"2026031216464872600_ocaf174-B24","doi-asserted-by":"crossref","first-page":"e2336997","DOI":"10.1001\/jamanetworkopen.2023.36997","article-title":"Large language model- based chatbot vs surgeon-generated informed consent documentation for common procedures","volume":"6","author":"Decker","year":"2023","journal-title":"JAMA Netw Open"},{"key":"2026031216464872600_ocaf174-B25","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1016\/j.jcms.2024.10.002","article-title":"Evaluating ai-generated informed consent documents in oral surgery: a comparative study of ChatGPT-4, Bard Gemini advanced, and human-written consents","volume":"53","author":"Vaira","year":"2025","journal-title":"J Craniomaxillofac Surg"},{"key":"2026031216464872600_ocaf174-B26","doi-asserted-by":"crossref","first-page":"e68139","DOI":"10.2196\/68139","article-title":"Transforming informed consent generation using large language models: mixed methods study","volume":"13","author":"Shi","year":"2025","journal-title":"JMIR Med Inform"},{"key":"2026031216464872600_ocaf174-B27","author":"OpenAI","year":"2024"},{"key":"2026031216464872600_ocaf174-B28","author":"Team, G","year":"2023"},{"key":"2026031216464872600_ocaf174-B29","doi-asserted-by":"crossref","first-page":"1145","DOI":"10.3390\/make6020053","article-title":"Evaluation of ai chatbots for the creation of patient-informed consent sheets","volume":"6","author":"Raimann","year":"2024","journal-title":"MAKE"},{"key":"2026031216464872600_ocaf174-B30","author":"Common cancer types"},{"key":"2026031216464872600_ocaf174-B31","first-page":"1","article-title":"Augmenting research consent: should large language models (LLMs) be used for informed consent to clinical research?","volume":"0","author":"Allen","year":"2024","journal-title":"Res Ethics"},{"key":"2026031216464872600_ocaf174-B32","first-page":"17407745251320806","article-title":"From rags to riches: Utilizing large language models to write documents for clinical trials","author":"Markey","year":"2025","journal-title":"Clin Trials"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/3\/563\/64959727\/ocaf174.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/33\/3\/563\/64959727\/ocaf174.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T20:46:54Z","timestamp":1773348414000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/33\/3\/563\/8304363"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,27]]},"references-count":32,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,10,27]]},"published-print":{"date-parts":[[2026,3,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf174","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,3]]},"published":{"date-parts":[[2025,10,27]]}}}