{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T17:54:38Z","timestamp":1791568478889,"version":"4.3.4"},"reference-count":40,"publisher":"Public Library of Science (PLoS)","issue":"10","license":[{"start":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T00:00:00Z","timestamp":1791504000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000092","name":"U.S. National Library of Medicine","doi-asserted-by":"publisher","award":["R01LM011934"],"award-info":[{"award-number":["R01LM011934"]}],"id":[{"id":"10.13039\/100000092","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100004917","name":"Cancer Prevention and Research Institute of Texas","doi-asserted-by":"publisher","award":["RR230020"],"award-info":[{"award-number":["RR230020"]}],"id":[{"id":"10.13039\/100004917","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosdigitalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Digit Health"],"abstract":"<jats:p>Systematic reviews of randomized controlled trials (RCTs) support evidence-based clinical advancement. Clinical trial reporting guidelines, including CONSORT, establish common data elements that support systematic synthesis of published trial outcomes. However, the rapid rise of digital health technologies presents new challenges for reporting guidelines, and substantial real-world reporting heterogeneity has been observed. Therefore, machine-assisted evaluation of reporting adherence offers a pathway for more agile information management to accommodate the diverse and disruptive nature of digital health technology-enabled RCTs. Here, we propose an accelerated retrieval-to-abstraction literature synthesis framework to conduct article screening, text and table data preprocessing, and full-text review using an unmodified pretrained large language model. We apply this methodology in two digital health technology-enabled RCT case studies evaluating adherence to technology and digital health equity guideline reporting items. Large language model performance was on par with inter-annotator agreement F1 scores (\u22650.9) in both screening and information-extraction tasks. These findings support the scalability and rigor of accelerated evidence synthesis and demonstrate the importance of common reporting elements across digital health technology-enabled RCTs.<\/jats:p>","DOI":"10.1371\/journal.pdig.0001768","type":"journal-article","created":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T17:25:37Z","timestamp":1791566737000},"page":"e0001768","update-policy":"https:\/\/doi.org\/10.1371\/journal.pdig.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Retrieval-to-abstraction LLM-assisted literature synthesis: Case demonstrations in digital health technology-enabled randomized controlled trial outcome reporting"],"prefix":"10.1371","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-9997-291X","authenticated-orcid":true,"given":"Taylor B.","family":"Harrison","sequence":"first","affiliation":[{"name":"Mayo Clinic","place":["Rochester, Minnesota, United States of America"],"department":["Department of Artificial Intelligence and Informatics"]},{"name":"University of Minnesota","place":["Minneapolis, Minnesota, United States of America"],"department":["Bioinformatics and Computational Biology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1277-142X","authenticated-orcid":true,"given":"Dian","family":"Hu","sequence":"additional","affiliation":[{"name":"University of Texas Health Science Center","place":["Houston, Texas, United States of America"],"department":["McWilliams School of Biomedical Informatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heling","family":"Jia","sequence":"additional","affiliation":[{"name":"Mayo Clinic","place":["Rochester, Minnesota, United States of America"],"department":["Department of Artificial Intelligence and Informatics"]},{"name":"University of Minnesota","place":["Minneapolis, Minnesota, United States of America"],"department":["Bioinformatics and Computational Biology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shyang Hong","family":"Tan","sequence":"additional","affiliation":[{"name":"University of Minnesota","place":["Minneapolis, Minnesota, United States of America"],"department":["Bioinformatics and Computational Biology"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-7453-3541","authenticated-orcid":true,"given":"Fang","family":"Chen","sequence":"additional","affiliation":[{"name":"University of Texas Health Science Center","place":["Houston, Texas, United States of America"],"department":["McWilliams School of Biomedical Informatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhongyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Northwestern University","place":["Chicago, Illinois, United States of America"],"department":["Integrated Science Program"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiuhao","family":"Lu","sequence":"additional","affiliation":[{"name":"University of Texas Health Science Center","place":["Houston, Texas, United States of America"],"department":["McWilliams School of Biomedical Informatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinlian","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Texas Health Science Center","place":["Houston, Texas, United States of America"],"department":["McWilliams School of Biomedical Informatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liwei","family":"Wang","sequence":"additional","affiliation":[{"name":"University of Texas Health Science Center","place":["Houston, Texas, United States of America"],"department":["McWilliams School of Biomedical Informatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ming","family":"Huang","sequence":"additional","affiliation":[{"name":"University of Texas Health Science Center","place":["Houston, Texas, United States of America"],"department":["McWilliams School of Biomedical Informatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Larry J.","family":"Prokop","sequence":"additional","affiliation":[{"name":"Mayo Clinic","place":["Rochester, Minnesota, United States of America"],"department":["Department of Library Services"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matt","family":"Hoy","sequence":"additional","affiliation":[{"name":"Mayo Clinic","place":["Rochester, Minnesota, United States of America"],"department":["Department of Library Services"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"St. Sauver","sequence":"additional","affiliation":[{"name":"Mayo Clinic","place":["Rochester, Minnesota, United States of America"],"department":["Division of Epidemiology, Department of Quantitative Health Sciences"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sunyang","family":"Fu","sequence":"additional","affiliation":[{"name":"University of Texas Health Science Center","place":["Houston, Texas, United States of America"],"department":["McWilliams School of Biomedical Informatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hongfang","family":"Liu","sequence":"additional","affiliation":[{"name":"University of Texas Health Science Center","place":["Houston, Texas, United States of America"],"department":["McWilliams School of Biomedical Informatics"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,10,9]]},"reference":[{"issue":"2","key":"pdig.0001768.ref001","doi-asserted-by":"crossref","first-page":"100","DOI":"10.4103\/0976-500X.72352","article-title":"CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials","volume":"1","author":"KF Schulz","year":"2010","journal-title":"J Pharmacol Pharmacother"},{"key":"pdig.0001768.ref002","article-title":"CONSORT 2025 statement: updated guideline for reporting randomised trials","author":"S Hopewell","year":"2025","journal-title":"The Lancet"},{"issue":"10","key":"pdig.0001768.ref003","article-title":"Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension","volume":"2","author":"SPIRIT-AI and CONSORT-AI Working Group","year":"2020","journal-title":"Lancet Digit Health"},{"issue":"4","key":"pdig.0001768.ref004","article-title":"CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions","volume":"13","author":"CONSORT-EHEALTH Group","year":"2011","journal-title":"J Med Internet Res"},{"issue":"1","key":"pdig.0001768.ref005","doi-asserted-by":"crossref","first-page":"1619","DOI":"10.1038\/s41467-024-45355-3","article-title":"Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines","volume":"15","author":"APL Martindale","year":"2024","journal-title":"Nat Commun"},{"issue":"9913","key":"pdig.0001768.ref006","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1016\/S0140-6736(13)62228-X","article-title":"Reducing waste from incomplete or unusable reports of biomedical research","volume":"383","author":"P Glasziou","year":"2014","journal-title":"Lancet"},{"key":"pdig.0001768.ref007","doi-asserted-by":"crossref","unstructured":"Srinivasan A, Kivelson S, Friedrich NA, Berkowitz J, Tatonetti N. Evaluating the reporting quality of 21,041 randomized controlled trial articles with large language models: A large-scale transparency analysis. In: 2025.","DOI":"10.1101\/2025.03.06.25323528"},{"issue":"1","key":"pdig.0001768.ref008","doi-asserted-by":"crossref","first-page":"5126","DOI":"10.1038\/s41467-022-32310-3","article-title":"Reproducibility of real-world evidence studies using clinical practice data to inform regulatory and coverage decisions","volume":"13","author":"REPEAT Initiative","year":"2022","journal-title":"Nat Commun"},{"issue":"2","key":"pdig.0001768.ref009","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2016-012545","article-title":"Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry","volume":"7","author":"R Borah","year":"2017","journal-title":"BMJ Open"},{"key":"pdig.0001768.ref010","doi-asserted-by":"crossref","first-page":"100443","DOI":"10.1016\/j.conctc.2019.100443","article-title":"The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials","volume":"16","author":"M Michelson","year":"2019","journal-title":"Contemp Clin Trials Commun"},{"key":"pdig.0001768.ref011","doi-asserted-by":"crossref","first-page":"101287","DOI":"10.1016\/j.cola.2024.101287","article-title":"Screening articles for systematic reviews with ChatGPT","volume":"80","author":"E Syriani","year":"2024","journal-title":"Journal of Computer Languages"},{"issue":"8","key":"pdig.0001768.ref012","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1007\/s10462-024-10844-w","article-title":"Towards the automation of systematic reviews using natural language processing, machine learning, and deep learning: a comprehensive review","volume":"57","author":"R Ofori-Boateng","year":"2024","journal-title":"Artificial intelligence review"},{"key":"pdig.0001768.ref013","doi-asserted-by":"crossref","first-page":"106589","DOI":"10.1016\/j.infsof.2021.106589","article-title":"Automation of systematic literature reviews: A systematic literature review","volume":"136","author":"R van Dinter","year":"2021","journal-title":"Information and Software Technology"},{"key":"pdig.0001768.ref014","article-title":"Using artificial intelligence methods for systematic review in health sciences: a systematic review","author":"A Blaizot","journal-title":"Appendices"},{"issue":"1","key":"pdig.0001768.ref015","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0313401","article-title":"ChatGPT-4o can serve as the second rater for data extraction in systematic reviews","volume":"20","author":"M Motzfeldt Jensen","year":"2025","journal-title":"PLoS One"},{"issue":"7","key":"pdig.0001768.ref016","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2024.20496","article-title":"Performance of a Large Language Model in Screening Citations","volume":"7","author":"T Oami","year":"2024","journal-title":"JAMA Netw Open"},{"key":"pdig.0001768.ref017","article-title":"Large language models are few-shot clinical information extractors. arXiv preprint","author":"M Agrawal","year":"2022"},{"key":"pdig.0001768.ref018","doi-asserted-by":"crossref","first-page":"105531","DOI":"10.1016\/j.ijmedinf.2024.105531","article-title":"Implementation and evaluation of an additional GPT-4-based reviewer in PRISMA-based medical systematic literature reviews","volume":"189","author":"A Landschaft","year":"2024","journal-title":"Int J Med Inform"},{"issue":"4","key":"pdig.0001768.ref019","doi-asserted-by":"crossref","first-page":"616","DOI":"10.1002\/jrsm.1715","article-title":"Can large language models replace humans in systematic reviews? Evaluating GPT-4\u2019s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages","volume":"15","author":"Q Khraisha","year":"2024","journal-title":"Res Synth Methods"},{"key":"pdig.0001768.ref020","unstructured":"Harrison TBH, Dian S, Liu H. The Use of Large Language Models to Accelerate Literature Review Towards Digital health equity and inclusiveness. 2024."},{"issue":"1","key":"pdig.0001768.ref021","doi-asserted-by":"crossref","first-page":"3280","DOI":"10.1038\/s41467-025-56989-2","article-title":"Benchmarking large language models for biomedical natural language processing applications and recommendations","volume":"16","author":"Q Chen","year":"2025","journal-title":"Nat Commun"},{"key":"pdig.0001768.ref022","article-title":"The PRISMA 2020 statement: an updated guideline for reporting systematic reviews","volume":"372","author":"MJ Page","year":"2021","journal-title":"BMJ"},{"issue":"6","key":"pdig.0001768.ref023","doi-asserted-by":"crossref","first-page":"1776","DOI":"10.1093\/bioinformatics\/btab880","article-title":"MedTator: a serverless annotation tool for corpus development","volume":"38","author":"H He","year":"2022","journal-title":"Bioinformatics"},{"key":"pdig.0001768.ref024","doi-asserted-by":"crossref","unstructured":"Hopewell S, Clarke M. CONSORT for abstracts. guidelines for reporting health research: A User\u2019s Manual. 2014:68\u201379.","DOI":"10.1002\/9781118715598.ch8"},{"issue":"6","key":"pdig.0001768.ref025","doi-asserted-by":"crossref","first-page":"1269","DOI":"10.1007\/s43441-023-00560-y","article-title":"Digital Health Technologies in Clinical Trials: An Ontology-Driven Analysis to Inform Digital Sustainability Policies","volume":"57","author":"SP Hey","year":"2023","journal-title":"Ther Innov Regul Sci"},{"key":"pdig.0001768.ref026","unstructured":"Organization WH. Classification of digital interventions, services and applications in health: A shared language to describe the uses of digital technology for health. 2023."},{"key":"pdig.0001768.ref027","unstructured":"Harrison TM, Moon S, Wang L, Fu S, Liu H. Digital solutions observed in clinical trials: A formative feasibility scoping review. In: AMIA Annual Symposium Proceedings, 2024."},{"key":"pdig.0001768.ref028","unstructured":"Colleges AOAM. Careers in medicine specialty profiles 2025."},{"issue":"3","key":"pdig.0001768.ref029","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1136\/jamia.2000.0070313","article-title":"Design and implementation of a national clinical trials registry","volume":"7","author":"AT McCray","year":"2000","journal-title":"J Am Med Inform Assoc"},{"key":"pdig.0001768.ref030","author":"World Health Organization","year":"2025"},{"key":"pdig.0001768.ref031","unstructured":"Administration USFD. Conducting clinical trials with decentralized elements. 2024."},{"key":"pdig.0001768.ref032","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1016\/j.jclinepi.2019.06.011","article-title":"Who is in this study, anyway? Guidelines for a useful Table 1","volume":"114","author":"E Hayes-Larson","year":"2019","journal-title":"Journal of clinical epidemiology"},{"key":"pdig.0001768.ref033"},{"issue":"1","key":"pdig.0001768.ref034","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1038\/s41746-023-00928-2","article-title":"Does clinical research account for diversity in deploying digital health technologies?","volume":"6","author":"NA Coss","year":"2023","journal-title":"NPJ Digit Med"},{"key":"pdig.0001768.ref035","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1016\/j.jclinepi.2021.12.005","article-title":"Tools to support the automation of systematic reviews: a scoping review","volume":"144","author":"H Khalil","year":"2022","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"pdig.0001768.ref036","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1186\/s13643-019-1074-9","article-title":"Toward systematic review automation: a practical guide to using machine learning tools in research synthesis","volume":"8","author":"IJ Marshall","year":"2019","journal-title":"Syst Rev"},{"key":"pdig.0001768.ref037","unstructured":"Agarwal S, Laradji IH, Charlin L, Pal C. LitLLM: A Toolkit for Scientific Literature Review. In: 2024. https:\/\/doi.org\/arXiv:240201788"},{"issue":"19","key":"pdig.0001768.ref038","doi-asserted-by":"crossref","first-page":"9103","DOI":"10.3390\/app14199103","article-title":"Automating Systematic Literature Reviews with Retrieval-Augmented Generation: A Comprehensive Overview","volume":"14","author":"B Han","year":"2024","journal-title":"Applied Sciences"},{"key":"pdig.0001768.ref039","article-title":"Automated Literature Review Using NLP Techniques and LLM-Based Retrieval-Augmented Generation","author":"NF Ali","year":"2024"},{"key":"pdig.0001768.ref040","article-title":"Automating research synthesis with domain-specific large language model fine-tuning","author":"T Susnjak","year":"2024"}],"container-title":["PLOS Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001768","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T17:25:44Z","timestamp":1791566744000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001768"}},"subtitle":[],"editor":[{"given":"Junhan","family":"Zhao","sequence":"first","affiliation":[{"name":"University of Chicago Pritzker School of Medicine","place":["UNITED STATES OF AMERICA"]}],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,10,9]]},"references-count":40,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2026,10,9]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pdig.0001768","relation":{},"ISSN":["2767-3170"],"issn-type":[{"value":"2767-3170","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,9]]}}}