{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T12:01:58Z","timestamp":1777636918519,"version":"3.51.4"},"reference-count":60,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2025,1,21]],"date-time":"2025-01-21T00:00:00Z","timestamp":1737417600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U24 CA265879"],"award-info":[{"award-number":["U24 CA265879"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100004325","name":"AstraZeneca","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100004325","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,4,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>Data extraction from the published literature is the most laborious step in conducting living systematic reviews (LSRs). We aim to build a generalizable, automated data extraction workflow leveraging large language models (LLMs) that mimics the real-world 2-reviewer process.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>A dataset of 10 trials (22 publications) from a published LSR was used, focusing on 23 variables related to trial, population, and outcomes data. The dataset was split into prompt development (n\u2009=\u20095) and held-out test sets (n\u2009=\u200917). GPT-4-turbo and Claude-3-Opus were used for data extraction. Responses from the 2 LLMs were considered concordant if they were the same for a given variable. The discordant responses from each LLM were provided to the other LLM for cross-critique. Accuracy, ie, the total number of correct responses divided by the total number of responses, was computed to assess performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>In the prompt development set, 110 (96%) responses were concordant, achieving an accuracy of 0.99 against the gold standard. In the test set, 342 (87%) responses were concordant. The accuracy of the concordant responses was 0.94. The accuracy of the discordant responses was 0.41 for GPT-4-turbo and 0.50 for Claude-3-Opus. Of the 49 discordant responses, 25 (51%) became concordant after cross-critique, increasing accuracy to 0.76.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Concordant responses by the LLMs are likely to be accurate. In instances of discordant responses, cross-critique can further increase the accuracy.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Large language models, when simulated in a collaborative, 2-reviewer workflow, can extract data with reasonable performance, enabling truly \u201cliving\u201d systematic reviews.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocae325","type":"journal-article","created":{"date-parts":[[2025,1,3]],"date-time":"2025-01-03T23:17:38Z","timestamp":1735946258000},"page":"638-647","source":"Crossref","is-referenced-by-count":30,"title":["Collaborative large language models for automated data extraction in living systematic reviews"],"prefix":"10.1093","volume":"32","author":[{"given":"Muhammad Ali","family":"Khan","sequence":"first","affiliation":[{"name":"Department of Medicine, Mayo Clinic , Phoenix, AZ, 85054,","place":["United States"]}]},{"given":"Umair","family":"Ayub","sequence":"additional","affiliation":[{"name":"Department of Medicine, Mayo Clinic , Phoenix, AZ, 85054,","place":["United States"]}]},{"given":"Syed Arsalan Ahmed","family":"Naqvi","sequence":"additional","affiliation":[{"name":"Department of Medicine, Mayo Clinic , Phoenix, AZ, 85054,","place":["United States"]}]},{"given":"Kaneez Zahra Rubab","family":"Khakwani","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Arizona , Tucson, AZ, 85721,","place":["United States"]}]},{"given":"Zaryab bin Riaz","family":"Sipra","sequence":"additional","affiliation":[{"name":"Department of Medicine and Surgery, Rashid Latif Medical College , Lahore, Punjab, 54000,","place":["Pakistan"]}]},{"given":"Ammad","family":"Raina","sequence":"additional","affiliation":[{"name":"Department of Medicine, Canyon Vista Hospital , Sierra Vista, AZ, 85635,","place":["United States"]}]},{"given":"Sihan","family":"Zhou","sequence":"additional","affiliation":[{"name":"Department of Medicine, Mayo Clinic , Phoenix, AZ, 85054,","place":["United States"]}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1312-4195","authenticated-orcid":false,"given":"Huan","family":"He","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics and Data Science, Yale University , New Haven, CT, 06520,","place":["United States"]}]},{"given":"Amir","family":"Saeidi","sequence":"additional","affiliation":[{"name":"Department of Medicine, Mayo Clinic , Phoenix, AZ, 85054,","place":["United States"]},{"name":"Department of Computing and Augmented Intelligence, Arizona State University , Tempe, AZ, 85287,","place":["United States"]}]},{"given":"Bashar","family":"Hasan","sequence":"additional","affiliation":[{"name":"Department of Medicine, Mayo Clinic , Rochester, MN, 55905,","place":["United States"]}]},{"given":"Robert Bryan","family":"Rumble","sequence":"additional","affiliation":[{"name":"American Society of Clinical Oncology , Alexandria, VA, 22314,","place":["United States"]}]},{"given":"Danielle S","family":"Bitterman","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology, Dana-Farber Cancer Institute , Boston, MA, 02215,","place":["United States"]}]},{"given":"Jeremy L","family":"Warner","sequence":"additional","affiliation":[{"name":"Department of Medicine, Brown University , Providence, RI, 02912,","place":["United States"]},{"name":"Department of Biostatistics, Brown University , Providence, RI, 02912,","place":["United States"]},{"name":"Department of Medicine, Rhode Island Hospital , Providence, RI, 02903,","place":["United States"]},{"name":"Center for Clinical Cancer Informatics and Data Science, Legorreta Cancer Center, Brown University , Providence, RI, 02912,","place":["United States"]}]},{"given":"Jia","family":"Zou","sequence":"additional","affiliation":[{"name":"Department of Computing and Augmented Intelligence, Arizona State University , Tempe, AZ, 85287,","place":["United States"]}]},{"given":"Amye J","family":"Tevaarwerk","sequence":"additional","affiliation":[{"name":"Department of Oncology, Mayo Clinic , Rochester, MN, 55905,","place":["United States"]}]},{"given":"Konstantinos","family":"Leventakos","sequence":"additional","affiliation":[{"name":"Department of Medicine, Mayo Clinic , Rochester, MN, 55905,","place":["United States"]}]},{"given":"Kenneth L","family":"Kehl","sequence":"additional","affiliation":[{"name":"Department of Medicine, Dana-Farber Cancer Institute , Boston, MA, 02215,","place":["United States"]}]},{"given":"Jeanne M","family":"Palmer","sequence":"additional","affiliation":[{"name":"Department of Medicine, Mayo Clinic , Phoenix, AZ, 85054,","place":["United States"]}]},{"given":"Mohammad Hassan","family":"Murad","sequence":"additional","affiliation":[{"name":"Department of Medicine, Mayo Clinic , Rochester, MN, 55905,","place":["United States"]}]},{"given":"Chitta","family":"Baral","sequence":"additional","affiliation":[{"name":"Department of Computing and Augmented Intelligence, Arizona State University , Tempe, AZ, 85287,","place":["United States"]}]},{"given":"Irbaz bin","family":"Riaz","sequence":"additional","affiliation":[{"name":"Department of Medicine, Mayo Clinic , Phoenix, AZ, 85054,","place":["United States"]},{"name":"Department of Artificial Intelligence and Informatics , Mayo Clinic, Rochester, MN, 55905,","place":["United States"]}]}],"member":"286","published-online":{"date-parts":[[2025,1,21]]},"reference":[{"key":"2025041716421655100_ocae325-B1","author":"Burky","year":"2022"},{"key":"2025041716421655100_ocae325-B2","author":"Simmonds"},{"key":"2025041716421655100_ocae325-B3","author":"Riaz"},{"key":"2025041716421655100_ocae325-B4","author":"Riaz"},{"key":"2025041716421655100_ocae325-B5","author":"Riaz"},{"key":"2025041716421655100_ocae325-B6","author":"Riaz"},{"key":"2025041716421655100_ocae325-B7","author":"Riaz"},{"key":"2025041716421655100_ocae325-B8","doi-asserted-by":"publisher","first-page":"e23","DOI":"10.1200\/JCO.23.02746","article-title":"Therapy for stage IV non-small cell lung cancer without driver alterations: ASCO living guideline, version 2023.3","volume":"42","author":"Jaiyesimi","year":"2024","journal-title":"J Clin Oncol"},{"key":"2025041716421655100_ocae325-B9","doi-asserted-by":"publisher","first-page":"e005427","DOI":"10.1136\/bmjgh-2021-005427","article-title":"Characterising long COVID: a living systematic review","volume":"6","author":"Michelen","year":"2021","journal-title":"BMJ Glob Health"},{"key":"2025041716421655100_ocae325-B10","author":"Morales-Plaza"},{"key":"2025041716421655100_ocae325-B11","author":"Mathes"},{"key":"2025041716421655100_ocae325-B12","author":"Michelson"},{"key":"2025041716421655100_ocae325-B13","author":"Singh","year":"2018"},{"key":"2025041716421655100_ocae325-B14","author":"Schmidt"},{"key":"2025041716421655100_ocae325-B15","author":"Jonnalagadda"},{"key":"2025041716421655100_ocae325-B16","author":"Kim"},{"key":"2025041716421655100_ocae325-B17","author":"Blaizot"},{"key":"2025041716421655100_ocae325-B18","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1561\/2200000013","article-title":"An introduction to conditional random fields","volume":"4","author":"Sutton","year":"2012","journal-title":"FNT Mach Learn"},{"key":"2025041716421655100_ocae325-B19","author":"Jeong","year":"2024"},{"key":"2025041716421655100_ocae325-B20","doi-asserted-by":"publisher","author":"Xu","DOI":"10.1007\/s11704-024-40555-y"},{"key":"2025041716421655100_ocae325-B21","author":"Zhou","year":"2022"},{"key":"2025041716421655100_ocae325-B22","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adva Neural Inf Process Syst"},{"key":"2025041716421655100_ocae325-B23","first-page":"8469","author":"Driess"},{"key":"2025041716421655100_ocae325-B24","doi-asserted-by":"publisher","first-page":"172","author":"Singhal","DOI":"10.1038\/s41586-023-06291-2"},{"key":"2025041716421655100_ocae325-B25","author":"Agrawal"},{"key":"2025041716421655100_ocae325-B26","doi-asserted-by":"publisher","author":"Hasan","DOI":"10.1136\/bmjebm-2023-112597"},{"key":"2025041716421655100_ocae325-B27","author":"Lee"},{"key":"2025041716421655100_ocae325-B28","author":"Wan","year":"2023"},{"key":"2025041716421655100_ocae325-B29","author":"Reynolds"},{"key":"2025041716421655100_ocae325-B30","author":"Akinwande","year":"2023"},{"key":"2025041716421655100_ocae325-B31","author":"Open AI","year":"2024"},{"key":"2025041716421655100_ocae325-B32","author":"Anthropic","year":"2024"},{"key":"2025041716421655100_ocae325-B33","author":"Wei","year":"2023"},{"key":"2025041716421655100_ocae325-B34","doi-asserted-by":"publisher","author":"Kartchner","year":"2023","DOI":"10.18653\/v1\/2023.bionlp-1.37"},{"key":"2025041716421655100_ocae325-B35","doi-asserted-by":"publisher","first-page":"576","author":"Gartlehner","DOI":"10.1002\/jrsm.1710"},{"key":"2025041716421655100_ocae325-B36","doi-asserted-by":"publisher","first-page":"818","author":"Konet","DOI":"10.1002\/jrsm.1732"},{"key":"2025041716421655100_ocae325-B37","author":"Reason"},{"key":"2025041716421655100_ocae325-B38","author":"PyMuPDF 1.24.9","year":"2024"},{"key":"2025041716421655100_ocae325-B39","author":"tiktoken 0.7.0","year":"2024"},{"key":"2025041716421655100_ocae325-B40","author":"Zhang","year":"2024"},{"key":"2025041716421655100_ocae325-B41","doi-asserted-by":"crossref","first-page":"4616","DOI":"10.1109\/TCSVT.2023.3245584","article-title":"Understanding and mitigating overfitting in prompt tuning for vision-language models","volume":"33","author":"Ma","year":"2023","journal-title":"IEEE Trans Circuits Syst Video Technol"},{"key":"2025041716421655100_ocae325-B42","author":"Rosner"},{"key":"2025041716421655100_ocae325-B43","author":"Siemieniuk"},{"key":"2025041716421655100_ocae325-B44","author":"Acharya","year":"2024"},{"key":"2025041716421655100_ocae325-B45","author":"Living Interactive Systematic Reviews"},{"key":"2025041716421655100_ocae325-B46","author":"Fuentes"},{"key":"2025041716421655100_ocae325-B47","doi-asserted-by":"publisher","first-page":"4567","DOI":"10.1200\/JCO.2018.36.15_suppl.4567","article-title":"A meta-analysis of randomized controlled trials for efficacy and safety of vascular endothelial growth factor tyrosine kinase inhibitors (VEGF-TKIs) adjuvant therapy in high-risk renal cell cancer (RCC)","volume":"36","author":"Riaz","year":"2018","journal-title":"J Clin Oncol"},{"key":"2025041716421655100_ocae325-B48","doi-asserted-by":"publisher","first-page":"708","DOI":"10.1200\/JCO.2020.38.6_suppl.708","article-title":"Adjuvant therapy in high-risk renal cell cancer: a systematic review and cumulative meta-analysis","volume":"38","author":"Siddiqi","year":"2020","journal-title":"J Clin Oncol"},{"key":"2025041716421655100_ocae325-B49","doi-asserted-by":"publisher","first-page":"5088","DOI":"10.1200\/JCO.2023.41.16_suppl.5088","article-title":"The role of volume of disease for treatment selection in patients with metastatic castration sensitive prostate cancer (mCSPC): a living meta-analysis","volume":"41","author":"Naqvi","year":"2023","journal-title":"J Clin Oncol"},{"key":"2025041716421655100_ocae325-B50","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1200\/JCO.2023.41.6_suppl.195","article-title":"Deep prostate-specific antigen response and overall survival in patients with metastatic castration-sensitive prostate cancer: a systematic review and meta-analysis","volume":"41","author":"Naqvi","year":"2023","journal-title":"J Clin Oncol"},{"key":"2025041716421655100_ocae325-B51","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1200\/JCO.2023.41.6_suppl.193","article-title":"A living interactive systematic review and network meta-analysis evaluating systemic therapies in metastatic castration-sensitive prostate cancer (mCSPC)","volume":"41","author":"Riaz","year":"2023","journal-title":"J Clin Oncol"},{"key":"2025041716421655100_ocae325-B52","doi-asserted-by":"publisher","first-page":"5083","DOI":"10.1200\/JCO.2022.40.16_suppl.5083","article-title":"Indirect comparisons of triplet therapy as compared to novel hormonal therapy doublets in patients with metastatic castration sensitive prostate cancer","volume":"40","author":"Naqvi","year":"2022","journal-title":"J Clin Oncol"},{"key":"2025041716421655100_ocae325-B53","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1200\/JCO.2022.40.6_suppl.132","article-title":"First-line treatment options in metastatic castration-sensitive prostate cancer: a systematic review and network meta-analysis","volume":"40","author":"Riaz","year":"2022","journal-title":"J Clin Oncol"},{"key":"2025041716421655100_ocae325-B54","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1200\/JCO.2022.40.6_suppl.136","article-title":"Triplet therapy in metastatic castration-sensitive prostate cancer: a systematic review and meta-analysis","volume":"40","author":"Naqvi","year":"2022","journal-title":"J Clin Oncol"},{"key":"2025041716421655100_ocae325-B55","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1200\/JCO.2024.42.4_suppl.173","article-title":"First-line PARP inhibitors (PARPi) with androgen receptor pathway inhibitors (ARPi) in metastatic castration-resistant prostate cancer (mCRPC): a systematic review and meta-analysis","volume":"42","author":"Khan","year":"2024","journal-title":"J Clin Oncol"},{"key":"2025041716421655100_ocae325-B56","doi-asserted-by":"publisher","first-page":"e17059","DOI":"10.1200\/JCO.2024.42.16_suppl.e17059","article-title":"Efficacy of first line (1L) poly ADP-ribose polymerase inhibitors (PARPi) in combination with androgen pathway inhibitors (API) by homologous recombination repair (HRR) genes in metastatic castration-resistant prostate cancer (mCRPC): a living meta-analysis","volume":"42","author":"Bibi","year":"2024","journal-title":"J Clin Oncol"},{"key":"2025041716421655100_ocae325-B57","doi-asserted-by":"publisher","first-page":"6433","DOI":"10.1182\/blood-2023-188107","article-title":"Toxicity profile of Janus kinase inhibitors (JAKi) in patients with myelofibrosis: a report from a living systematic review and meta-analysis","volume":"142","author":"Khan","year":"2023","journal-title":"Blood"},{"key":"2025041716421655100_ocae325-B58"},{"key":"2025041716421655100_ocae325-B59","author":"Huang","year":"2024"},{"key":"2025041716421655100_ocae325-B60","first-page":"75993","article-title":"On the planning abilities of large language models\u2014a critical investigation","volume":"36","author":"Valmeekam","year":"2023","journal-title":"Adv Neural Inf Process Syst"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/4\/638\/61530064\/ocae325.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/4\/638\/61530064\/ocae325.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,17]],"date-time":"2025-04-17T16:42:25Z","timestamp":1744908145000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/4\/638\/7965841"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,21]]},"references-count":60,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,1,21]]},"published-print":{"date-parts":[[2025,4,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae325","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2024.09.20.24314108","asserted-by":"object"}]},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,4]]},"published":{"date-parts":[[2025,1,21]]}}}