{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T17:05:22Z","timestamp":1790355922410,"version":"4.1.0"},"reference-count":60,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Large language models (LLMs) have the potential to improve the efficiency of evidence synthesis, but their reliability in performing complex tasks such as risk-of-bias (ROB) assessment in randomized controlled trials (RCTs) remains unclear.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to evaluate whether LLMs can reliably assess ROB in RCTs using version 2 of the Cochrane ROB tool for randomized trials (ROB 2).<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n This study was conducted between December 28, 2024, and February 28, 2025, in adherence to American Association for Public Opinion Research reporting guidelines. Twenty-nine RCTs were selected from published Cochrane systematic reviews across diverse medical fields. We developed a structured prompt engineering framework that transformed ROB 2 decision trees into logical rules for the LLM. Each RCT was independently evaluated twice by ChatGPT, with Cochrane review authors\u2019 assessments serving as the reference standard for comparison. The main outcomes were the accuracy and consistency of ROB 2 assessments at both the domain and trial levels, evaluated using accuracy, sensitivity, specificity, and\n F<\/jats:italic>\n 1<\/jats:sub>\n -score. Consistency between the repeated assessments was quantified using the Cohen \u03ba and prevalence-adjusted, bias-adjusted \u03ba.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The LLM demonstrated a moderate aggregate domain accuracy of 73.1% (95% CI 64.7%-81.5%) in the first assessment and 75.9% (95% CI 66.3%-85.4%) in the second assessment. Domain-averaged sensitivity decreased from 61.4% (95% CI 48.1%\u201074.7%) to 53.4% (95% CI 41.7%\u201065.0%), whereas domain-averaged specificity increased from 75.8% (95% CI 65.3%-86.3%) to 81.1% (95%CI 67.1%-95%), indicating a conservative tendency in identifying a high ROB. Domain-level accuracy ranged from 62.1% to 87.9%, with the lowest accuracy observed in domain 1 and the lowest\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores observed in domain 2. Consistency between repeated assessments was high, with a mean agreement of 89.0% (SD 7.5%), and Cohen \u03ba values were 0.86, 0.39, 0.56, 0.84, and 0.85 in domains 1 to 5, respectively.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n In this exploratory study, ChatGPT demonstrated moderate accuracy and high consistency in assessing ROB in RCTs using the ROB 2. However, its reliability diminished in complex scenarios requiring interpretation of implicit narratives or behavioral nuance. These findings suggest that LLMs may support methodological evaluations in systematic reviews by acting as automated screeners to reduce reviewer burden, but current implementation still requires expert oversight, particularly for trials involving subjective outcomes or nonstandard reporting.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/84915","type":"journal-article","created":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T16:14:10Z","timestamp":1790352850000},"page":"e84915-e84915","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-2140-4044","authenticated-orcid":false,"given":"Yu-Ju","family":"Lai","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2376-8114","authenticated-orcid":false,"given":"Shen-Hua","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-4971-3476","authenticated-orcid":false,"given":"Jen-Wei","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,25]]},"reference":[{"issue":"10199","key":"R1","doi-asserted-by":"publisher","first-page":"633","DOI":"10.1016\/S0140-6736(19)31256-5","article-title":"High-quality evidence to inform clinical practice","volume":"394","author":"Fanaroff","journal-title":"Lancet"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2024-081903","article-title":"Core GRADE 1: overview of the Core GRADE approach","volume":"389","author":"Guyatt","journal-title":"BMJ"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.l4898","article-title":"RoB 2: a revised tool for assessing risk of bias in randomised trials","volume":"366","author":"Sterne","journal-title":"BMJ"},{"issue":"4","key":"R4","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","journal-title":"JAMA"},{"issue":"5","key":"R5","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.12687","article-title":"Assessing the risk of bias in randomized clinical trials with large language models","volume":"7","author":"Lai","journal-title":"JAMA Netw Open"},{"issue":"8","key":"R6","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1001\/jamasurg.2021.0543","article-title":"AAPOR reporting guidelines for survey studies","volume":"156","author":"Pitt","journal-title":"JAMA Surg"},{"key":"R7","unstructured":"OpenAI o3\u2011mini. OpenAI. URL: https:\/\/openai.com\/index\/openai-o3-mini\/ [Accessed 26-04-2025]"},{"key":"R8","unstructured":"Scope of human research projects exempt from institutional review board review [Article in Chinese]. The Executive Yuan Gazette. 2012. URL: https:\/\/gazette.nat.gov.tw\/EG_FileManager\/eguploadpub\/eg018127\/ch08\/type1\/gov70\/num35\/Eg.htm [Accessed 25-05-2025]"},{"key":"R9","unstructured":"RoB 2: a revised Cochrane risk-of-bias tool for randomized trials. Cochrane Methods Bias. URL: https:\/\/methods.cochrane.org\/bias\/resources\/rob-2-revised-cochrane-risk-bias-tool-randomized-trials [Accessed 26-04-2025]"},{"issue":"18","key":"R10","doi-asserted-by":"publisher","first-page":"2179","DOI":"10.1080\/14767058.2025.2509344","article-title":"Different routes and forms of uterotonics for treatment of retained placenta: a randomized clinical trial","volume":"30","author":"Maher","journal-title":"J Matern Fetal Neonatal Med"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","first-page":"2509344","DOI":"10.1080\/14767058.2025.2509344","article-title":"Statement of retraction: Different routes and forms of uterotonics for treatment of retained placenta: a randomized clinical trial","volume":"38","journal-title":"J Matern Fetal Neonatal Med"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-018-0493-y","article-title":"Comparison of methodological quality rating of systematic reviews on neuropathic pain using AMSTAR and R-AMSTAR","volume":"18","author":"Dosenovic","journal-title":"BMC Med Res Methodol"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2019.11.012","article-title":"Methodological and reporting quality in non-Cochrane systematic review updates could be improved: a comparative study","volume":"119","author":"Gao","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"R14","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1016\/j.cardfail.2013.11.004","article-title":"Continuous ultrafiltration for congestive heart failure: the CUORE trial","volume":"20","author":"Marenzi","journal-title":"J Card Fail"},{"issue":"8","key":"R15","doi-asserted-by":"publisher","first-page":"883","DOI":"10.1080\/08941939.2019.1710626","article-title":"Effect of pretreatment of S-ketamine on postoperative depression for breast cancer patients","volume":"34","author":"Liu","journal-title":"J Invest Surg"},{"issue":"2","key":"R16","doi-asserted-by":"publisher","first-page":"127","DOI":"10.2500\/aap.2011.32.3426","article-title":"Long-term treatment with fluticasone propionate\/salmeterol via Diskus improves asthma control versus fluticasone propionate alone","volume":"32","author":"Katial","journal-title":"Allergy Asthma Proc"},{"issue":"4","key":"R17","doi-asserted-by":"publisher","first-page":"480","DOI":"10.1016\/j.joca.2021.02.002","article-title":"Shoe-stiffening inserts for first metatarsophalangeal joint osteoarthritis: a randomised trial","volume":"29","author":"Munteanu","journal-title":"Osteoarthritis Cartilage"},{"issue":"10","key":"R18","doi-asserted-by":"publisher","DOI":"10.1001\/jamasurg.2018.2734","article-title":"Feasibility of formal mindfulness-based stress-resilience training among surgery interns: a randomized clinical trial","volume":"153","author":"Lebares","journal-title":"JAMA Surg"},{"issue":"6","key":"R19","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1590\/s0036-46652000000600005","article-title":"Evaluation of penicillin therapy in patients with leptospirosis and acute renal failure","volume":"42","author":"Daher","journal-title":"Rev Inst Med Trop Sao Paulo"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","first-page":"98","DOI":"10.1136\/gut.46.1.98","article-title":"Manometry based randomised trial of endoscopic sphincterotomy for sphincter of Oddi dysfunction","volume":"46","author":"Toouli","journal-title":"Gut"},{"issue":"1","key":"R21","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1111\/j.1751-7133.2011.00231.x","article-title":"Extracorporeal ultrafiltration vs. conventional diuretic therapy in advanced decompensated heart failure","volume":"18","author":"Hanna","journal-title":"Congest Heart Fail"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","DOI":"10.1186\/s12872-020-01733-5","article-title":"Efficacy and safety of early ultrafiltration in patients with acute decompensated heart failure with volume overload: a prospective, randomized, controlled clinical trial","volume":"20","author":"Hu","journal-title":"BMC Cardiovasc Disord"},{"issue":"4","key":"R23","doi-asserted-by":"publisher","first-page":"830","DOI":"10.1038\/s41409-019-0623-0","article-title":"Sertraline treatment decreased the serum levels of interleukin-6 and high-sensitivity C-reactive protein in hematopoietic stem cell transplantation patients with depression; a randomized double-blind, placebo-controlled clinical trial","volume":"55","author":"Tavakoli Ardakani","journal-title":"Bone Marrow Transplant"},{"issue":"21","key":"R24","doi-asserted-by":"publisher","first-page":"2041","DOI":"10.1056\/NEJMoa2019014","article-title":"Hydroxychloroquine with or without azithromycin in mild-to-moderate Covid-19","volume":"383","author":"Cavalcanti","journal-title":"N Engl J Med"},{"issue":"9","key":"R25","doi-asserted-by":"publisher","first-page":"850","DOI":"10.1056\/NEJMoa1511190","article-title":"Serious asthma events with budesonide plus formoterol vs. budesonide alone","volume":"375","author":"Peters","journal-title":"N Engl J Med"},{"issue":"3","key":"R26","doi-asserted-by":"publisher","first-page":"672","DOI":"10.1007\/s11606-019-05284-0","article-title":"Effects of a required large-group mindfulness meditation course on first-year medical students\u2019 mental health and quality of life: a randomized controlled trial","volume":"35","author":"Dami\u00e3o Neto","journal-title":"J Gen Intern Med"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1016\/j.eclinm.2020.100645","article-title":"Randomized double-blinded placebo-controlled trial of hydroxychloroquine with or without azithromycin for virologic cure of non-severe Covid-19","volume":"29","author":"Omrani","journal-title":"EClinicalMedicine"},{"issue":"4","key":"R28","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1080\/10401334.2014.945025","article-title":"Abridged mindfulness intervention to support wellness in first-year medical students","volume":"26","author":"Erogul","journal-title":"Teach Learn Med"},{"issue":"4","key":"R29","doi-asserted-by":"publisher","first-page":"106143","DOI":"10.1016\/j.ijantimicag.2020.106143","article-title":"Safety and effectiveness of azithromycin in patients with COVID-19: an open-label randomised trial","volume":"56","author":"Sekhavati","journal-title":"Int J Antimicrob Agents"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1245\/s10434-020-08710-4","article-title":"Total laparoscopic uncut Roux-en-Y for radical distal gastrectomy: an interim analysis of a randomized, controlled, clinical trial","volume":"28","author":"Wang","journal-title":"Ann Surg Oncol"},{"issue":"4","key":"R31","doi-asserted-by":"publisher","first-page":"440","DOI":"10.1192\/bjp.169.4.440","article-title":"Pharmacological treatment of depression in cancer patients. A placebo-controlled study of mianserin","volume":"169","author":"van Heeringen","journal-title":"Br J Psychiatry"},{"issue":"11","key":"R32","doi-asserted-by":"publisher","first-page":"2043","DOI":"10.1016\/j.jacc.2005.05.098","article-title":"Ultrafiltration versus usual care for hospitalized patients with heart failure: the Relief for Acutely Fluid-Overloaded Patients With Decompensated Congestive Heart Failure (RAPID-CHF) trial","volume":"46","author":"Bart","journal-title":"J Am Coll Cardiol"},{"issue":"10274","key":"R33","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1016\/S0140-6736(21)00149-5","article-title":"Azithromycin in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial","volume":"397","author":"RECOVERY Collaborative Group","journal-title":"Lancet"},{"issue":"4","key":"R34","doi-asserted-by":"publisher","first-page":"398","DOI":"10.1111\/aogs.12065","article-title":"No reduction of manual removal after misoprostol for retained placenta: a double-blind, randomized trial","volume":"92","author":"van Stralen","journal-title":"Acta Obstet Gynecol Scand"},{"issue":"7","key":"R35","doi-asserted-by":"publisher","first-page":"631","DOI":"10.1055\/s-2004-814538","article-title":"Influence of endoscopic papillary balloon dilation and endoscopic sphincterotomy on sphincter of Oddi function: a randomized controlled trial","volume":"36","author":"Takezawa","journal-title":"Endoscopy"},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2466-14-113","article-title":"Efficacy and safety of fluticasone furoate 100 \u03bcg and 200 \u03bcg once daily in the treatment of moderate-severe asthma in adults and adolescents: a 24-week randomised study","volume":"14","author":"Woodcock","journal-title":"BMC Pulm Med"},{"issue":"1","key":"R37","first-page":"79","volume":"25","author":"\u015eeker","journal-title":"Turk J Nephrol"},{"issue":"1","key":"R38","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/S2213-2600(20)30389-1","article-title":"Efficacy and safety of once-daily single-inhaler triple therapy (FF\/UMEC\/VI) versus FF\/VI in patients with inadequately controlled asthma (CAPTAIN): a double-blind, randomised, phase 3A trial","volume":"9","author":"Lee","journal-title":"Lancet Respir Med"},{"issue":"3","key":"R39","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1056\/NEJMoa062897","article-title":"Maintenance therapy with certolizumab pegol for Crohn\u2019s disease","volume":"357","author":"Schreiber","journal-title":"N Engl J Med"},{"issue":"1","key":"R40","doi-asserted-by":"publisher","DOI":"10.1186\/s13063-020-04593-8","article-title":"A multi-centre open-label two-arm randomised superiority clinical trial of azithromycin versus usual care in ambulatory COVID-19: study protocol for the ATOMIC2 trial","volume":"21","author":"Hinks","journal-title":"Trials"},{"issue":"20","key":"R41","doi-asserted-by":"publisher","first-page":"2101","DOI":"10.1001\/jama.2014.5220","article-title":"Effect of endoscopic sphincterotomy for suspected sphincter of Oddi dysfunction on pain-related disability following cholecystectomy: the EPISOD randomized clinical trial","volume":"311","author":"Cotton","journal-title":"JAMA"},{"issue":"1","key":"R42","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD013593.pub2","article-title":"Ultrafiltration for acute heart failure","volume":"1","author":"Srivastava","journal-title":"Cochrane Database Syst Rev"},{"issue":"3","key":"R43","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD011006.pub4","article-title":"Antidepressants for the treatment of depression in people with cancer","volume":"3","author":"Vita","journal-title":"Cochrane Database Syst Rev"},{"issue":"8","key":"R44","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD013797.pub2","article-title":"Addition of long-acting beta2 agonists or long-acting muscarinic antagonists versus doubling the dose of inhaled corticosteroids (ICS) in adolescents and adults with uncontrolled asthma with medium dose ICS: a systematic review and network meta-analysis","volume":"8","author":"Oba","journal-title":"Cochrane Database Syst Rev"},{"issue":"6","key":"R45","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD007809.pub3","article-title":"Non-surgical interventions for treating osteoarthritis of the big toe joint","volume":"6","author":"Munteanu","journal-title":"Cochrane Database Syst Rev"},{"issue":"12","key":"R46","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD013740.pub2","article-title":"Mindfulness-based psychological interventions for improving mental well-being in medical students and junior doctors","volume":"12","author":"Sekhar","journal-title":"Cochrane Database Syst Rev"},{"issue":"10","key":"R47","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD016147","article-title":"Uterotonics for management of retained placenta","volume":"10","author":"Sothornwit","journal-title":"Cochrane Database Syst Rev"},{"issue":"3","key":"R48","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD014960.pub2","article-title":"Antibiotics for treatment of leptospirosis","volume":"3","author":"Win","journal-title":"Cochrane Database Syst Rev"},{"issue":"3","key":"R49","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD014944.pub2","article-title":"Endoscopic sphincterotomy for adults with biliary sphincter of Oddi dysfunction","volume":"3","author":"Naing","journal-title":"Cochrane Database Syst Rev"},{"issue":"10","key":"R50","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD015025","article-title":"Antibiotics for the treatment of COVID-19","volume":"10","author":"Popp","journal-title":"Cochrane Database Syst Rev"},{"issue":"2","key":"R51","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD015014.pub2","article-title":"Uncut Roux-en-Y reconstruction after distal gastrectomy for gastric cancer","volume":"2","author":"Cai","journal-title":"Cochrane Database Syst Rev"},{"issue":"6","key":"R52","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.CD013747.pub2","article-title":"Certolizumab pegol for maintenance of medically induced remission in Crohn\u2019s disease","volume":"6","author":"Okabayashi","journal-title":"Cochrane Database Syst Rev"},{"key":"R53","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2016.08.012","article-title":"There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review","volume":"81","author":"Jordan","journal-title":"J Clin Epidemiol"},{"key":"R54","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2020.06.015","article-title":"The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application","volume":"126","author":"Minozzi","journal-title":"J Clin Epidemiol"},{"issue":"3","key":"R55","doi-asserted-by":"publisher","first-page":"144","DOI":"10.11919\/j.issn.1002-0829.215042","article-title":"Addressing risk of bias in trials of cognitive behavioral therapy","volume":"27","author":"Button","journal-title":"Shanghai Arch Psychiatry"},{"key":"R56","doi-asserted-by":"publisher","DOI":"10.3389\/fcvm.2022.830070","article-title":"Methodological quality and risk of bias assessment of cardiovascular disease research: analysis of randomized controlled trials published in 2017","volume":"9","author":"Baasan","journal-title":"Front Cardiovasc Med"},{"key":"R57","doi-asserted-by":"crossref","unstructured":"Pitre T Jassal T Talukdar JR Shahab M Ling M Zeraatkar D . ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: a methods study. medRxiv. Preprint posted online on Nov 22, 2023. [doi: 10.1101\/2023.11.19.23298727]","DOI":"10.1101\/2023.11.19.23298727"},{"key":"R58","unstructured":"Claude 3.7 Sonnet and Claude Code. Anthropic. 2025. URL: https:\/\/www.anthropic.com\/news\/claude-3-7-sonnet [Accessed 23-05-2025]"},{"key":"R59","unstructured":"Tool to assess risk of bias in randomized controlled trials. DistillerSR. URL: https:\/\/www.distillersr.com\/resources\/methodological-resources\/tool-to-assess-risk-of-bias-in-randomized-controlled-trials-distillersr [Accessed 23-05-2025]"},{"issue":"3","key":"R60","doi-asserted-by":"publisher","first-page":"491","DOI":"10.1017\/rsm.2025.12","article-title":"Exploring the potential of Claude 2 for risk of bias assessment: using a large language model to assess randomized controlled trials with RoB 2","volume":"16","author":"Eisele-Metzger","journal-title":"Res Synth Methods"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T16:14:14Z","timestamp":1790352854000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e84915"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,25]]},"references-count":60,"URL":"https:\/\/doi.org\/10.2196\/84915","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,25]]},"article-number":"v28i10199e84915"}}