{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:47:38Z","timestamp":1782841658552,"version":"3.54.5"},"reference-count":54,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Despite the transformative potential of large language models (LLMs) in health care, the rapid development of these tools has outpaced their rigorous evaluation. While artificial intelligence\u2013specific reporting guidelines have been developed to address standardized reporting of artificial intelligence studies, there is currently no specific tool available for risk of bias assessment of LLM question-answer (QA) studies. Existing risk-of-bias tools for medical research are not well suited to the unique challenges of evaluating LLM-QA studies, which creates a critical gap in assessing their safety and effectiveness.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aims to develop the Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB) for LLM-QA studies to systematically evaluate the validity and risk of bias in LLM-QA studies.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We conducted 2 literature reviews. The first was on quality assessment tools for LLM-QA studies, and the second was on LLM-QA studies, which informed the first draft of the AQAT:RoB. The draft AQAT:ROB was further refined through a prespecified iterative process of modified Delphi, consensus meeting, and validation. The first Delphi process occurred between May 1 and May 20, 2025, and the first consensus meeting was held on May 22. The first round of validation was completed by 4 evaluators, who were not part of the consensus meeting, on 16 randomly selected studies. As this first round of validation surpassed our a priori threshold of \u226580% agreement and a Cohen \u03ba of \u22650.61 between evaluators, no further rounds of development and validation were undertaken. A second Delphi process occurred between February 20 and February 23, 2026, to vote on postpilot changes in response to peer review.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The AQAT:RoB consists of 5 high-level domains (Questions, Reference Answers, LLM Answers, Evaluators, Outcomes). These domains are subdivided into 9 subdomains. Each subdomain includes at least one \u201cSupport for Judgment\u201d and at least one \u201cType of Bias\u201d and is to be rated \u201clow,\u201d \u201chigh,\u201d or \u201cunclear\u201d for risk of bias. A pilot evaluation was completed by internal validators who were not part of the consensus discussion and were asked to complete the AQAT:RoB form for each assigned study. Each of the 16 studies was evaluated by 2 evaluators independently. Pilot validation showed a percent agreement of 86.1% and a Cohen \u03ba of 0.70 between assessors.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The AQAT:RoB demonstrates promising initial reliability for assessing the validity or risk of bias in LLM-QA studies. The tool will benefit from future refinements, external validation, and periodic updates to keep pace with evolving technology.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/87057","type":"journal-article","created":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T04:10:05Z","timestamp":1772079005000},"page":"e87057-e87057","source":"Crossref","is-referenced-by-count":1,"title":["The Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB) for the Evaluation of Medical Large Language Model Question-Answer Studies: Development and Pilot Validation"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9858-5398","authenticated-orcid":false,"given":"Carrie","family":"Ye","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2340-4708","authenticated-orcid":false,"given":"Joseph Ross","family":"Mitchell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2146-507X","authenticated-orcid":false,"given":"Daniel C","family":"Baumgart","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3064-8118","authenticated-orcid":false,"given":"Zechen","family":"Ma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-2904-0521","authenticated-orcid":false,"given":"Angela Lim","family":"Fung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-4813-3541","authenticated-orcid":false,"given":"Daniela Garcia","family":"Orellana","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4329-2392","authenticated-orcid":false,"given":"Juel","family":"Chowdhury","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-0643-0945","authenticated-orcid":false,"given":"Abdullah","family":"Abass","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6718-1393","authenticated-orcid":false,"given":"Steven","family":"Katz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5314-2297","authenticated-orcid":false,"given":"Jacob L","family":"Jaremko","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4219-1699","authenticated-orcid":false,"given":"Pierre","family":"Boulanger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3062-5488","authenticated-orcid":false,"given":"Claire E H","family":"Barber","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5658-3573","authenticated-orcid":false,"given":"Gillian","family":"Lemermeyer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7155-2297","authenticated-orcid":false,"given":"Hosna","family":"Jabbari","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7753-4295","authenticated-orcid":false,"given":"Lili","family":"Mou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5309-9786","authenticated-orcid":false,"given":"Maryam","family":"Mirzaei","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-6324-7237","authenticated-orcid":false,"given":"Mary Waithera Beckett","family":"Githumbi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0486-0174","authenticated-orcid":false,"given":"Puneeta","family":"Tandon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0739-2946","authenticated-orcid":false,"given":"Randy","family":"Goebel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-1162-0259","authenticated-orcid":false,"given":"Rhys","family":"Clark","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9273-8798","authenticated-orcid":false,"given":"Whitney","family":"Hung","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7252-6798","authenticated-orcid":false,"given":"Marjan","family":"Abbasi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5673-8210","authenticated-orcid":false,"given":"Farhad","family":"Maleki","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8611-1056","authenticated-orcid":false,"given":"Scott","family":"Klarenbach","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2776-6036","authenticated-orcid":false,"given":"Mohamed","family":"Abdalla","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,4,8]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","DOI":"10.1186\/s12929-025-01131-z","article-title":"Impact of large language model (ChatGPT) in healthcare: an umbrella review and evidence synthesis","volume":"32","author":"Iqbal","journal-title":"J Biomed Sci"},{"issue":"4","key":"R2","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","journal-title":"JAMA"},{"issue":"9","key":"R3","doi-asserted-by":"publisher","first-page":"866","DOI":"10.1001\/jama.2023.14217","article-title":"Creation and adoption of large language models in medicine","volume":"330","author":"Shah","journal-title":"JAMA"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.l4898","article-title":"RoB 2: a revised tool for assessing risk of bias in randomised trials","volume":"366","author":"Sterne","journal-title":"BMJ"},{"issue":"8","key":"R5","doi-asserted-by":"publisher","first-page":"529","DOI":"10.7326\/0003-4819-155-8-201110180-00009","article-title":"QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies","volume":"155","author":"Whiting","journal-title":"Ann Intern Med"},{"key":"R6","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-14-45","article-title":"Newcastle-Ottawa Scale: comparing reviewers\u2019 to authors\u2019 assessments","volume":"14","author":"Lo","journal-title":"BMC Med Res Methodol"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.envint.2024.108602","article-title":"A tool to assess risk of bias in non-randomized follow-up studies of exposure effects (ROBINS-E)","volume":"186","author":"Higgins","journal-title":"Environ Int"},{"issue":"7","key":"R8","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2020-048008","article-title":"Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence","volume":"11","author":"Collins","journal-title":"BMJ Open"},{"issue":"9","key":"R9","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.35377","article-title":"APPRAISE-AI tool for quantitative evaluation of AI studies for clinical decision support","volume":"6","author":"Kwong","journal-title":"JAMA Netw Open"},{"issue":"2","key":"R10","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.57879","article-title":"Large language models for chatbot health advice studies: a systematic review","volume":"8","author":"Huo","journal-title":"JAMA Netw Open"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.2196\/64486","article-title":"Accuracy of large language models when answering clinical research questions: systematic review and network meta-analysis","volume":"27","author":"Wang","journal-title":"J Med Internet Res"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2023-078378","article-title":"TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods","volume":"385","author":"Collins","journal-title":"BMJ"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","DOI":"10.1136\/bmjmed-2025-001632","article-title":"Reporting guideline for chatbot health advice studies: the Chatbot Assessment Reporting Tool (CHART) statement","volume":"4","author":"CHART Collaborative","journal-title":"BMJ Med"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.2196\/52508","article-title":"Consolidated reporting guidelines for prognostic and diagnostic machine learning models (CREMLS)","volume":"26","author":"El Emam","journal-title":"J Med Internet Res"},{"issue":"1","key":"R15","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","article-title":"The TRIPOD-LLM reporting guideline for studies using large language models","volume":"31","author":"Gallifant","journal-title":"Nat Med"},{"key":"R16","unstructured":"The use of large language models in patient education interventions: a systematic review. PROSPERO. URL: https:\/\/www.crd.york.ac.uk\/PROSPERO\/view\/CRD42023461630 [Accessed 03-09-2025]"},{"key":"R17","unstructured":"TCPS 2 (2022) \u2013 chapter 2: scope and approach. Government of Canada. 2023. URL: https:\/\/ethics.gc.ca\/eng\/tcps2-eptc2_2022_chapter2-chapitre2.html [Accessed 27-03-2026]"},{"key":"R18","unstructured":"Tools in development. Latitudes Network. 2023. URL: https:\/\/www.latitudes-network.org\/library\/tools-in-development\/ [Accessed 02-09-2025]"},{"key":"R19","unstructured":"Latitudes Network. 2023. URL: https:\/\/www.latitudes-network.org\/ [Accessed 16-09-2025]"},{"key":"R20","unstructured":"RoB 2 tool. Risk of bias tools. URL: https:\/\/sites.google.com\/site\/riskofbiastool\/welcome\/rob-2-0-tool?authuser=0 [Accessed 18-06-2025]"},{"key":"R21","unstructured":"QUADAS | Bristol medical school: population health sciences. University of Bristol. URL: https:\/\/www.bristol.ac.uk\/population-health-sciences\/projects\/quadas\/ [Accessed 18-06-2025]"},{"key":"R22","unstructured":"Alberta Machine Intelligence Institute. URL: https:\/\/www.amii.ca\/ [Accessed 02-09-2025]"},{"key":"R23","unstructured":"AI + Health Hub. University of Alberta. URL: https:\/\/www.ualberta.ca\/en\/health-sciences\/research\/ai-and-health-hub.html [Accessed 02-09-2025]"},{"key":"R24","unstructured":"Centre for health informatics. University of Calgary. URL: https:\/\/cumming.ucalgary.ca\/centres\/centre-health-informatics [Accessed 03-09-2025]"},{"issue":"8","key":"R25","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2023.30320","article-title":"Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions","volume":"6","author":"Bernstein","journal-title":"JAMA Netw Open"},{"issue":"7","key":"R26","doi-asserted-by":"publisher","first-page":"1368","DOI":"10.1038\/s41433-023-02906-0","article-title":"Reliability and accuracy of artificial intelligence ChatGPT in providing information on ophthalmic diseases and management to patients","volume":"38","author":"Cappellani","journal-title":"Eye"},{"issue":"4","key":"R27","doi-asserted-by":"publisher","first-page":"574","DOI":"10.1177\/10556656231223596","article-title":"Easing the burden on caregivers\u2014applications of artificial intelligence for physicians and caregivers of children with cleft lip and palate","volume":"62","author":"Chaker","journal-title":"Cleft Palate Craniofac J"},{"key":"R28","doi-asserted-by":"crossref","unstructured":"Chen S Kann BH Foote MB et al. The utility of ChatGPT for cancer treatment information. medRxiv. Preprint posted online on Mar 23, 2023. [doi: 10.1101\/2023.03.16.23287316]","DOI":"10.1101\/2023.03.16.23287316"},{"issue":"3","key":"R29","doi-asserted-by":"publisher","first-page":"575","DOI":"10.1016\/j.fertnstert.2023.05.151","article-title":"The promise and peril of using a large language model to obtain clinical information: ChatGPT performs strongly as a fertility counseling tool with limitations","volume":"120","author":"Chervenak","journal-title":"Fertil Steril"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.1016\/j.urology.2023.05.040","article-title":"Can ChatGPT, an artificial intelligence language model, provide accurate and high-quality patient information on prostate cancer?","volume":"180","author":"Coskun","journal-title":"Urology"},{"issue":"11","key":"R31","doi-asserted-by":"publisher","first-page":"2717","DOI":"10.1007\/s11255-023-03729-4","article-title":"The utility of the ChatGPT artificial intelligence tool for patient education and enquiry in robotic radical prostatectomy","volume":"55","author":"Gabriel","journal-title":"Int Urol Nephrol"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.1016\/j.bjps.2023.07.039","article-title":"Dr. ChatGPT will see you now: how do Google and ChatGPT compare in answering patient questions on breast reconstruction?","volume":"85","author":"Liu","journal-title":"J Plast Reconstr Aesthet Surg"},{"issue":"2","key":"R33","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1097\/IJG.0000000000002338","article-title":"Can ChatGPT aid clinicians in educating patients on the surgical management of glaucoma?","volume":"33","author":"Kianian","journal-title":"J Glaucoma"},{"issue":"10","key":"R34","doi-asserted-by":"publisher","first-page":"1760","DOI":"10.1016\/j.jvir.2023.05.037","article-title":"Evaluation of an artificial intelligence chatbot for delivery of IR patient education material: a comparison with societal website content","volume":"34","author":"McCarthy","journal-title":"J Vasc Interv Radiol"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","DOI":"10.3390\/bioengineering11010057","article-title":"ChatGPT in occupational medicine: a comparative study with human experts","volume":"11","author":"Padovan","journal-title":"Bioengineering (Basel)"},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.2147\/RRU.S440633","article-title":"ChatGPT: is this patient education tool for urological malignancies readable for the general population?","volume":"16","author":"Thia","journal-title":"Res Rep Urol"},{"issue":"4","key":"R37","doi-asserted-by":"publisher","first-page":"3465","DOI":"10.1007\/s12070-024-04729-1","article-title":"Examining the performance of ChatGPT 3.5 and Microsoft Copilot in otolaryngology: a comparative study with otolaryngologists\u2019 evaluation","volume":"76","author":"Mayo-Y\u00e1\u00f1ez","journal-title":"Indian J Otolaryngol Head Neck Surg"},{"issue":"10","key":"R38","doi-asserted-by":"publisher","DOI":"10.3390\/jcm13102832","article-title":"AI in hand surgery: assessing large language models in the classification and management of hand injuries","volume":"13","author":"Pressman","journal-title":"J Clin Med"},{"issue":"10","key":"R39","doi-asserted-by":"publisher","first-page":"485","DOI":"10.1016\/j.acvd.2023.07.009","article-title":"Relevance of medical information obtained from ChatGPT: are large language models friends or foes?","volume":"116","author":"Mesnier","journal-title":"Arch Cardiovasc Dis"},{"issue":"8","key":"R40","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.43106","article-title":"The use of AI in diagnosing diseases and providing management plans: a consultation on cardiovascular disorders with ChatGPT","volume":"15","author":"Rizwan","journal-title":"Cureus"},{"key":"R41","unstructured":"Haahr M . RANDOM.ORG. URL: https:\/\/www.random.org\/ [Accessed 03-09-2025]"},{"issue":"1","key":"R42","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","journal-title":"Biometrics"},{"key":"R43","unstructured":"Alberta Quality Assessment Tools. URL: https:\/\/aqat.ai\/#tools [Accessed 18-03-2026]"},{"key":"#cr-split#-R44.1","doi-asserted-by":"crossref","unstructured":"Abdalla M Abdalla M . The grey hoodie project: big tobacco, big tech, and the threat on academic integrity. Presented at: AIES '21: Proceedings of the 2021 AAAI\/ACM Conference on AI, Ethics, and Society. May 19-21, 2021","DOI":"10.1145\/3461702.3462563"},{"key":"#cr-split#-R44.2","doi-asserted-by":"crossref","unstructured":"ACM. Virtual event. 2021.[doi: 10.1145\/3461702.3462563]","DOI":"10.1145\/3461702.3462563"},{"key":"#cr-split#-R45.1","doi-asserted-by":"crossref","unstructured":"Aitken W Abdalla M Rudie K Stinson C . Collaboration or corporate capture? Quantifying NLP's reliance on industry artifacts and contributions. In: Ku LW Martins A Srikumar V , editors. Presented at: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Aug 11-16, 2024:Association for Computational Linguistics. 3433-3448","DOI":"10.18653\/v1\/2024.acl-long.188"},{"key":"#cr-split#-R45.2","doi-asserted-by":"crossref","unstructured":"Bangkok, Thailand. 2024.[doi: 10.18653\/v1\/2024.acl-long.188]","DOI":"10.18653\/v1\/2024.acl-long.188"},{"key":"#cr-split#-R46.1","doi-asserted-by":"crossref","unstructured":"Young M Katell M Krafft PM . Confronting power and corporate capture at the FAccT conference. Presented at: FAccT '22: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. Jun 21-24, 2022","DOI":"10.1145\/3531146.3533194"},{"key":"#cr-split#-R46.2","doi-asserted-by":"crossref","unstructured":"ACM. Seoul, South Korea. 2022.[doi: 10.1145\/3531146.3533194]","DOI":"10.1145\/3531146.3533194"},{"issue":"1","key":"R47","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01786-w","article-title":"Framework for bias evaluation in large language models in healthcare settings","volume":"8","author":"Templin","journal-title":"NPJ Digit Med"},{"key":"R48","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2020.06.015","article-title":"The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application","volume":"126","author":"Minozzi","journal-title":"J Clin Epidemiol"},{"issue":"9","key":"R49","doi-asserted-by":"publisher","first-page":"973","DOI":"10.1016\/j.jclinepi.2012.07.005","article-title":"Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs","volume":"66","author":"Hartling","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"R50","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-023-02389-w","article-title":"Inter-rater reliability of risk of bias tools for non-randomized studies","volume":"12","author":"Kalaycioglu","journal-title":"Syst Rev"},{"key":"R51","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2024.111370","article-title":"Common challenges and suggestions for risk of bias tool development: a systematic review of methodological studies","volume":"171","author":"Tomlinson","journal-title":"J Clin Epidemiol"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e87057","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:14:37Z","timestamp":1782839677000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e87057"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,8]]},"references-count":54,"URL":"https:\/\/doi.org\/10.2196\/87057","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,8]]},"article-number":"v28i1e87057"}}