{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T15:41:16Z","timestamp":1786117276271,"version":"build-2736575974"},"reference-count":24,"publisher":"BMJ","issue":"8","license":[{"start":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T00:00:00Z","timestamp":1785974400000},"content-version":"unspecified","delay-in-days":5,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Open"],"accepted":{"date-parts":[[2026,7,11]]},"published-print":{"date-parts":[[2026,8]]},"abstract":"<jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To compare the reliability of four commonly used structured tools for risk-of-bias assessment in prognostic cohort studies and evaluate their agreement with expert appraisal.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Design<\/jats:title>\n                    <jats:p>Methodological comparison study.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Setting<\/jats:title>\n                    <jats:p>Secondary analysis of published prognostic cohort studies in patients with acute pulmonary embolism from a previously conducted systematic review.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Participants<\/jats:title>\n                    <jats:p>Sixty-three cohort studies assessing the prognostic role of echocardiography in patients with acute pulmonary embolism.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Primary and secondary outcome measures<\/jats:title>\n                    <jats:p>Four independent reviewers assessed study quality\/risk of bias using the Newcastle-Ottawa Scale (NOS), Risk Of Bias in Non-Randomised Studies of Interventions (ROBINS-I), Quality In Prognostic Studies (QUIPS) and Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2). Concomitantly and independently, expert reviewers\u2019 implicit global judgements were used as an external comparator. Inter-rater reliability, agreement with expert appraisal, internal consistency and floor\/ceiling effects were assessed.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Inter-rater agreement was fair for NOS (Gwet\u2019s agreement coefficient 1, 0.25, 95%\u2009CI 0.15 to 0.34), ROBINS-I (0.38, 95%\u2009CI 0.28 to 0.47) and QUADAS-2 (0.35, 95%\u2009CI 0.27 to 0.43) and almost perfect for QUIPS (0.85, 95%\u2009CI 0.73 to 0.93). Agreement with expert appraisal was limited for all tools except ROBINS-I, which showed fair concordance. QUIPS frequently classified studies as low risk of bias, suggesting potential overestimation of study quality. Internal consistency was generally low across tools, while ceiling effects were observed for NOS and QUIPS. ROBINS-I showed the most balanced distribution of ratings.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Structured tools for risk-of-bias assessment in prognostic cohort studies have variable reliability and limited agreement with expert appraisal. ROBINS-I showed the strongest concordance with expert judgement, whereas QUIPS may provide optimistic ratings. These findings support further refinement and standardisation of risk-of-bias assessment methods for observational prognostic research.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bmjopen-2025-113000","type":"journal-article","created":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T13:40:20Z","timestamp":1786023620000},"page":"e113000","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":0,"title":["Risk-of-bias assessment in prognostic cohort studies: a methodological comparison study of structured appraisal tools in acute pulmonary embolism research"],"prefix":"10.1136","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5058-6140","authenticated-orcid":false,"given":"Ludovica Anna","family":"Cimini","sequence":"first","affiliation":[{"name":"Internal and Cardiovascular Medicine, Stroke Unit, University of Perugia, Perugia, Umbria, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Frederikus A","family":"Klok","sequence":"additional","affiliation":[{"name":"Department of Medicine, Thrombosis and Hemostasis, Leiden University Medical Center, Leiden, the Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marc","family":"Carrier","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kerstin","family":"de Wit","sequence":"additional","affiliation":[{"name":"Division of Emergency Medicine, Department of Medicine, McMaster University, Hamilton, Ontario, Canada"},{"name":"Department of Emergency Medicine, Queen\u2019s University, Kingston, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2522-2705","authenticated-orcid":false,"given":"Scott C","family":"Woller","sequence":"additional","affiliation":[{"name":"Department of Medicine, Intermountain Health, Salt Lake City, Utah, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea Galeazzo","family":"Rigutini","sequence":"additional","affiliation":[{"name":"Internal and Cardiovascular Medicine, Stroke Unit, University of Perugia, Perugia, Umbria, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roupen","family":"Odabashian","sequence":"additional","affiliation":[{"name":"Department of Hematology\/Oncology, Barbara Ann Karmanos Cancer Institute, Detroit, Michigan, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rosa","family":"Talerico","sequence":"additional","affiliation":[{"name":"Department of Geriatric, Orthopedic, and Rheumatologic Sciences, Fondazione Policlinico Universitario Agostino Gemelli IRCCS, Rome, Lazio, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maria Giovanna","family":"Ranalli","sequence":"additional","affiliation":[{"name":"Department of Political Science, University of Perugia, Perugia, Umbria, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cecilia","family":"Becattini","sequence":"additional","affiliation":[{"name":"Internal and Cardiovascular Medicine, Stroke Unit, University of Perugia, Perugia, Umbria, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2026,8,6]]},"reference":[{"key":"2026080606400939000_16.8.e113000.1","doi-asserted-by":"crossref","DOI":"10.1136\/bmj.k4597","article-title":"A guide to systematic review and meta-analysis of prognostic factor studies","volume":"364","author":"Riley","year":"2019","journal-title":"BMJ"},{"key":"2026080606400939000_16.8.e113000.2","doi-asserted-by":"crossref","first-page":"1895","DOI":"10.1093\/eurheartj\/ehaf016","article-title":"2024 Revision of the level of evidence grading system for ESC clinical practice guideline recommendations II: diagnostic tests and prediction models","volume":"46","author":"Di Angelantonio","year":"2025","journal-title":"Eur Heart J"},{"key":"2026080606400939000_16.8.e113000.3","doi-asserted-by":"crossref","first-page":"472","DOI":"10.11124\/JBIES-22-00434","article-title":"From critical appraisal to risk of bias assessment: clarifying the terminology for study evaluation in JBI systematic reviews","volume":"21","author":"Stone","year":"2023","journal-title":"JBI Evid Synth"},{"key":"2026080606400939000_16.8.e113000.4","doi-asserted-by":"publisher","DOI":"10.1093\/ije\/dym018"},{"key":"2026080606400939000_16.8.e113000.5","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-158-4-201302190-00009"},{"key":"2026080606400939000_16.8.e113000.6","unstructured":"Wells GA , Shea B , O\u2019Connell D , et al . The newcastle\u2013Ottawa scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa, ON: Ottawa Hospital Research Institute; 2011. Available: https:\/\/www.ohri.ca\/programs\/clinical_epidemiology\/oxford.asp"},{"key":"2026080606400939000_16.8.e113000.7","doi-asserted-by":"crossref","DOI":"10.1136\/bmj.i4919","article-title":"ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions","volume":"355","author":"Sterne","year":"2016","journal-title":"BMJ"},{"key":"2026080606400939000_16.8.e113000.8","doi-asserted-by":"publisher","DOI":"10.1059\/0003-4819-155-8-201110180-00009"},{"key":"2026080606400939000_16.8.e113000.9","doi-asserted-by":"publisher","DOI":"10.1007\/s10654-010-9491-z"},{"key":"2026080606400939000_16.8.e113000.10","unstructured":"Higgins JPT , Thomas J , Chandler J , eds. Cochrane Handbook for Systematic Reviews of Interventions Version. Cochrane, 2022. Available: https:\/\/training.cochrane.org\/handbook"},{"key":"2026080606400939000_16.8.e113000.11","doi-asserted-by":"crossref","first-page":"4186","DOI":"10.1093\/eurheartj\/ehy631","article-title":"Differential impact of syncope on the prognosis of patients with acute pulmonary embolism: a systematic review and meta-analysis","volume":"39","author":"Barco","year":"2018","journal-title":"Eur Heart J"},{"key":"2026080606400939000_16.8.e113000.12","doi-asserted-by":"crossref","first-page":"2504","DOI":"10.1111\/jth.15453","article-title":"Detection of right ventricular dysfunction in acute pulmonary embolism by computed tomography or echocardiography: A systematic review and meta-analysis","volume":"19","author":"Chornenki","year":"2021","journal-title":"J Thromb Haemost"},{"key":"2026080606400939000_16.8.e113000.13","doi-asserted-by":"publisher","DOI":"10.1093\/eurheartj\/ehad417"},{"key":"2026080606400939000_16.8.e113000.14","doi-asserted-by":"publisher","DOI":"10.1111\/jth.14477"},{"key":"2026080606400939000_16.8.e113000.15","doi-asserted-by":"crossref","first-page":"1158","DOI":"10.1111\/jth.15661","article-title":"Risk of recurrence in women with venous thromboembolism related to estrogen-containing contraceptives: Systematic review and meta-analysis","volume":"20","author":"Wiegers","year":"2022","journal-title":"J Thromb Haemost"},{"key":"2026080606400939000_16.8.e113000.16","doi-asserted-by":"crossref","DOI":"10.1183\/23120541.00641-2022","article-title":"Prognostic role of different findings at echocardiography in acute pulmonary embolism: a critical review and meta-analysis","volume":"9","author":"Cimini","year":"2023","journal-title":"ERJ Open Res"},{"key":"2026080606400939000_16.8.e113000.17","doi-asserted-by":"crossref","DOI":"10.1186\/1471-2288-13-61","article-title":"A comparison of Cohen\u2019s Kappa and Gwet\u2019s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples","volume":"13","author":"Wongpakaran","year":"2013","journal-title":"BMC Med Res Methodol"},{"key":"2026080606400939000_16.8.e113000.18","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"2026080606400939000_16.8.e113000.19","doi-asserted-by":"publisher","DOI":"10.5116\/ijme.4dfb.8dfd"},{"key":"2026080606400939000_16.8.e113000.20","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2006.03.012"},{"key":"2026080606400939000_16.8.e113000.21","doi-asserted-by":"crossref","DOI":"10.1186\/s12874-018-0495-9","article-title":"Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations","volume":"18","author":"Mueller","year":"2018","journal-title":"BMC Med Res Methodol"},{"key":"2026080606400939000_16.8.e113000.22","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2019.04.001"},{"key":"2026080606400939000_16.8.e113000.23","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2013.03.003"},{"key":"2026080606400939000_16.8.e113000.24","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1016\/j.bjpt.2019.01.009","article-title":"Measurement properties of quality assessment tools for studies of diagnostic accuracy","volume":"24","author":"Kaizik","year":"2020","journal-title":"Braz J Phys Ther"}],"container-title":["BMJ Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjopen-2025-113000","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T13:40:24Z","timestamp":1786023624000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmjopen.bmj.com\/lookup\/doi\/10.1136\/bmjopen-2025-113000"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8]]},"references-count":24,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2026,8,6]]},"published-print":{"date-parts":[[2026,8]]}},"alternative-id":["10.1136\/bmjopen-2025-113000"],"URL":"https:\/\/doi.org\/10.1136\/bmjopen-2025-113000","relation":{},"ISSN":["2044-6055","2044-6055"],"issn-type":[{"value":"2044-6055","type":"print"},{"value":"2044-6055","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8]]}}}