{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,26]],"date-time":"2026-08-26T18:52:28Z","timestamp":1787770348960,"version":"build-2784847793"},"reference-count":62,"publisher":"Wiley","issue":"23-24","license":[{"start":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T00:00:00Z","timestamp":1760572800000},"content-version":"vor","delay-in-days":15,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Statistics in Medicine"],"published-print":{"date-parts":[[2025,10]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:p>Healthcare professionals routinely perform clinical examinations and diagnostic assessments. How the findings of these assessments are interpreted can have significant implications for patient care and outcomes. A recent systematic review on reliability and agreement studies in intrapartum fetal heart rate monitoring highlighted three methodological issues: (1) confusion between the concepts of agreement and reliability, (2) lack of clarity on how agreement and reliability measures are calculated when more than two raters are involved, and (3) confidence intervals seldom reported. This paper aims to clarify how agreement measures can be computed and interpreted when the outcome is binary (e.g., normal\/abnormal test result). Using a motivating example in which five experienced obstetricians assessed 20 CTGs, we demonstrate how agreement can be defined, computed, and interpreted in various scenarios. The paper further explains the relationship between agreement measures and the concept of reliability, the distinction between intra\u2010 and inter\u2010observer studies, and approaches to make statistical inference and sample size calculations. Particular emphasis is placed on the proportion of agreement, the proportion of specific agreement and kappa coefficients. A shiny application has also been developed to support researchers in their agreement studies. This work completes existing tools such as the Guidelines for Reporting Reliability and Agreement Studies (GRRAS), the Quality Appraisal Tool for Studies of Diagnostic Reliability (QAREL) and STARD guidelines for reporting diagnostic accuracy studies. It is intended to help researchers improve the methodological quality of studies that evaluate the agreement of clinical tests.<\/jats:p>","DOI":"10.1002\/sim.70299","type":"journal-article","created":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T05:20:41Z","timestamp":1760678441000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Measuring Agreement in Diagnostics: A Practical Guide for Researchers"],"prefix":"10.1002","volume":"44","author":[{"given":"Sophie","family":"Vanbelle","sequence":"first","affiliation":[{"name":"Methodology and Statistics, CAPHRI Maastricht University  Maastricht Limburg the Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5494-8783","authenticated-orcid":false,"given":"Christina Hernandez","family":"Engelhart","sequence":"additional","affiliation":[{"name":"Norwegian Research Center for Women's Health Oslo University Hospital  Oslo Norway"},{"name":"Faculty of Health Sciences Oslo Metropolitan University  Oslo Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ellen","family":"Blix","sequence":"additional","affiliation":[{"name":"Faculty of Health Sciences Oslo Metropolitan University  Oslo Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2025,10,16]]},"reference":[{"key":"e_1_2_14_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2010.12.001"},{"key":"e_1_2_14_3_1","doi-asserted-by":"publisher","DOI":"10.1111\/aogs.14591"},{"key":"e_1_2_14_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2010.03.002"},{"key":"e_1_2_14_5_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen\u20102016\u2010012799"},{"key":"e_1_2_14_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2009.10.002"},{"key":"e_1_2_14_7_1","volume-title":"Statistical Methods for Rates and Proportions","author":"Fleiss J. L.","year":"1981"},{"key":"e_1_2_14_8_1","doi-asserted-by":"publisher","DOI":"10.2307\/1932409"},{"key":"e_1_2_14_9_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469\u20108137.1912.tb05611.x"},{"key":"e_1_2_14_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0140\u20106736(75)90304\u20109"},{"key":"e_1_2_14_11_1","doi-asserted-by":"publisher","DOI":"10.1086\/266577"},{"key":"e_1_2_14_12_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_2_14_13_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0026256"},{"key":"e_1_2_14_14_1","doi-asserted-by":"publisher","DOI":"10.2307\/2529886"},{"key":"e_1_2_14_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02296208"},{"key":"e_1_2_14_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/0140\u20106736(91)92169\u20103"},{"key":"e_1_2_14_17_1","doi-asserted-by":"publisher","DOI":"10.2466\/pr0.1966.19.1.3"},{"key":"e_1_2_14_18_1","doi-asserted-by":"publisher","DOI":"10.1080\/10543400701376480"},{"key":"e_1_2_14_19_1","doi-asserted-by":"publisher","DOI":"10.1177\/0962280218794733"},{"key":"e_1_2_14_20_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316446402400402"},{"key":"e_1_2_14_21_1","doi-asserted-by":"publisher","DOI":"10.1192\/bjp.130.1.79"},{"key":"e_1_2_14_22_1","first-page":"1409","article-title":"A Statistical Method for Evaluating Systematic Relationships","volume":"38","author":"Sokal R.","year":"1958","journal-title":"University of Kansas Science Bulletin"},{"key":"e_1_2_14_23_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1971.10482356"},{"key":"e_1_2_14_24_1","first-page":"1","article-title":"A Method of Establishing Groups of Equal Amplitude in Plant Sociology Based on Similarity of Species Content, and Its Application to Analysis of Vegetation on Danish Commons","volume":"5","author":"Sorenson T.","year":"1948","journal-title":"Kongelige Danske Videnskabernes Selskabs. Biologiske Skrifter"},{"key":"e_1_2_14_25_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1751\u20105823.2012.00183.x"},{"key":"e_1_2_14_26_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.76.10.5269"},{"key":"e_1_2_14_27_1","article-title":"Measures of Association for Cross Classifications I, II, III, IV","author":"Goodman L. A.","year":"1954","journal-title":"Journal of the American Statistical Association"},{"key":"e_1_2_14_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/0021-9681(66)90032-4"},{"key":"e_1_2_14_29_1","volume-title":"An Elementary Mathematical Theory of Classification and Prediction","author":"Tanimoto T.","year":"1958"},{"key":"e_1_2_14_30_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316448104100307"},{"key":"e_1_2_14_31_1","volume-title":"Handbook of Inter\u2010Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters","author":"Gwet K. L.","year":"2014"},{"key":"e_1_2_14_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/0895\u20104356(93)90018\u2010V"},{"key":"e_1_2_14_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(90)90158-L"},{"key":"e_1_2_14_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(90)90159-M"},{"key":"e_1_2_14_35_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316447003000105"},{"key":"e_1_2_14_36_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.88.2.322"},{"key":"e_1_2_14_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2016.12.007"},{"key":"e_1_2_14_38_1","unstructured":"J. J.Randolph \u201cFree\u2010Marginal Multirater Kappa (multirater K[free]): An Alternative to Fleiss' Fixed\u2010Marginal Multirater Kappa \u201d2005."},{"key":"e_1_2_14_39_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.84.2.289"},{"key":"e_1_2_14_40_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0031619"},{"key":"e_1_2_14_41_1","doi-asserted-by":"publisher","DOI":"10.2307\/1412159"},{"key":"e_1_2_14_42_1","volume-title":"Statistical Theories of Mental Test scoresOxford","author":"Lord F.","year":"1968"},{"key":"e_1_2_14_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4757-3456-0"},{"key":"e_1_2_14_44_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.0006\u2010341X.2005.031040.x"},{"key":"e_1_2_14_45_1","doi-asserted-by":"publisher","DOI":"10.1037\/met0000738"},{"key":"e_1_2_14_46_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-985X.2004.00365.x"},{"key":"e_1_2_14_47_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.0006-341X.1999.00137.x"},{"key":"e_1_2_14_48_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.0006\u2010341X.2004.00232.x"},{"key":"e_1_2_14_49_1","unstructured":"S.Vanbelle \u201cFrom Tetrachoric to Kappa: How to Assess Reliability on Binary Scales \u201d Submitted."},{"key":"e_1_2_14_50_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469-185X.2010.00141.x"},{"key":"e_1_2_14_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cct.2012.05.004"},{"key":"e_1_2_14_52_1","doi-asserted-by":"publisher","DOI":"10.1080\/02664763.2011.578623"},{"key":"e_1_2_14_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11336-003-1110-4"},{"key":"e_1_2_14_54_1","doi-asserted-by":"publisher","DOI":"10.1111\/bmsp.12333"},{"key":"e_1_2_14_55_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467\u20109884.00331"},{"key":"e_1_2_14_56_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-0258(20000315)19:5<723::AID-SIM379>3.0.CO;2-A"},{"key":"e_1_2_14_57_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-0258(19980430)17:8<857::AID-SIM777>3.0.CO;2-E"},{"key":"e_1_2_14_58_1","doi-asserted-by":"publisher","DOI":"10.2307\/2529786"},{"key":"e_1_2_14_59_1","doi-asserted-by":"publisher","DOI":"10.1348\/000711006X126600"},{"key":"e_1_2_14_60_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.mex.2023.102212"},{"key":"e_1_2_14_61_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-024-02431-y"},{"key":"e_1_2_14_62_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0895\u20104356(98)00048\u20101"},{"key":"e_1_2_14_63_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9574.1982.tb00774.x"}],"container-title":["Statistics in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/sim.70299","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,29]],"date-time":"2025-10-29T06:09:10Z","timestamp":1761718150000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/sim.70299"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10]]},"references-count":62,"journal-issue":{"issue":"23-24","published-print":{"date-parts":[[2025,10]]}},"alternative-id":["10.1002\/sim.70299"],"URL":"https:\/\/doi.org\/10.1002\/sim.70299","archive":["Portico"],"relation":{},"ISSN":["0277-6715","1097-0258"],"issn-type":[{"value":"0277-6715","type":"print"},{"value":"1097-0258","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10]]},"assertion":[{"value":"2025-04-09","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-03","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70299"}}