{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T14:54:03Z","timestamp":1786978443811,"version":"build-2736575974"},"reference-count":36,"publisher":"Wiley","issue":"15-17","license":[{"start":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T00:00:00Z","timestamp":1752537600000},"content-version":"vor","delay-in-days":14,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01\u2010CA172463"],"award-info":[{"award-number":["R01\u2010CA172463"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01\u2010CA226805"],"award-info":[{"award-number":["R01\u2010CA226805"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["U01CA163304"],"award-info":[{"award-number":["U01CA163304"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["U54CA163303"],"award-info":[{"award-number":["U54CA163303"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["U54CA163307"],"award-info":[{"award-number":["U54CA163307"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["U54CA163313"],"award-info":[{"award-number":["U54CA163313"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Statistics in Medicine"],"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:p>Cohen's kappa and other summary measures are often used in clinical studies to describe agreement and association between two experts' ordered categorical ratings. However, a key limitation of Cohen's kappa and similar measures is their inability to evaluate the impact of patient\u2010related factors such as family history and age on the agreement and association between experts. Strong agreement between experts is an essential component of effective clinical procedures where subjective interpretation of patients' images or test results by an expert is required, for example, in the visual assessment of breast density from a mammogram. Not accounting for important patient\u2010related factors can lead to inflated and biased assessments of agreement and association. In this article, our objective is to propose novel model\u2010based measures that appropriately account for the impact of patient\u2010related covariates on chance\u2010corrected agreement and association between two experts' ordinal ratings that overcome limitations of existing measures. Our population\u2010based approach is based on an ordinal generalized linear mixed model (GLMM). Rigorous simulation studies evaluating performance of the new model\u2010based measures in a broad range of settings are reported. Existing and new measures are compared in two clinical applications assessing breast density and multiple sclerosis. Key advantages of the new kappa measures over existing measures such as Cohen's kappa include incorporating patient\u2010related factors, robustness to underlying disease prevalence and marginal distributions of experts' ratings, and appropriately correcting for chance agreement. Sample R code is provided by the authors for application of proposed measures in other studies.<\/jats:p>","DOI":"10.1002\/sim.70141","type":"journal-article","created":{"date-parts":[[2025,7,15]],"date-time":"2025-07-15T10:39:53Z","timestamp":1752575993000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Accounting for Patient Characteristics in a Model\u2010Based Kappa of Agreement Between Two Experts' Ordinal Ratings"],"prefix":"10.1002","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1195-8105","authenticated-orcid":false,"given":"Kerrie P.","family":"Nelson","sequence":"first","affiliation":[{"name":"Department of Biostatistics Boston University  Boston Massachusetts USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas J.","family":"Zhou","sequence":"additional","affiliation":[{"name":"Center for Biologics Evaluation and Research, Food and Drug Administration  Silver Springs Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2025,7,15]]},"reference":[{"key":"e_1_2_11_2_1","doi-asserted-by":"publisher","DOI":"10.1177\/2058460120937381"},{"key":"e_1_2_11_3_1","volume-title":"ACRBI\u2010RADS_Atlas, Breast Imaging Reporting and Data System","author":"D'Orsi C. J.","year":"2013"},{"key":"e_1_2_11_4_1","unstructured":"DenseBreastInfo.Org Website. accessed May 26 2025 https:\/\/densebreast\u2010info.org\/legislative\u2010information\/national\u2010reporting\u2010standard\/."},{"key":"e_1_2_11_5_1","doi-asserted-by":"publisher","DOI":"10.7326\/M15\u20102934"},{"key":"e_1_2_11_6_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-9876.00175"},{"key":"e_1_2_11_7_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_2_11_8_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0026256"},{"key":"e_1_2_11_9_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.4780100913"},{"key":"e_1_2_11_10_1","doi-asserted-by":"publisher","DOI":"10.2307\/2532907"},{"key":"e_1_2_11_11_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033\u20102909.86.2.420"},{"key":"e_1_2_11_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/0895\u2010\u20104356(90)90158\u2010l"},{"key":"e_1_2_11_13_1","doi-asserted-by":"publisher","DOI":"10.1177\/001316448104100307"},{"key":"e_1_2_11_14_1","doi-asserted-by":"publisher","DOI":"10.2307\/3315487"},{"key":"e_1_2_11_15_1","doi-asserted-by":"publisher","DOI":"10.1093\/aje\/126.2.161"},{"key":"e_1_2_11_16_1","doi-asserted-by":"publisher","DOI":"10.1002\/cjs.5550360306"},{"key":"e_1_2_11_17_1","doi-asserted-by":"publisher","DOI":"10.1086\/266577"},{"key":"e_1_2_11_18_1","doi-asserted-by":"publisher","DOI":"10.2307\/2532052"},{"key":"e_1_2_11_19_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0031643"},{"key":"e_1_2_11_20_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033\u20102909.88.2.322"},{"key":"e_1_2_11_21_1","unstructured":"National Cancer Institute Website.\u201cDense Breasts: Answers to Commonly Asked Questions \u201daccessed May 26 2025 https:\/\/www.cancer.gov\/types\/breast\/breast\u2010changes\/dense\u2010breastsand DenseBreastInfo.Org website accessed May 26 2025 https:\/\/densebreast\u2010info.org\/5\u2010facts\u2010you\u2010should\u2010know\/."},{"key":"e_1_2_11_22_1","doi-asserted-by":"publisher","DOI":"10.2307\/2531866"},{"key":"e_1_2_11_23_1","doi-asserted-by":"publisher","DOI":"10.2307\/2288068"},{"key":"e_1_2_11_24_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.98.2.408"},{"key":"e_1_2_11_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/0895-4356(93)90173-X"},{"key":"e_1_2_11_26_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.7323"},{"key":"e_1_2_11_27_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.6546"},{"key":"e_1_2_11_28_1","doi-asserted-by":"publisher","DOI":"10.2307\/2533433"},{"key":"e_1_2_11_29_1","volume-title":"R: A Language and Environment for Statistical Computing","author":"R Core Team","year":"2019"},{"key":"e_1_2_11_30_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1997.10473598"},{"key":"e_1_2_11_31_1","doi-asserted-by":"publisher","DOI":"10.1177\/1471082X0100100302"},{"key":"e_1_2_11_32_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1993.10594284"},{"key":"e_1_2_11_33_1","doi-asserted-by":"publisher","DOI":"10.1080\/10618600.1995.10474663"},{"key":"e_1_2_11_34_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.0006-341X.1999.00085.x"},{"key":"e_1_2_11_35_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.2599"},{"key":"e_1_2_11_36_1","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"issue":"3","key":"e_1_2_11_37_1","first-page":"380","article-title":"Studies on Multiple Sclerosis in Winnipeg, Manitoba, and New Orleans, Louisiana. I. Prevalence; Comparison Between the Patient Groups in Winnipeg and New Orleans","volume":"57","author":"Westlund K. B.","year":"1953","journal-title":"American Journal of Hygiene"}],"container-title":["Statistics in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/sim.70141","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,14]],"date-time":"2025-11-14T17:13:45Z","timestamp":1763140425000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/sim.70141"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7]]},"references-count":36,"journal-issue":{"issue":"15-17","published-print":{"date-parts":[[2025,7]]}},"alternative-id":["10.1002\/sim.70141"],"URL":"https:\/\/doi.org\/10.1002\/sim.70141","archive":["Portico"],"relation":{},"ISSN":["0277-6715","1097-0258"],"issn-type":[{"value":"0277-6715","type":"print"},{"value":"1097-0258","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7]]},"assertion":[{"value":"2024-04-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-16","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-15","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70141"}}