{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T17:19:07Z","timestamp":1791479947017,"version":"4.3.4"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,3,6]],"date-time":"2024-03-06T00:00:00Z","timestamp":1709683200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,3,6]],"date-time":"2024-03-06T00:00:00Z","timestamp":1709683200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100006393","name":"Universidad de Granada","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100006393","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Adv Data Anal Classif"],"published-print":{"date-parts":[[2025,3]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>To measure the degree of agreement between <jats:italic>R<\/jats:italic> observers who independently classify <jats:italic>n<\/jats:italic> subjects within <jats:italic>K<\/jats:italic> categories, various <jats:italic>kappa<\/jats:italic>-type coefficients are often used. When <jats:italic>R<\/jats:italic>\u2009=\u20092, it is common to use the Cohen' <jats:italic>kappa<\/jats:italic>, Scott's <jats:italic>pi<\/jats:italic>, Gwet\u2019s <jats:italic>AC1\/2<\/jats:italic>, and Krippendorf's <jats:italic>alpha<\/jats:italic> coefficients (weighted or not). When <jats:italic>R<\/jats:italic>\u2009&gt;\u20092, some pairwise version based on the aforementioned coefficients is normally used; with the same order as above: Hubert's <jats:italic>kappa<\/jats:italic>, Fleiss's <jats:italic>kappa<\/jats:italic>, Gwet's <jats:italic>AC1\/2,<\/jats:italic> and Krippendorf's <jats:italic>alpha<\/jats:italic>. However, all these statistics are based on biased estimators of the expected index of agreements, since they estimate the product of two population proportions through the product of their sample estimators. The aims of this article are three. First, to provide statistics based on unbiased estimators of the expected index of agreements and determine their variance based on the variance of the original statistic. Second, to make pairwise extensions of some measures. And third, to show that the old and new estimators of the Cohen\u2019s <jats:italic>kappa<\/jats:italic> and Hubert\u2019s <jats:italic>kappa<\/jats:italic> coefficients match the well-known estimators of concordance and intraclass correlation coefficients, if the former are defined by assuming quadratic weights. The article shows that the new estimators are always greater than or equal the classic ones, except for the case of Gwet where it is the other way around, although these differences are only relevant with small sample sizes (e.g. <jats:italic>n<\/jats:italic>\u2009\u2264\u200930).<\/jats:p>","DOI":"10.1007\/s11634-024-00581-x","type":"journal-article","created":{"date-parts":[[2024,3,6]],"date-time":"2024-03-06T15:03:47Z","timestamp":1709737427000},"page":"177-207","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Estimators of various kappa coefficients based on the unbiased estimator of the expected index of agreements"],"prefix":"10.1007","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2548-2638","authenticated-orcid":false,"given":"A.","family":"Mart\u00edn Andr\u00e9s","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"M.","family":"\u00c1lvarez Hern\u00e1ndez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,3,6]]},"reference":[{"key":"581_CR1","doi-asserted-by":"publisher","first-page":"1020","DOI":"10.1111\/j.0006-341X.2002.01020.x","volume":"58","author":"HX Barnhart","year":"2002","unstructured":"Barnhart HX, Haber M, Song J (2002) Overall concordance correlation coefficient for evaluating agreement among multiple observers. Biometrics 58:1020\u20131027. https:\/\/doi.org\/10.1111\/j.0006-341X.2002.01020.x","journal-title":"Biometrics"},{"key":"581_CR2","doi-asserted-by":"crossref","first-page":"849","DOI":"10.1111\/j.0006-341X.2003.00099.x","volume":"59","author":"JL Carrasco","year":"2003","unstructured":"Carrasco JL, Jover LL (2003) Estimating the generalized concordance correlation coefficient through variance components. Biometrics 59:849\u2013858","journal-title":"Biometrics"},{"key":"581_CR3","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","volume":"20","author":"J Cohen","year":"1960","unstructured":"Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Measur 20:37\u201346","journal-title":"Educ Psychol Measur"},{"key":"581_CR4","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1037\/h0026256","volume":"70","author":"J Cohen","year":"1968","unstructured":"Cohen J (1968) Weighted kappa: nominal scale agreement with provision for scaled disagreemet or parcial credit. Psychol Bull 70:213\u2013220","journal-title":"Psychol Bull"},{"key":"581_CR5","doi-asserted-by":"publisher","first-page":"322","DOI":"10.1037\/0033-2909.88.2.322","volume":"88","author":"AJ Conger","year":"1980","unstructured":"Conger AJ (1980) Integration and generalization of kappas for multiple raters. Psychol Bull 88:322\u2013328. https:\/\/doi.org\/10.1037\/0033-2909.88.2.322","journal-title":"Psychol Bull"},{"key":"581_CR6","doi-asserted-by":"crossref","first-page":"1047","DOI":"10.2307\/2529886","volume":"38","author":"M Davies","year":"1982","unstructured":"Davies M, Fleiss JL (1982) Measuring agreement for multinomial data. Biometrics 38:1047\u20131051","journal-title":"Biometrics"},{"key":"581_CR7","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1037\/h0031619","volume":"76","author":"JL Fleiss","year":"1971","unstructured":"Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76:378\u2013382","journal-title":"Psychol Bull"},{"issue":"3","key":"581_CR8","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1177\/001316447303300309","volume":"33","author":"JL Fleiss","year":"1973","unstructured":"Fleiss JL, Cohen J (1973) The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Measur 33(3):613\u2013619","journal-title":"Educ Psychol Measur"},{"key":"581_CR9","doi-asserted-by":"publisher","first-page":"323","DOI":"10.1037\/h0028106","volume":"72","author":"JL Fleiss","year":"1969","unstructured":"Fleiss JL, Cohen J, Everitt BS (1969) Large sample standard errors of kappa and weighted kappa. Psychol Bull 72:323\u2013327. https:\/\/doi.org\/10.1037\/h0028106","journal-title":"Psychol Bull"},{"key":"581_CR10","doi-asserted-by":"crossref","DOI":"10.1002\/0471445428","volume-title":"Statistical methods for rates and proportions","author":"JL Fleiss","year":"2003","unstructured":"Fleiss JL, Levin B, Paik MC (2003) Statistical methods for rates and proportions, 3rd edn. Wiley, New York","edition":"3"},{"issue":"1","key":"581_CR11","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1348\/000711006X126600","volume":"61","author":"KL Gwet","year":"2008","unstructured":"Gwet KL (2008) Computing inter-rater reliability and its variance in the presence of high agreement. Br J Math Stat Psychol 61(1):29\u201368","journal-title":"Br J Math Stat Psychol"},{"issue":"4","key":"581_CR12","doi-asserted-by":"publisher","first-page":"781","DOI":"10.1177\/0013164420973080","volume":"81","author":"KL Gwet","year":"2021","unstructured":"Gwet KL (2021a) Large-sample variance of Fleiss generalized kappa. Educ Psychol Measur 81(4):781\u2013790. https:\/\/doi.org\/10.1177\/0013164420973080","journal-title":"Educ Psychol Measur"},{"key":"581_CR13","unstructured":"Gwet KL (2021b) Handbook of inter-rater reliability. Volume 1: analysis of categorical ratings, 5th edn. Gaithersburg, MD, USA"},{"issue":"2","key":"581_CR14","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1037\/0033-2909.84.2.289","volume":"48","author":"L Hubert","year":"1977","unstructured":"Hubert L (1977) Kappa revisited. Psychol Bull 48(2):289\u2013297","journal-title":"Psychol Bull"},{"issue":"2","key":"581_CR15","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1177\/00131640121971239","volume":"61","author":"S Janson","year":"2001","unstructured":"Janson S, Olsson U (2001) A measure of agreement for interval or nominal multivariate observations. Educ Psychol Measur 61(2):277\u2013289","journal-title":"Educ Psychol Measur"},{"issue":"14","key":"581_CR29","doi-asserted-by":"publisher","first-page":"2131","DOI":"10.1002\/sim.845","volume":"20","author":"TS King","year":"2001","unstructured":"King TS, Chinchilli VM (2001) A generalized concordance correlation coefficient for continuous and categorical data. Statist  Med 20(14):2131\u20132147. https:\/\/doi.org\/10.1002\/sim.845","journal-title":"Statist Med"},{"key":"581_CR16","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1177\/001316447003000105","volume":"30","author":"K Krippendorff","year":"1970","unstructured":"Krippendorff K (1970) Estimating the reliability, systematic error, and random error of interval data. Educ Psychol Measur 30:61\u201370","journal-title":"Educ Psychol Measur"},{"key":"581_CR17","doi-asserted-by":"crossref","first-page":"787","DOI":"10.1007\/s11135-004-8107-7","volume":"38","author":"K Krippendorff","year":"2004","unstructured":"Krippendorff K (2004) Measuring the reliability of qualitative text analysis data. Qual Quant 38:787\u2013800","journal-title":"Qual Quant"},{"key":"581_CR18","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1111\/j.1467-9574.1975.tb00254.x","volume":"29","author":"JR Landis","year":"1975","unstructured":"Landis JR, Koch GG (1975a) A review of statistical methods in the analysis of data arising from observer reliability studies (Part I). Stat Neerl 29:101\u2013123","journal-title":"Stat Neerl"},{"key":"581_CR19","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1111\/j.1467-9574.1975.tb00259.x","volume":"29","author":"JR Landis","year":"1975","unstructured":"Landis JR, Koch GG (1975b) A review of statistical methods in the analysis of data arising from observer reliability studies (Part II). Stat Neerl 29:151\u2013161","journal-title":"Stat Neerl"},{"key":"581_CR20","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159\u2013174","journal-title":"Biometrics"},{"key":"581_CR21","doi-asserted-by":"crossref","first-page":"255","DOI":"10.2307\/2532051","volume":"45","author":"LI-K Lin","year":"1989","unstructured":"Lin LI-K (1989) A concordance correlation coefficient to evaluate reproducibility. Biometrics 45:255\u2013268","journal-title":"Biometrics"},{"key":"581_CR22","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1111\/j.0006-341X.2000.00324.x","volume":"56","author":"LI-K Lin","year":"2000","unstructured":"Lin LI-K (2000) A note of the concordance correlation coefficient. Letter to the Editor (Corrections). Biometrics 56:324\u2013325","journal-title":"Biometrics"},{"issue":"1","key":"581_CR23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/bmsp.12167","volume":"73","author":"A Mart\u00edn Andr\u00e9s","year":"2020","unstructured":"Mart\u00edn Andr\u00e9s A, \u00c1lvarez Hern\u00e1ndez M (2020) Hubert\u2019s multi-rater kappa revisited. Br J Math Stat Psychol 73(1):1\u201322","journal-title":"Br J Math Stat Psychol"},{"key":"581_CR24","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1002\/sim.4780040211","volume":"4","author":"O Miettinen","year":"1985","unstructured":"Miettinen O, Nurminen M (1985) Comparative analysis of two rates. Stat Med 4:213\u2013226. https:\/\/doi.org\/10.1002\/sim.4780040211","journal-title":"Stat Med"},{"issue":"1","key":"581_CR25","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1007\/s11336-003-1110-4","volume":"70","author":"C Schuster","year":"2005","unstructured":"Schuster C, Smith DA (2005) Dispersion-weighted kappa: an integrative framework for metric and nominal scale agreement coefficients. Psychometrika 70(1):135\u2013146","journal-title":"Psychometrika"},{"key":"581_CR26","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1086\/266577","volume":"19","author":"WA Scott","year":"1955","unstructured":"Scott WA (1955) Reliability of content analysis: the case of nominal scale coding. Public Opin Q 19:321\u2013325","journal-title":"Public Opin Q"},{"key":"581_CR27","doi-asserted-by":"crossref","first-page":"420","DOI":"10.1037\/0033-2909.86.2.420","volume":"86","author":"PE Shrout","year":"1979","unstructured":"Shrout PE, Fleiss JL (1979) Intraclass correlations: Uses in assessing rater reliability. Psychol Bull 86:420\u2013428","journal-title":"Psychol Bull"},{"key":"581_CR28","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1007\/s11634-010-0073-4","volume":"4","author":"MJ Warrens","year":"2010","unstructured":"Warrens MJ (2010) Inequalities between multi-rater kappas. Adv Data Anal Classif 4:271\u2013286","journal-title":"Adv Data Anal Classif"}],"container-title":["Advances in Data Analysis and Classification"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11634-024-00581-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11634-024-00581-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11634-024-00581-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,30]],"date-time":"2025-03-30T10:05:24Z","timestamp":1743329124000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11634-024-00581-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,6]]},"references-count":29,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,3]]}},"alternative-id":["581"],"URL":"https:\/\/doi.org\/10.1007\/s11634-024-00581-x","relation":{},"ISSN":["1862-5347","1862-5355"],"issn-type":[{"value":"1862-5347","type":"print"},{"value":"1862-5355","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,6]]},"assertion":[{"value":"14 May 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 January 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 March 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}