{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T18:02:41Z","timestamp":1781114561423,"version":"3.54.1"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Adv Data Anal Classif"],"published-print":{"date-parts":[[2010,12]]},"DOI":"10.1007\/s11634-010-0073-4","type":"journal-article","created":{"date-parts":[[2010,10,13]],"date-time":"2010-10-13T08:36:19Z","timestamp":1286958979000},"page":"271-286","source":"Crossref","is-referenced-by-count":171,"title":["Inequalities between multi-rater kappas"],"prefix":"10.1007","volume":"4","author":[{"given":"Matthijs J.","family":"Warrens","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2010,10,10]]},"reference":[{"key":"73_CR1","unstructured":"Artstein R, Poesio M (2005) Kappa3\u00a0=\u00a0Alpha (or Beta). NLE Technical Note 05-1, University of Essex"},{"key":"73_CR2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.2307\/3315487","volume":"27","author":"M Banerjee","year":"1999","unstructured":"Banerjee M, Capozzoli M, McSweeney L, Sinha D (1999) Beyond kappa: a review of interrater agreement measures. Can J Stat 27: 3\u201323","journal-title":"Can J Stat"},{"key":"73_CR3","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1086\/266520","volume":"18","author":"EM Bennett","year":"1954","unstructured":"Bennett EM, Alpert R, Goldstein AC (1954) Communications through limited response questioning. Public Opin Q 18: 303\u2013308","journal-title":"Public Opin Q"},{"key":"73_CR4","doi-asserted-by":"crossref","first-page":"921","DOI":"10.1177\/0013164488484007","volume":"48","author":"KJ Berry","year":"1988","unstructured":"Berry KJ, Mielke PW (1988) A generalization of Cohen\u2019s kappa agreement measure to interval measurement and multiple raters. Educ Psychol Meas 48: 921\u2013933","journal-title":"Educ Psychol Meas"},{"key":"73_CR5","doi-asserted-by":"crossref","first-page":"687","DOI":"10.1177\/001316448104100307","volume":"41","author":"RL Brennan","year":"1981","unstructured":"Brennan RL, Prediger DJ (1981) Coefficient kappa: some uses, misuses, and alternatives. Edu Psychol Meas 41: 687\u2013699","journal-title":"Edu Psychol Meas"},{"key":"73_CR6","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","volume":"20","author":"J Cohen","year":"1960","unstructured":"Cohen J (1960) A coefficient of agreement for nominal scales. Edu Psychol Meas 20: 37\u201346","journal-title":"Edu Psychol Meas"},{"key":"73_CR7","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1037\/h0026256","volume":"70","author":"J Cohen","year":"1968","unstructured":"Cohen J (1968) Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 70: 213\u2013220","journal-title":"Psychol Bull"},{"key":"73_CR8","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1037\/0033-2909.88.2.322","volume":"88","author":"AJ Conger","year":"1980","unstructured":"Conger AJ (1980) Integration and generalization of kappas for multiple raters. Psychol Bull 88: 322\u2013328","journal-title":"Psychol Bull"},{"key":"73_CR9","doi-asserted-by":"crossref","first-page":"260","DOI":"10.1086\/268657","volume":"45","author":"RT Craig","year":"1981","unstructured":"Craig RT (1981) Generalization of Scott\u2019s index of intercoder agreement. Public Opin Q 45: 260\u2013264","journal-title":"Public Opin Q"},{"key":"73_CR10","doi-asserted-by":"crossref","first-page":"1047","DOI":"10.2307\/2529886","volume":"38","author":"M Davies","year":"1982","unstructured":"Davies M, Fleiss JL (1982) Measuring agreement for multinomial data. Biometrics 38: 1047\u20131051","journal-title":"Biometrics"},{"key":"73_CR11","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1198\/000313007X192392","volume":"61","author":"J De Mast","year":"2007","unstructured":"De Mast J (2007) Agreement and kappa-type indices. Am Stat 61: 148\u2013153","journal-title":"Am Stat"},{"key":"73_CR12","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1162\/089120104773633402","volume":"30","author":"B Di Eugenio","year":"2004","unstructured":"Di Eugenio B, Glass M (2004) The kappa statistic: a second look. Comput Linguist 30: 95\u2013101","journal-title":"Comput Linguist"},{"key":"73_CR13","doi-asserted-by":"crossref","first-page":"726","DOI":"10.1016\/j.neucom.2006.10.007","volume":"70","author":"W Dou","year":"2007","unstructured":"Dou W, Ren Y, Wu Q, Ruan S, Chen Y, Bloyet D, Constans J-M (2007) Fuzzy kappa for the agreement measure of fuzzy classifications. Neurocomputing 70: 726\u2013734","journal-title":"Neurocomputing"},{"key":"73_CR14","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1037\/h0031619","volume":"76","author":"JL Fleiss","year":"1971","unstructured":"Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76: 378\u2013382","journal-title":"Psychol Bull"},{"key":"73_CR15","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1007\/s11336-007-9054-8","volume":"73","author":"KL Gwet","year":"2008","unstructured":"Gwet KL (2008) Variance estimation of nominal-scale inter-rater reliability with random selection of raters. Psychometrika 73: 407\u2013430","journal-title":"Psychometrika"},{"key":"73_CR16","unstructured":"Heuvelmans APJM, Sanders PF (1993) Beoordelaarsovereenstemming. In: Eggen TJHM, Sanders PF (eds) Psychometrie in de Praktijk. Cito Instituut voor Toestontwikkeling, Arnhem, pp 443\u2013470"},{"key":"73_CR17","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1207\/S15328031US0203_03","volume":"2","author":"LM Hsu","year":"2003","unstructured":"Hsu LM, Field R (2003) Interrater agreement measures: comments on kappa n, Cohen\u2019s kappa, Scott\u2019s \u03c0 and Aickin\u2019s \u03b1. Underst Stat 2: 205\u2013219","journal-title":"Underst Stat"},{"key":"73_CR18","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1037\/0033-2909.84.2.289","volume":"84","author":"L Hubert","year":"1977","unstructured":"Hubert L (1977) Kappa revisited. Psychol Bull 84: 289\u2013297","journal-title":"Psychol Bull"},{"key":"73_CR19","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1192\/bjp.134.6.617","volume":"134","author":"CL Janes","year":"1979","unstructured":"Janes CL (1979) An extension of the random error coefficient of agreement to N\u00a0\u00d7 N tables. Br J Psychiatry 134: 617\u2013619","journal-title":"Br J Psychiatry"},{"key":"73_CR20","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1177\/00131640121971239","volume":"61","author":"H Janson","year":"2001","unstructured":"Janson H, Olsson U (2001) A measure of agreement for interval or nominal multivariate observations. Educ Psychol Meas 61: 277\u2013289","journal-title":"Educ Psychol Meas"},{"key":"73_CR21","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1207\/s15327906mbr1402_9","volume":"14","author":"S Janson","year":"1979","unstructured":"Janson S, Vegelius J (1979) On generalizations of the G index and the Phi coefficient to nominal scales. Multivar Behav Res 14: 255\u2013269","journal-title":"Multivar Behav Res"},{"key":"73_CR22","doi-asserted-by":"crossref","first-page":"461","DOI":"10.1007\/BF02296208","volume":"44","author":"HC Kraemer","year":"1979","unstructured":"Kraemer HC (1979) Ramifications of a population model for \u03ba as a coefficient of reliability. Psychometrika 44: 461\u2013472","journal-title":"Psychometrika"},{"key":"73_CR23","doi-asserted-by":"crossref","first-page":"207","DOI":"10.2307\/2529972","volume":"36","author":"HC Kraemer","year":"1980","unstructured":"Kraemer HC (1980) Extensions of the kappa coefficient. Biometrics 36: 207\u2013216","journal-title":"Biometrics"},{"key":"73_CR24","doi-asserted-by":"crossref","first-page":"2109","DOI":"10.1002\/sim.1180","volume":"21","author":"HC Kraemer","year":"2002","unstructured":"Kraemer HC, Periyakoil VS, Noda A (2002) Tutorial in biostatistics: kappa coefficients in medical research. Stat Med 21: 2109\u20132129","journal-title":"Stat Med"},{"key":"73_CR25","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1007\/BF00167603","volume":"21","author":"K Krippendorff","year":"1987","unstructured":"Krippendorff K (1987) Association, agreement, and equity. Qual Quant 21: 109\u2013123","journal-title":"Qual Quant"},{"key":"73_CR26","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33: 159\u2013174","journal-title":"Biometrics"},{"key":"73_CR27","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1037\/h0031643","volume":"76","author":"RJ Light","year":"1971","unstructured":"Light RJ (1971) Measures of response agreement for qualitative data: some generalizations and alternatives. Psychol Bull 76: 365\u2013377","journal-title":"Psychol Bull"},{"key":"73_CR28","volume-title":"Elementary inequalities","author":"DS Mitrinovi\u0107","year":"1964","unstructured":"Mitrinovi\u0107 DS (1964) Elementary inequalities. P. Noordhoff, Groningen"},{"key":"73_CR29","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1038\/modpathol.3800514","volume":"19","author":"FP O\u2019Malley","year":"2006","unstructured":"O\u2019Malley FP, Mohsin SK, Badve S, Bose S, Collins LC, Ennis M, Kleer CG, Pinder SE, Schnitt SJ (2006) Interobserver reproducibility in the diagnosis of flat epithelial atypia of the breast. Mod Pathol 19: 172\u2013179","journal-title":"Mod Pathol"},{"key":"73_CR30","unstructured":"Popping R (1983) Overeenstemmingsmaten voor nominale data. PhD thesis, Rijksuniversiteit Groningen, Groningen"},{"key":"73_CR31","unstructured":"Randolph JJ (2005) Free-marginal multirater kappa (multirater \u03ba free): an alternative to Fleiss\u2019 fixed-Marginal multirater kappa. Paper presented at the Joensuu Learning and Instruction Symposium, Joensuu, Finland"},{"key":"73_CR32","doi-asserted-by":"crossref","first-page":"497","DOI":"10.1002\/bimj.4710220605","volume":"22","author":"HJA Schouten","year":"1980","unstructured":"Schouten HJA (1980) Measuring agreement among many observers. Biom J 22: 497\u2013504","journal-title":"Biom J"},{"key":"73_CR33","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1002\/bimj.4710240502","volume":"24","author":"HJA Schouten","year":"1982","unstructured":"Schouten HJA (1982) Measuring pairwise agreement among many observers. Biom J 24: 431\u2013435","journal-title":"Biom J"},{"key":"73_CR34","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1007\/BF02294066","volume":"51","author":"HJA Schouten","year":"1986","unstructured":"Schouten HJA (1986) Nominal scale agreement among observers. Psychometrika 51: 453\u2013466","journal-title":"Psychometrika"},{"key":"73_CR35","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1086\/266577","volume":"19","author":"WA Scott","year":"1955","unstructured":"Scott WA (1955) Reliability of content analysis: the case of nominal scale coding. Public Opin Q 19: 321\u2013325","journal-title":"Public Opin Q"},{"key":"73_CR36","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1016\/j.stamet.2008.06.001","volume":"6","author":"S Vanbelle","year":"2009","unstructured":"Vanbelle S, Albert A (2009) A note on the linearly weighted kappa coefficient for ordinal scales. Stat Methodol 6: 157\u2013163","journal-title":"Stat Methodol"},{"key":"73_CR37","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1007\/s11336-008-9059-y","volume":"73","author":"MJ Warrens","year":"2008","unstructured":"Warrens MJ (2008a) On similarity coefficients for 2\u00a0\u00d7 2 tables and correction for chance. Psychometrika 73: 487\u2013502","journal-title":"Psychometrika"},{"key":"73_CR38","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1007\/s00357-008-9024-6","volume":"25","author":"MJ Warrens","year":"2008","unstructured":"Warrens MJ (2008b) Bounds of resemblance measures for binary (presence\/absence) variables. J Classif 25: 195\u2013208","journal-title":"J Classif"},{"key":"73_CR39","doi-asserted-by":"crossref","first-page":"777","DOI":"10.1007\/s11336-008-9070-3","volume":"73","author":"MJ Warrens","year":"2008","unstructured":"Warrens MJ (2008c) On association coefficients for 2 \u00d7 2 tables and properties that do not depend on the marginal distributions. Psychometrika 73: 777\u2013789","journal-title":"Psychometrika"},{"key":"73_CR40","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1007\/s00357-008-9023-7","volume":"25","author":"MJ Warrens","year":"2008","unstructured":"Warrens MJ (2008d) On the equivalence of Cohen\u2019s kappa and the Hubert-Arabie adjusted Rand index. J Classif 25: 177\u2013183","journal-title":"J Classif"},{"key":"73_CR41","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1007\/s00357-008-9006-8","volume":"25","author":"MJ Warrens","year":"2008","unstructured":"Warrens MJ (2008e) On the indeterminacy of resemblance measures for (presence\/absence) data. J Classif 25: 125\u2013136","journal-title":"J Classif"},{"key":"73_CR42","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1007\/s11336-009-9138-8","volume":"75","author":"MJ Warrens","year":"2010","unstructured":"Warrens MJ (2010a) Inequalities between kappa and kappa-like statistics for k\u00a0\u00d7 k tables. Psychometrika 75: 176\u2013185","journal-title":"Psychometrika"},{"key":"73_CR43","doi-asserted-by":"crossref","unstructured":"Warrens MJ (2010b) A formal proof of a paradox associated with Cohen\u2019s kappa. J Classif (in press)","DOI":"10.1007\/s00357-010-9060-x"},{"key":"73_CR44","doi-asserted-by":"crossref","first-page":"673","DOI":"10.1016\/j.stamet.2010.05.003","volume":"7","author":"MJ Warrens","year":"2010","unstructured":"Warrens MJ (2010c) Cohen\u2019s kappa can always be increased and decreased by combining categories. Stat Methodol 7: 673\u2013677","journal-title":"Stat Methodol"},{"key":"73_CR45","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1007\/s11336-010-9155-7","volume":"75","author":"MJ Warrens","year":"2010","unstructured":"Warrens MJ (2010d) A Kraemer-type rescaling that transforms the odds ratio into the weighted kappa coefficient. Psychometrika 75: 328\u2013330","journal-title":"Psychometrika"},{"key":"73_CR46","doi-asserted-by":"crossref","first-page":"374","DOI":"10.1037\/0033-2909.103.3.374","volume":"103","author":"R Zwick","year":"1988","unstructured":"Zwick R (1988) Another look at interrater agreement. Psychol Bull 103: 374\u2013378","journal-title":"Psychol Bull"}],"container-title":["Advances in Data Analysis and Classification"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.springerlink.com\/index\/pdf\/10.1007\/s11634-010-0073-4","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,3,5]],"date-time":"2019-03-05T11:09:35Z","timestamp":1551784175000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s11634-010-0073-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,10,10]]},"references-count":46,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2010,12]]}},"alternative-id":["73"],"URL":"https:\/\/doi.org\/10.1007\/s11634-010-0073-4","relation":{},"ISSN":["1862-5347","1862-5355"],"issn-type":[{"value":"1862-5347","type":"print"},{"value":"1862-5355","type":"electronic"}],"subject":[],"published":{"date-parts":[[2010,10,10]]}}}