{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T12:44:51Z","timestamp":1792759491991,"version":"4.5.0"},"reference-count":48,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T00:00:00Z","timestamp":1660694400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJERPH"],"abstract":"<jats:p>The need to search for new measures describing the classification of a logistic regression model stems from the difficulty in searching for previously unknown factors that predict the occurrence of a disease. A classification quality assessment can be performed by testing the change in the area under the receiver operating characteristic curve (AUC). Another approach is to use the Net Reclassification Improvement (NRI), which is based on a comparison between the predicted risk, determined on the basis of the basic model, and the predicted risk that comes from the model enriched with an additional factor. In this paper, we draw attention to Cohen\u2019s Kappa coefficient, which examines the actual agreement in the correction of a random agreement. We proposed to extend this coefficient so that it may be used to detect the quality of a logistic regression model reclassification. The results provided by Kappa\u2018s reclassification were compared with the results obtained using NRI. The random variables\u2019 distribution attached to the model on the classification change, measured by NRI, Kappa, and AUC, was presented. A simulation study was conducted on the basis of a cohort containing 3971 Poles obtained during the implementation of a lower limb atherosclerosis prevention program.<\/jats:p>","DOI":"10.3390\/ijerph191610213","type":"journal-article","created":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T21:23:56Z","timestamp":1660771436000},"page":"10213","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":75,"title":["Cohen\u2019s Kappa Coefficient as a Measure to Assess Classification Improvement following the Addition of a New Marker to a Regression Model"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1811-2583","authenticated-orcid":false,"given":"Barbara","family":"Wi\u0119ckowska","sequence":"first","affiliation":[{"name":"Department of Computer Science and Statistics, Poznan University of Medical Sciences, 60-806 Poznan, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Katarzyna B.","family":"Kubiak","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Statistics, Poznan University of Medical Sciences, 60-806 Poznan, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paulina","family":"J\u00f3\u017awiak","sequence":"additional","affiliation":[{"name":"Department of Preventive Medicine, Poznan University of Medical Sciences, 60-781 Poznan, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wac\u0142aw","family":"Moryson","sequence":"additional","affiliation":[{"name":"Department of Epidemiology and Hygiene, Chair of Social Medicine, Poznan University of Medical Sciences, 60-806 Poznan, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Barbara","family":"Stawi\u0144ska-Witoszy\u0144ska","sequence":"additional","affiliation":[{"name":"Department of Epidemiology and Hygiene, Chair of Social Medicine, Poznan University of Medical Sciences, 60-806 Poznan, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,8,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1016\/j.jclinepi.2015.04.005","article-title":"Prediction Models Need Appropriate Internal, Internal-External, and External Validation","volume":"69","author":"Steyerberg","year":"2016","journal-title":"J. Clin. Epidemiol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1038\/nmeth.4210","article-title":"Interpreting P Values","volume":"14","author":"Altman","year":"2017","journal-title":"Nat. Methods"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"279","DOI":"10.4300\/JGME-D-12-00156.1","article-title":"Using Effect Size\u2014Or Why the P Value Is Not Enough","volume":"4","author":"Sullivan","year":"2012","journal-title":"J. Grad. Med. Educ."},{"key":"ref_4","unstructured":"American Psychological Association (2010). Publication Manual of the American Psychological Association, American Psychological Association. [6th ed.]."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10649-019-09908-4","article-title":"Beyond Small, Medium, or Large: Points of Consideration When Interpreting Effect Sizes","volume":"102","author":"Bakker","year":"2019","journal-title":"Educ. Stud. Math."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"917","DOI":"10.1093\/jpepsy\/jsp004","article-title":"How to Select, Calculate, and Interpret Effect Sizes","volume":"34","author":"Durlak","year":"2009","journal-title":"J. Pediatric Psychol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"989","DOI":"10.1037\/a0019507","article-title":"A Comprehensive Review of Effect Size Reporting and Interpreting Practices in Academic Journals in Education and Psychology","volume":"102","author":"Sun","year":"2010","journal-title":"J. Educ. Psychol."},{"key":"ref_8","unstructured":"Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, L. Erlbaum Associates. [2nd ed.]."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/01973533.2015.1012991","article-title":"Editorial","volume":"37","author":"Trafimow","year":"2015","journal-title":"Basic Appl. Soc. Psychol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1086\/266577","article-title":"Reliability of Content Analysis: The Case of Nominal Scale Coding","volume":"19","author":"Scott","year":"1955","journal-title":"Public Opin. Q."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A Coefficient of Agreement for Nominal Scales","volume":"20","author":"Cohen","year":"1960","journal-title":"Educ. Psychol. Meas."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1465","DOI":"10.1002\/sim.4780100913","article-title":"A Comparison of Methods for Calculating a Stratified Kappa","volume":"10","author":"Barlow","year":"1991","journal-title":"Stat. Med."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1037\/h0026256","article-title":"Weighted Kappa: Nominal Scale Agreement Provision for Scaled Disagreement or Partial Credit","volume":"70","author":"Cohen","year":"1968","journal-title":"Psychol. Bull."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1053","DOI":"10.1016\/0895-4356(96)00057-1","article-title":"The Statistical Analysis of Kappa Statistics in Multiple Samples","volume":"49","author":"Donner","year":"1996","journal-title":"J. Clin. Epidemiol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1002\/sim.4780140308","article-title":"Modelling Covariate Effects in Observer Agreement Studies: The Case of Nominal Scale Agreement","volume":"14","author":"Graham","year":"1995","journal-title":"Stat. Med."},{"key":"ref_16","first-page":"309","article-title":"Simple Moment Estimates of the \u03ba-Coefficient and Its Variance","volume":"43","author":"Lipsitz","year":"1994","journal-title":"J. R. Stat. Soc. Ser. C (Appl. Stat.)"},{"key":"ref_17","first-page":"45","article-title":"An Estimating Equations Approach for Modelling Kappa","volume":"42","author":"Klar","year":"2000","journal-title":"Biom. J. J. Math. Methods Biosci."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1093\/biostatistics\/1.2.191","article-title":"Modeling Kappa for Measuring Dependent Categorical Agreement Data","volume":"1","author":"Williamson","year":"2000","journal-title":"Biostatistics"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2109","DOI":"10.1002\/sim.1180","article-title":"Kappa Coefficients in Medical Research","volume":"21","author":"Periyakoil","year":"2002","journal-title":"Statist. Med."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"928","DOI":"10.1161\/CIRCULATIONAHA.106.672402","article-title":"Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction","volume":"115","author":"Cook","year":"2007","journal-title":"Circulation"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1093\/aje\/kws207","article-title":"Interpreting Incremental Value of Markers Added to Risk Prediction Models","volume":"176","author":"Pencina","year":"2012","journal-title":"Am. J. Epidemiol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"661","DOI":"10.1002\/sim.5598","article-title":"Predictive Accuracy of Risk Factors and Markers: A Simulation Study of the Effect of Novel Markers on Different Performance Measures for Logistic Regression Models","volume":"32","author":"Austin","year":"2013","journal-title":"Stat. Med."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1002\/sim.2929","article-title":"Evaluating the Added Predictive Ability of a New Marker: From Area under the ROC Curve to Reclassification and Beyond","volume":"27","author":"Pencina","year":"2008","journal-title":"Stat. Med."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1002\/sim.4085","article-title":"Extensions of Net Reclassification Improvement Calculations to Measure Usefulness of New Biomarkers","volume":"30","author":"Pencina","year":"2011","journal-title":"Stat. Med."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"4455","DOI":"10.1002\/sim.7041","article-title":"Net Reclassification Index at Event Rate: Properties and Relationships","volume":"36","author":"Pencina","year":"2017","journal-title":"Stat. Med."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"282","DOI":"10.1007\/s12561-014-9118-0","article-title":"The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets","volume":"7","author":"Pepe","year":"2015","journal-title":"Stat. Biosci."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"3405","DOI":"10.1002\/sim.5804","article-title":"A Note on the Evaluation of Novel Biomarkers: Do Not Rely on Integrated Discrimination Improvement and Net Reclassification Index","volume":"33","author":"Hilden","year":"2014","journal-title":"Stat. Med."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1097\/EDE.0000000000000018","article-title":"Net Reclassification Indices for Evaluating Risk-Prediction Instruments: A Critical Review","volume":"25","author":"Kerr","year":"2014","journal-title":"Epidemiology"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"122","DOI":"10.7326\/M13-1522","article-title":"Net Reclassification Improvement: Computation, Interpretation, and Controversies: A Literature Review and Clinician\u2019s Guide","volume":"160","author":"Leening","year":"2014","journal-title":"Ann. Intern. Med."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"488","DOI":"10.1093\/aje\/kws208","article-title":"Clinically Relevant Measures of Fit? A Note of Caution","volume":"176","author":"Cook","year":"2012","journal-title":"Am. J. Epidemiol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"dju041","DOI":"10.1093\/jnci\/dju041","article-title":"Net Risk Reclassification P Values: Valid or Misleading?","volume":"106","author":"Pepe","year":"2014","journal-title":"J. Natl. Cancer Inst."},{"key":"ref_32","first-page":"11","article-title":"Net Reclassification Index and Integrated Discrimination Index Are Not Appropriate for Testing Whether a Biomarker Improves Predictive Performance","volume":"156","author":"Burch","year":"2017","journal-title":"Toxicol. Sci."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1703","DOI":"10.1515\/CCLM.2010.340","article-title":"Statistical Methods for Assessment of Added Usefulness of New Biomarkers","volume":"48","author":"Pencina","year":"2010","journal-title":"Clin. Chem. Lab. Med."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1880","DOI":"10.1093\/eurheartj\/ehy345","article-title":"Application of Net Reclassification Index to Non-Nested and Point-Based Risk Prediction Models: A Review","volume":"40","author":"Thomas","year":"2019","journal-title":"Eur. Heart J."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"935","DOI":"10.1002\/bimj.201500029","article-title":"Assessing Agreement with Multiple Raters on Correlated Kappa Statistics","volume":"58","author":"Cao","year":"2016","journal-title":"Biom. J."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"999","DOI":"10.1016\/S0140-6736(13)61752-3","article-title":"The Framingham Heart Study and the Epidemiology of Cardiovascular Disease: A Historical Perspective","volume":"383","author":"Mahmood","year":"2014","journal-title":"Lancet"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"2315","DOI":"10.1093\/eurheartj\/ehw106","article-title":"2016 European Guidelines on Cardiovascular Disease Prevention in Clinical Practice: The Sixth Joint Task Force of the European Society of Cardiology and Other Societies on Cardiovascular Disease Prevention in Clinical Practice (Constituted by Representatives of 10 Societies and by Invited Experts)Developed with the Special Contribution of the European Association for Cardiovascular Prevention & Rehabilitation (EACPR)","volume":"37","author":"Piepoli","year":"2016","journal-title":"Eur. Heart J."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"987","DOI":"10.1016\/S0195-668X(03)00114-3","article-title":"Estimation of Ten-Year Risk of Fatal Cardiovascular Disease in Europe: The SCORE Project","volume":"24","author":"Conroy","year":"2003","journal-title":"Eur. Heart J."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"958","DOI":"10.5603\/KP.2015.0182","article-title":"[A new version of cardiovascular risk assessment system and risk charts calibrated for Polish population]","volume":"73","author":"Zdrojewski","year":"2015","journal-title":"Kardiol. Pol."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"2577","DOI":"10.1002\/sim.5328","article-title":"Misuse of DeLong Test to Compare AUCs for Nested Models","volume":"31","author":"Demler","year":"2012","journal-title":"Stat. Med."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"796","DOI":"10.1001\/jama.2012.9630","article-title":"Common Carotid Intima-Media Thickness Measurements in Cardiovascular Risk Prediction: A Meta-Analysis","volume":"308","author":"Peters","year":"2012","journal-title":"JAMA"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"438","DOI":"10.7326\/0003-4819-156-6-201203200-00006","article-title":"Evaluation of Newer Risk Markers for Coronary Heart Disease Risk Classification: A Cohort Study","volume":"156","author":"Kavousi","year":"2012","journal-title":"Ann. Intern. Med."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"788","DOI":"10.1001\/jama.2012.9624","article-title":"Comparison of Novel Risk Markers for Improvement in Cardiovascular Risk Assessment in Intermediate-Risk Individuals","volume":"308","author":"Yeboah","year":"2012","journal-title":"JAMA"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1002\/sim.2987","article-title":"Comments on \u2018Evaluating the Added Predictive Ability of a New Marker: From Area under the ROC Curve to Reclassification and beyond\u2019 by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002\/Sim.2929)","volume":"27","author":"Cook","year":"2008","journal-title":"Stat. Med."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1002\/bimj.201000078","article-title":"Performance of Reclassification Statistics in Comparing Risk Prediction Models","volume":"53","author":"Cook","year":"2011","journal-title":"Biom. J."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1177\/0272989X12461856","article-title":"A Bias-Corrected Net Reclassification Improvement for Clinical Subgroups","volume":"33","author":"Paynter","year":"2013","journal-title":"Med. Decis. Mak."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"840","DOI":"10.1177\/1741826711410256","article-title":"Risk Prediction of Incident Coronary Heart Disease in The Netherlands: Re-Estimation and Improvement of the SCORE Risk Function","volume":"19","author":"Merry","year":"2012","journal-title":"Eur. J. Prev. Cardiol."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"e3318","DOI":"10.1136\/bmj.e3318","article-title":"Comparisons of Established Risk Prediction Models for Cardiovascular Disease: Systematic Review","volume":"344","author":"Siontis","year":"2012","journal-title":"BMJ"}],"container-title":["International Journal of Environmental Research and Public Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1660-4601\/19\/16\/10213\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:11:10Z","timestamp":1760141470000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1660-4601\/19\/16\/10213"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,17]]},"references-count":48,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["ijerph191610213"],"URL":"https:\/\/doi.org\/10.3390\/ijerph191610213","relation":{},"ISSN":["1660-4601"],"issn-type":[{"value":"1660-4601","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,17]]}}}