{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T18:20:47Z","timestamp":1773080447900,"version":"3.50.1"},"reference-count":52,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,4,10]],"date-time":"2022-04-10T00:00:00Z","timestamp":1649548800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Classification is one of the main problems of machine learning, and assessing the quality of classification is one of the most topical tasks, all the more difficult as it depends on many factors. Many different measures have been proposed to assess the quality of the classification, often depending on the application of a specific classifier. However, in most cases, these measures are focused on binary classification, and for the problem of many decision classes, they are significantly simplified. Due to the increasing scope of classification applications, there is a growing need to select a classifier appropriate to the situation, including more complex data sets with multiple decision classes. This paper aims to propose a new measure of classifier quality assessment (called the preference-driven measure, abbreviated p-d), regardless of the number of classes, with the possibility of establishing the relative importance of each class. Furthermore, we propose a solution in which the classifier\u2019s assessment can be adapted to the analyzed problem using a vector of preferences. To visualize the operation of the proposed measure, we present it first on an example involving two decision classes and then test its operation on real, multi-class data sets. Additionally, in this case, we demonstrate how to adjust the assessment to the user\u2019s preferences. The results obtained allow us to confirm that the use of a preference-driven measure indicates that other classifiers are better to use according to preferences, particularly as opposed to the classical measures of classification quality assessment.<\/jats:p>","DOI":"10.3390\/e24040531","type":"journal-article","created":{"date-parts":[[2022,4,10]],"date-time":"2022-04-10T06:02:54Z","timestamp":1649570574000},"page":"531","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Preference-Driven Classification Measure"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2128-6998","authenticated-orcid":false,"given":"Jan","family":"Kozak","sequence":"first","affiliation":[{"name":"Department of Machine Learning, University of Economics in Katowice, 1 Maja 50, 40-287 Katowice, Poland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5122-2645","authenticated-orcid":false,"given":"Barbara","family":"Probierz","sequence":"additional","affiliation":[{"name":"Department of Machine Learning, University of Economics in Katowice, 1 Maja 50, 40-287 Katowice, Poland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7123-6972","authenticated-orcid":false,"given":"Krzysztof","family":"Kania","sequence":"additional","affiliation":[{"name":"Department of Knowledge Engineering, University of Economics in Katowice, 1 Maja 50, 40-287 Katowice, Poland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7893-5410","authenticated-orcid":false,"given":"Przemys\u0142aw","family":"Juszczuk","sequence":"additional","affiliation":[{"name":"Department of Machine Learning, University of Economics in Katowice, 1 Maja 50, 40-287 Katowice, Poland"}]}],"member":"1968","published-online":{"date-parts":[[2022,4,10]]},"reference":[{"key":"ref_1","unstructured":"G\u00f6sgens, M., Zhiyanov, A., Tikhonov, A., and Prokhorenkova, L. (2021, January 6\u201314). Good Classification Measures and How to Find Them. Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS 2021), Online."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"124","DOI":"10.1016\/j.knosys.2016.11.017","article-title":"Ensemble feature selection: Homogeneous and heterogeneous approaches","volume":"118","year":"2017","journal-title":"Knowl.-Based Syst."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Lewis, D.D., and Catlett, J. (1994). Heterogeneous uncertainty sampling for supervised learning. Machine Learning Proceedings 1994, Elsevier.","DOI":"10.1016\/B978-1-55860-335-6.50026-X"},{"key":"ref_4","unstructured":"Campagner, A., Sconfienza, L., and Cabitza, F. (2020). H-accuracy, an alternative metric to assess classification models in medicine. Digital Personalized Health and Medicine, IOS Press."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Gilli, M., and Schumann, E. (2015). Accuracy and precision in finance. Available SSRN 2698114.","DOI":"10.2139\/ssrn.2698114"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"14623","DOI":"10.1007\/s00521-021-06103-6","article-title":"BenchMetrics: A systematic benchmarking method for binary classification performance metrics","volume":"33","author":"Canbek","year":"2021","journal-title":"Neural Comput. Appl."},{"key":"ref_7","first-page":"105","article-title":"Power to the people: The role of humans in interactive machine learning","volume":"35","author":"Amershi","year":"2014","journal-title":"Ai Mag."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., and He, L. (2021). A Survey of Human-in-the-loop for Machine Learning. arXiv.","DOI":"10.1016\/j.future.2022.05.014"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Talbot, J., Lee, B., Kapoor, A., and Tan, D.S. (2009, January 4\u20139). EnsembleMatrix: Interactive visualization to support machine learning with multiple classifiers. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA.","DOI":"10.1145\/1518701.1518895"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3359152","article-title":"The principles and limits of algorithm-in-the-loop decision making","volume":"3","author":"Green","year":"2019","journal-title":"Proc. ACM Hum. -Comput. Interact."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1007\/BF00153760","article-title":"Information-Based Evaluation Criterion for Classifier\u2019s Performance","volume":"6","author":"Kononenko","year":"1991","journal-title":"Mach. Learn."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Valverde-Albacete, F.J., and Pel\u00e1ez-Moreno, C. (2014). 100% classification accuracy considered harmful: The normalized information transfer factor explains the accuracy paradox. PLoS ONE, 9.","DOI":"10.1371\/journal.pone.0084217"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Saito, T., and Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0118432"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"875","DOI":"10.1016\/j.engappai.2007.01.001","article-title":"A lot of randomness is hiding in accuracy","volume":"20","year":"2007","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_15","first-page":"24","article-title":"Beyond accuracy, F-score and ROC: A family of discriminant measures for performance evaluation","volume":"Volume 4304","author":"Sokolova","year":"2006","journal-title":"Australasian Joint Conference on Artificial Intelligence"},{"key":"ref_16","unstructured":"Grandini, M., Bagli, E., and Visani, G. (2020). Metrics for multi-class classification: An overview. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.5121\/ijdkp.2015.5201","article-title":"A Review on Evaluation Metrics for Data Classification Evaluations","volume":"5","author":"Hossin","year":"2015","journal-title":"Int. J. Data Min. I Knowl. Manag. Process"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1016\/j.patrec.2008.08.010","article-title":"An experimental comparison of performance measures for classification","volume":"30","author":"Ferri","year":"2009","journal-title":"Pattern Recognit. Lett."},{"key":"ref_19","first-page":"1","article-title":"A comprehensive survey of error measures for evaluating binary decision making in data science","volume":"9","author":"Moutari","year":"2019","journal-title":"Wiley Interdiscip. Rev. Data Min. Knowl. Discov."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1016\/j.ipm.2009.03.002","article-title":"A systematic analysis of performance measures for classification tasks","volume":"45","author":"Sokolova","year":"2009","journal-title":"Inf. Process. Manag."},{"key":"ref_21","first-page":"1","article-title":"Statistical comparisons of classifiers over multiple data sets","volume":"7","year":"2006","journal-title":"J. Mach. Learn. Res."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1016\/j.aci.2018.08.003","article-title":"Classification assessment methods","volume":"17","author":"Tharwat","year":"2018","journal-title":"Appl. Comput. Inform."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1016\/j.neucom.2016.02.001","article-title":"A classification performance measure considering the degree of classification difficulty","volume":"193","author":"Zhang","year":"2016","journal-title":"Neurocomputing"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"598","DOI":"10.1016\/j.ins.2021.08.094","article-title":"An instance-oriented performance measure for classification","volume":"580","author":"Yu","year":"2021","journal-title":"Inf. Sci."},{"key":"ref_25","first-page":"11","article-title":"A Novel Performance Measure for Machine Learning Classification","volume":"13","author":"Gong","year":"2021","journal-title":"Int. J. Manag. Inf. Technol."},{"key":"ref_26","first-page":"60","article-title":"A two dimensional accuracy-based measure for classification performance","volume":"382\u2013383","year":"2017","journal-title":"Inf. Sci."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2863","DOI":"10.1016\/j.procs.2021.09.057","article-title":"Automatic system for IBD diagnosis","volume":"192","author":"Kasperczuk","year":"2021","journal-title":"Procedia Comput. Sci."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1016\/j.compag.2013.05.004","article-title":"Robust pixel-based classification of obstacles for robotic harvesting of sweet-pepper","volume":"96","author":"Bac","year":"2013","journal-title":"Comput. Electron. Agric."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1203","DOI":"10.1016\/j.patrec.2007.01.015","article-title":"Volume measure in 2DPCA-based face recognition","volume":"28","author":"Meng","year":"2007","journal-title":"Pattern Recognit. Lett."},{"key":"ref_30","unstructured":"Burduk, R. (2020). Classification Performance Metric for Imbalance Data Based on Recall and Selectivity Normalized in Class Labels. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"451","DOI":"10.1007\/s10994-021-05964-1","article-title":"F*: An interpretable transformation of the F-measure","volume":"110","author":"Hand","year":"2021","journal-title":"Mach. Learn."},{"key":"ref_32","unstructured":"Mitchell, T.M. (1997). Machine Learning, International Edition, McGraw-Hill Education."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"40","DOI":"10.3758\/BF03213026","article-title":"Theoretical analysis of an alphabetic confusion matrix","volume":"9","author":"Townsend","year":"1971","journal-title":"Percept. Psychophys."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1023\/A:1007442505281","article-title":"Glossary of terms","volume":"30","author":"Provost","year":"1998","journal-title":"J. Mach. Learn."},{"key":"ref_35","first-page":"27","article-title":"Confusion Matrix","volume":"6","author":"Room","year":"2019","journal-title":"Mach. Learn."},{"key":"ref_36","unstructured":"Lee, N., Yang, H., and Yoo, H. (2021). A surrogate loss function for optimization of F\u03b2 score in binary classification with imbalanced data. arXiv."},{"key":"ref_37","unstructured":"Van Rijsbergen, C.J. (1979). Information Retrieval, Butterworth-Heinemann."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1002\/(SICI)1097-4571(199401)45:1<12::AID-ASI2>3.0.CO;2-L","article-title":"The relationship between recall and precision","volume":"45","author":"Buckland","year":"1994","journal-title":"J. Am. Soc. Inf. Sci."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12864-019-6413-7","article-title":"The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation","volume":"21","author":"Chicco","year":"2020","journal-title":"BMC Genom."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1016\/0005-2795(75)90109-9","article-title":"Comparison of the predicted and observed secondary structure of T4 phage lysozyme","volume":"405","author":"Matthews","year":"1975","journal-title":"Biochim. Biophys. Acta (BBA)-Protein Struct."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Brodersen, K.H., Ong, C.S., Stephan, K.E., and Buhmann, J.M. (2010, January 23\u201326). The Balanced Accuracy and Its Posterior Distribution. Proceedings of the 2010 20th International Conference on Pattern Recognition, Istanbul, Turkey.","DOI":"10.1109\/ICPR.2010.764"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"45","DOI":"10.4103\/0301-4738.37595","article-title":"Understanding and using sensitivity, specificity and predictive values","volume":"56","author":"Parikh","year":"2008","journal-title":"Indian J. Ophthalmol."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.4018\/jdwm.2007070101","article-title":"Multi-label classification: An overview","volume":"3","author":"Tsoumakas","year":"2007","journal-title":"Int. J. Data Warehous. Min. (IJDWM)"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"4961","DOI":"10.1007\/s10489-021-02635-5","article-title":"Confidence interval for micro-averaged F1 and macro-averaged F1 scores","volume":"28","author":"Takahashi","year":"2022","journal-title":"Appl. Intell."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Jurman, G., Riccadonna, S., and Furlanello, C. (2012). A comparison of MCC and CEN error measures in multi-class prediction. PLoS ONE, 7.","DOI":"10.1371\/journal.pone.0041882"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1016\/j.compbiolchem.2004.09.006","article-title":"Comparing two K-category assignments by a K-category correlation coefficient","volume":"28","author":"Gorodkin","year":"2004","journal-title":"Comput. Biol. Chem."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1145\/1656274.1656278","article-title":"The WEKA data mining software: An update","volume":"11","author":"Hall","year":"2009","journal-title":"ACM SIGKDD Explor. Newsl."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1007\/BF00058655","article-title":"Bagging predictors","volume":"24","author":"Breiman","year":"1996","journal-title":"Mach. Learn."},{"key":"ref_49","unstructured":"Bouckaert, R.R. (2004). Bayesian Network Classifiers in Weka, University of Waikato. Working Paper No. 14\/2004."},{"key":"ref_50","unstructured":"Kohavi, R. (1995, January 25\u201327). The Power of Decision Tables. Proceedings of the 8th European Conference on Machine Learning, Crete, Greece."},{"key":"ref_51","unstructured":"Quinlan, R. (1993). C4.5: Programs for Machine Learning, Morgan Kaufmann Publishers."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random Forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/4\/531\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:51:18Z","timestamp":1760136678000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/4\/531"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,10]]},"references-count":52,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,4]]}},"alternative-id":["e24040531"],"URL":"https:\/\/doi.org\/10.3390\/e24040531","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,10]]}}}