{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:01:20Z","timestamp":1760241680079,"version":"build-2065373602"},"reference-count":27,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2018,6,28]],"date-time":"2018-06-28T00:00:00Z","timestamp":1530144000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001655","name":"Deutscher Akademischer Austauschdienst","doi-asserted-by":"publisher","award":["PPP Italien"],"award-info":[{"award-number":["PPP Italien"]}],"id":[{"id":"10.13039\/501100001655","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>In this article, a two-tiered 2D tool is described, called \u27e8\u03c6,\u03b4\u27e9 diagrams, and this tool has been devised to support the assessment of classifiers in terms of accuracy and bias. In their standard versions, these diagrams provide information, as the underlying data were in fact balanced. Their generalization, i.e., ability to account for the imbalance, will be also briefly described. In either case, the isometrics of accuracy and bias are immediately evident therein, as\u2014according to a specific design choice\u2014they are in fact straight lines parallel to the x-axis and y-axis, respectively. \u27e8\u03c6,\u03b4\u27e9 diagrams can also be used to assess the importance of features, as highly discriminant ones are immediately evident therein. In this paper, a comprehensive introduction on how to adopt \u27e8\u03c6,\u03b4\u27e9 diagrams as a standard tool for classifier and feature assessment is given. In particular, with the goal of illustrating all relevant details from a pragmatic perspective, their implementation and usage as Python and R packages will be described.<\/jats:p>","DOI":"10.3390\/make1010007","type":"journal-article","created":{"date-parts":[[2018,6,28]],"date-time":"2018-06-28T10:53:33Z","timestamp":1530183213000},"page":"121-137","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Phi-Delta-Diagrams: Software Implementation of a Visual Tool for Assessing Classifier and Feature Performance"],"prefix":"10.3390","volume":"1","author":[{"given":"Giuliano","family":"Armano","sequence":"first","affiliation":[{"name":"Department of Electrical and Electronic Engineering, University of Cagliari, 09124 Cagliari, Italy"}]},{"given":"Alessandro","family":"Giuliani","sequence":"additional","affiliation":[{"name":"Department of Electrical and Electronic Engineering, University of Cagliari, 09124 Cagliari, Italy"}]},{"given":"Ursula","family":"Neumann","sequence":"additional","affiliation":[{"name":"Department of Mathematics &amp; Computer Science, University of Marburg, 35037 Marburg, Germany"}]},{"given":"Nikolas","family":"Rothe","sequence":"additional","affiliation":[{"name":"Department of Mathematics &amp; Computer Science, University of Marburg, 35037 Marburg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3108-8311","authenticated-orcid":false,"given":"Dominik","family":"Heider","sequence":"additional","affiliation":[{"name":"Department of Mathematics &amp; Computer Science, University of Marburg, 35037 Marburg, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2018,6,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"626","DOI":"10.1016\/j.gie.2014.02.1028","article-title":"Endoscopic management is the treatment of choice for bile leaks after liver resection","volume":"80","author":"Jochum","year":"2014","journal-title":"Gastrointest. Endosc."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1177\/153303460900800503","article-title":"A computational approach for the identification of small GTPases based on preprocessed amino acid sequences","volume":"8","author":"Heider","year":"2009","journal-title":"Technol. Cancer Res. Treat."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"402","DOI":"10.1016\/j.jneumeth.2010.11.007","article-title":"Dynamic causal modeling with genetic algorithms","volume":"194","author":"Pyka","year":"2011","journal-title":"J. Neurosci. Methods"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.ins.2003.03.023","article-title":"A hybrid genetic-neural architecture for stock indexes forecasting","volume":"170","author":"Armano","year":"2005","journal-title":"Inf. Sci."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1038\/nphys4035","article-title":"Machine learning phases of matter","volume":"13","author":"Carrasquilla","year":"2017","journal-title":"Nat. Phys."},{"key":"ref_6","unstructured":"Hand, D. (1997). Construction and Assessment of Classification Rules, Wiley."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Pepe, M. (2004). The Statistical Evaluation of Medical Tests for Classification and Prediction, Oxford University Press.","DOI":"10.1093\/oso\/9780198509844.001.0001"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","article-title":"An introduction to ROC analysis","volume":"27","author":"Fawcett","year":"2006","journal-title":"Patt. Recognit. Lett."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1038\/nmeth.3945","article-title":"Points of Significance: Classification Evaluation","volume":"13","author":"Lever","year":"2016","journal-title":"Nat. Methods"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"849","DOI":"10.1016\/S0031-3203(02)00257-1","article-title":"Strategies for learning in class imbalance problems","volume":"36","author":"Barandela","year":"2003","journal-title":"Patt. Recognit."},{"key":"ref_11","unstructured":"Elazmeh, W., Japkowicz, N., and Matwin, S. (2006, January 29). A framework for comparative evaluation of classifiers in the presence of class imbalance. Proceedings of the third Workshop on ROC Analysis in Machine Learning, Pittsburgh, PA, USA."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Guo, X., Yin, Y., Dong, C., Yang, G., and Zhou, G. (2008, January 18\u201320). On the class imbalance problem. Proceedings of the Fourth International Conference on Natural Computation, Jinan, China.","DOI":"10.1109\/ICNC.2008.871"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1007\/s10994-005-5011-x","article-title":"Roc \u2018n\u2019 Rule Learning\u2014Towards a Better Understanding of Covering Algorithms","volume":"58","author":"Flach","year":"2005","journal-title":"Mach. Learn."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1007\/s10994-006-8199-5","article-title":"Cost curves: An improved method for visualizing classifier performance","volume":"65","author":"Drummond","year":"2006","journal-title":"Mach. Learn."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"875","DOI":"10.1016\/j.engappai.2007.01.001","article-title":"A lot of randomness is hiding in accuracy","volume":"20","year":"2007","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"959","DOI":"10.1007\/s00500-008-0392-y","article-title":"A study of statistical techniques and performance measures for genetics-based machine learning: accuracy and interpretability","volume":"13","author":"Luengo","year":"2009","journal-title":"Soft Comput."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"466","DOI":"10.1016\/j.ins.2015.07.028","article-title":"A Direct Measure of Discriminant and Characteristic Capability for Classifier Building and Assessment","volume":"325","author":"Armano","year":"2015","journal-title":"Inf. Sci."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Bellman, R. (1961). Adaptive Control Processes, Princeton University Press.","DOI":"10.1515\/9781400874668"},{"key":"ref_19","first-page":"253","article-title":"VII. Mathematical contributions to the theory of evolution.\u2014III. Regression, heredity, and panmixia","volume":"187","author":"Pearson","year":"1896","journal-title":"Philos. Trans. R. Soc. A"},{"key":"ref_20","unstructured":"Cramer, H. (1946). Mathematical Methods of Statistics, Princeton University Press."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Armano, G., and Giuliani, A. (2018). A two-tiered 2D Visual Tool for Assessing Classifier Performance. Inf. Sci., in press.","DOI":"10.1016\/j.ins.2018.06.052"},{"key":"ref_22","unstructured":"Kalinov, P., Stantic, B., and Sattar, A. (2010, January 18\u201322). Building a dynamic classifier for large text data collections. Proceedings of the Twenty-First Australasian Database Conference, Brisbane, Australia."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1015","DOI":"10.1007\/s11192-014-1292-9","article-title":"Automatic Classification of Academic Web Page Types","volume":"101","author":"Kenekayoro","year":"2014","journal-title":"Scientometrics"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1007\/s10618-015-0428-8","article-title":"Exploiting link structure for web page genre identification","volume":"30","author":"Zhu","year":"2016","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1007\/s00521-013-1490-z","article-title":"Predicting phishing websites based on self-structuring neural network","volume":"25","author":"Mohammad","year":"2014","journal-title":"Neural Comput. Appl."},{"key":"ref_26","unstructured":"Zipf, G. (1949). Human Behavior and the Principle of Least Effort, Addison Wesley."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1186\/s13040-017-0142-8","article-title":"EFS: an ensemble feature selection tool implemented as R-package and web-application","volume":"10","author":"Neumann","year":"2017","journal-title":"BioData Min."}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/1\/1\/7\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:10:37Z","timestamp":1760195437000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/1\/1\/7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,6,28]]},"references-count":27,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2019,3]]}},"alternative-id":["make1010007"],"URL":"https:\/\/doi.org\/10.3390\/make1010007","relation":{},"ISSN":["2504-4990"],"issn-type":[{"type":"electronic","value":"2504-4990"}],"subject":[],"published":{"date-parts":[[2018,6,28]]}}}