{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,1]],"date-time":"2026-10-01T16:04:59Z","timestamp":1790870699976,"version":"4.1.0"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"7","license":[{"start":{"date-parts":[[2024,4,9]],"date-time":"2024-04-09T00:00:00Z","timestamp":1712620800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Brazilian Research Agencies FAPESP","award":["2021\/06870-3"],"award-info":[{"award-number":["2021\/06870-3"]}]},{"DOI":"10.13039\/501100003593","name":"CNPq","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100003593","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2024,7,31]]},"abstract":"<jats:p>Machine Learning solutions have spread along many domains, including critical applications. The development of such models usually relies on a dataset containing labeled data. This dataset is then split into training and test sets and the accuracy of the models in replicating the test labels is assessed. This process is often iterated in a cross-validation procedure for obtaining average performance estimates. But is the average of the predictive performance on test sets enough for assessing the trustfulness of a Machine Learning model? This paper discusses the importance of knowing which individual observations of a dataset are more challenging than others and how this characteristic can be measured and used in order to improve classification performance and trustfulness. A set of strategies for measuring the hardness level of the instances of a dataset is surveyed and a Python package containing their implementation is provided.<\/jats:p>","DOI":"10.1145\/3615354","type":"journal-article","created":{"date-parts":[[2023,8,10]],"date-time":"2023-08-10T06:25:24Z","timestamp":1691648724000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":29,"title":["Trusting My Predictions: On the Value of Instance-Level Analysis"],"prefix":"10.1145","volume":"56","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6140-571X","authenticated-orcid":false,"given":"Ana C.","family":"Lorena","sequence":"first","affiliation":[{"name":"Instituto Tecnol\u00f3gico de Aeron\u00e1utica, S\u00e3o Jos\u00e9 dos Campos, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9768-7563","authenticated-orcid":false,"given":"Pedro Y. A.","family":"Paiva","sequence":"additional","affiliation":[{"name":"Instituto Tecnol\u00f3gico de Aeron\u00e1utica, S\u00e3o Jos\u00e9 dos Campos, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7084-1233","authenticated-orcid":false,"given":"Ricardo B. C.","family":"Prud\u00eancio","sequence":"additional","affiliation":[{"name":"Centro de Inform\u00e1tica, Universidade Federal de Pernambuco, Recife, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,4,9]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-61380-8_33"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2020.12.006"},{"key":"e_1_3_2_4_2","volume-title":"Pattern Recognition and Machine Learning","author":"Bishop Christopher M.","year":"2006","unstructured":"Christopher M. Bishop and Nasser M. Nasrabadi. 2006. Pattern Recognition and Machine Learning, Vol. 4. Springer."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.is.2020.101641"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00058655"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2016.7727774"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2021.3127698"},{"key":"e_1_3_2_10_2","series-title":"Proceedings of Machine Learning Research","first-page":"1013","volume":"89","author":"Chen Yu","year":"2019","unstructured":"Yu Chen, Telmo M. Silva Filho, Ricardo B. Prudencio, Tom Diethe, and Peter Flach. 2019. \\(\\beta ^3\\) -IRT: A new item response model and its applications. In Proceedings of Machine Learning Research(Proceedings of Machine Learning Research, Vol. 89). PMLR, 1013\u20131021."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-98074-4"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1214\/aos\/1013203451"},{"key":"e_1_3_2_13_2","volume-title":"Advances in Neural Information Processing Systems","author":"Hardt Moritz","year":"2016","unstructured":"Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29. Curran Associates, Inc.https:\/\/proceedings.neurips.cc\/paper\/2016\/file\/9d2682367c3935defcb1f9e247a97c0d-Paper.pdf"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/34.990132"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-021-03825-4"},{"key":"e_1_3_2_16_2","first-page":"5546","volume-title":"Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3\u20138, 2018, Montr\u00e9al, Canada","author":"Jiang Heinrich","year":"2018","unstructured":"Heinrich Jiang, Been Kim, Melody Y. Guan, and Maya R. Gupta. 2018. To trust or not to trust a classifier. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3\u20138, 2018, Montr\u00e9al, Canada, Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol\u00f2 Cesa-Bianchi, and Roman Garnett (Eds.). 5546\u20135557."},{"key":"e_1_3_2_17_2","first-page":"175","volume-title":"Fourth International Workshop on Learning with Imbalanced Domains: Theory and Applications","author":"Komorniczak Joanna","year":"2022","unstructured":"Joanna Komorniczak, Pawe\u0142 Ksieniewicz, and Micha\u0142 Wo\u017aniak. 2022. Data complexity and classification accuracy correlation in oversampling algorithms. In Fourth International Workshop on Learning with Imbalanced Domains: Theory and Applications. PMLR, 175\u2013186."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10489-022-03793-w"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2014.2327034"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2014.10.001"},{"key":"e_1_3_2_21_2","article-title":"Data complexity in machine learning","author":"Li Ling","year":"2006","unstructured":"Ling Li and Yaser S. Abu-Mostafa. 2006. Data complexity in machine learning. Technical Report CaltechCSTR:2006.004 (2006).","journal-title":"Technical Report CaltechCSTR:2006.004"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3347711"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2018.09.004"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i7.20739"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2021.108076"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-017-5629-5"},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","first-page":"177","DOI":"10.5753\/eniac.2021.18251","volume-title":"Anais do XVIII Encontro Nacional de Intelig\u00eancia Artificial e Computacional","author":"Nunes Gustavo H.","year":"2021","unstructured":"Gustavo H. Nunes, Gustavo O. Martins, Carlos H. Q. Forster, and Ana C. Lorena. 2021. Using instance hardness measures in curriculum learning. In Anais do XVIII Encontro Nacional de Intelig\u00eancia Artificial e Computacional. SBC, 177\u2013188."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-022-06205-9"},{"key":"e_1_3_2_29_2","article-title":"PyHard: A novel tool for generating hardness embeddings to support data-centric analysis","author":"Paiva Pedro Yuri Arbs","year":"2021","unstructured":"Pedro Yuri Arbs Paiva, Kate Smith-Miles, Maria Gabriela Valeriano, and Ana Carolina Lorena. 2021. PyHard: A novel tool for generating hardness embeddings to support data-centric analysis. arXiv preprint arXiv:2109.14430 (2021).","journal-title":"arXiv preprint arXiv:2109.14430"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_3_2_31_2","article-title":"Subgroup mining for performance analysis of regression models (to appear)","author":"Pimentel J.","year":"2022","unstructured":"J. Pimentel, P. J. Azevedo, and L. Torgo. 2022. Subgroup mining for performance analysis of regression models (to appear). Expert Systems (2022).","journal-title":"Expert Systems"},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1007\/978-3-030-46147-8_6","volume-title":"Machine Learning and Knowledge Discovery in Databases","author":"Prud\u00eancio Ricardo B. C.","year":"2020","unstructured":"Ricardo B. C. Prud\u00eancio. 2020. Cost sensitive evaluation of instance hardness in machine learning. In Machine Learning and Knowledge Discovery in Databases, Ulf Brefeld, Elisa Fromont, Andreas Hotho, Arno Knobbe, Marloes Maathuis, and C\u00e9line Robardet (Eds.). Springer International Publishing, 86\u2013102."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-21686-2_38"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00116251"},{"key":"e_1_3_2_35_2","first-page":"1","volume-title":"21st International Scientific Conference AMSE Applications of Mathematics and Statistics in Economics","author":"\u0158ezankov\u00e1 Hana","year":"2018","unstructured":"Hana \u0158ezankov\u00e1. 2018. Different approaches to the silhouette coefficient calculation in cluster evaluation. In 21st International Scientific Conference AMSE Applications of Mathematics and Statistics in Economics. 1\u201310."},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-012-0570-1"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-04191-5_17"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2022.08.017"},{"key":"e_1_3_2_39_2","series-title":"The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16\u201318 April 2019, Naha, Okinawa, Japan","first-page":"1022","volume":"89","author":"Schulam Peter","year":"2019","unstructured":"Peter Schulam and Suchi Saria. 2019. Can you trust this prediction? Auditing pointwise reliability after learning. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16\u201318 April 2019, Naha, Okinawa, Japan(Proceedings of Machine Learning Research, Vol. 89), Kamalika Chaudhuri and Masashi Sugiyama (Eds.). PMLR, 1022\u20131031."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-013-5422-z"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/1456650.1456656"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2018.08.004"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2022.09.010"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2020.113397"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.5555\/1481236"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05318-5_2"},{"key":"e_1_3_2_47_2","volume-title":"Artificial Neural Networks: Approximation and Learning Theory","author":"White Halbert","year":"1992","unstructured":"Halbert White. 1992. Artificial Neural Networks: Approximation and Learning Theory. Blackwell Cambridge, Mass."},{"key":"e_1_3_2_48_2","first-page":"1","article-title":"Instance hardness and multivariate Gaussian distribution-based oversampling technique for imbalance classification","author":"Xie Jie","year":"2023","unstructured":"Jie Xie, Mingying Zhu, Kai Hu, and Jinglan Zhang. 2023. Instance hardness and multivariate Gaussian distribution-based oversampling technique for imbalance classification. Pattern Analysis and Applications (2023), 1\u201315.","journal-title":"Pattern Analysis and Applications"},{"key":"e_1_3_2_49_2","article-title":"Fair class balancing: Enhancing model fairness without observing sensitive attributes","author":"Yan Shen","year":"2020","unstructured":"Shen Yan, Hsien-Te Kao, and Emilio Ferrara. 2020. Fair class balancing: Enhancing model fairness without observing sensitive attributes. Proceedings of the 29th ACM International Conference on Information & Knowledge Management (2020).","journal-title":"Proceedings of the 29th ACM International Conference on Information & Knowledge Management"},{"issue":"2","key":"e_1_3_2_50_2","first-page":"3","article-title":"The optimality of naive Bayes","volume":"1","author":"Zhang Harry","year":"2004","unstructured":"Harry Zhang. 2004. The optimality of naive Bayes. AAAI 1, 2 (2004), 3.","journal-title":"AAAI"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3615354","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3615354","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:36:53Z","timestamp":1750178213000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3615354"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,9]]},"references-count":49,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2024,7,31]]}},"alternative-id":["10.1145\/3615354"],"URL":"https:\/\/doi.org\/10.1145\/3615354","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,9]]},"assertion":[{"value":"2022-10-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-08-03","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}