{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,11]],"date-time":"2026-02-11T12:38:06Z","timestamp":1770813486398,"version":"3.50.1"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,2,13]],"date-time":"2024-02-13T00:00:00Z","timestamp":1707782400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"NSFC","doi-asserted-by":"crossref","award":["62076178"],"award-info":[{"award-number":["62076178"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"TJF","award":["22ZYYYJC00020"],"award-info":[{"award-number":["22ZYYYJC00020"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2024,5,31]]},"abstract":"<jats:p>\u2018\u2018Easy\/hard sample\u201d is a popular parlance in machine learning. Learning difficulty of samples refers to how easy\/hard a sample is during a learning procedure. An increasing need of measuring learning difficulty demonstrates its importance in machine learning (e.g., difficulty-based weighting learning strategies). Previous literature has proposed a number of learning difficulty measures. However, no comprehensive investigation for learning difficulty is available to date, resulting in that nearly all existing measures are heuristically defined without a rigorous theoretical foundation. This study attempts to conduct a pilot theoretical study for learning difficulty of samples. First, influential factors for learning difficulty are summarized. Under various situations conducted by summarized influential factors, correlations between learning difficulty and two vital criteria of machine learning, namely, generalization error and model complexity, are revealed. Second, a theoretical definition of learning difficulty is proposed on the basis of these two criteria. A practical measure of learning difficulty is proposed under the direction of the theoretical definition by importing the bias-variance trade-off theory. Subsequently, the rationality of theoretical definition and the practical measure are demonstrated, respectively, by analysis of several classical weighting methods and abundant experiments realized under all situations conducted by summarized influential factors. The mentioned weighting methods can be reasonably explained under the proposed theoretical definition and concerned propositions. The comparison in these experiments indicates that the proposed measure significantly outperforms the other measures throughout the experiments.<\/jats:p>","DOI":"10.1145\/3636512","type":"journal-article","created":{"date-parts":[[2023,12,7]],"date-time":"2023-12-07T11:53:57Z","timestamp":1701950037000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Exploring the Learning Difficulty of Data: Theory and Measure"],"prefix":"10.1145","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7610-295X","authenticated-orcid":false,"given":"Weiyao","family":"Zhu","sequence":"first","affiliation":[{"name":"Center for Applied Mathematics, Tianjin University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6106-6914","authenticated-orcid":false,"given":"Ou","family":"Wu","sequence":"additional","affiliation":[{"name":"Center for Applied Mathematics, Tianjin University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3824-8306","authenticated-orcid":false,"given":"Fengguang","family":"Su","sequence":"additional","affiliation":[{"name":"Center for Applied Mathematics, Tianjin University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8467-7526","authenticated-orcid":false,"given":"Yingjun","family":"Deng","sequence":"additional","affiliation":[{"name":"Center for Applied Mathematics, Tianjin University, China"}]}],"member":"320","published-online":{"date-parts":[[2024,2,13]]},"reference":[{"key":"e_1_3_6_2_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2015.2458858"},{"key":"e_1_3_6_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01012"},{"key":"e_1_3_6_4_2","doi-asserted-by":"publisher","unstructured":"E. Aguilar-Torres B. Nagarajan R. Khatun M. Bola\u00f1os and P. Radeva. 2021. Uncertainty modeling and deep learning applied to food image analysis. Biomed. Eng. Syst. Technol. (2021) 3\u201316. 10.5220\/0009429400090016","DOI":"10.5220\/0009429400090016"},{"key":"e_1_3_6_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3429447"},{"key":"e_1_3_6_6_2","doi-asserted-by":"publisher","unstructured":"L. Arkesteijn and S. Pande. 2013. On hydrological model complexity its geometrical interpretations and prediction uncertainty. Water Resour. Res. (2013). 10.1002\/wrcr.20529","DOI":"10.1002\/wrcr.20529"},{"key":"e_1_3_6_7_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1706.05394"},{"key":"e_1_3_6_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE51399.2021.00219"},{"key":"e_1_3_6_9_2","unstructured":"J. Astola and P. Kuosmanen. 1997. Fundamentals of Nonlinear Digital Filtering. CRC Press Boca Raton FL."},{"key":"e_1_3_6_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/18.720554"},{"key":"e_1_3_6_11_2","volume-title":"Proceedings of the International Conference on Machine Learning. 41\u201348","author":"Bengio Y.","unstructured":"Y. Bengio, J. Louradour, R. Collobert, and J. Weston. 2009. Curriculum learning. In Proceedings of the International Conference on Machine Learning. 41\u201348."},{"key":"e_1_3_6_12_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2009.14119"},{"key":"e_1_3_6_13_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_3_6_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2018.07.011"},{"key":"e_1_3_6_15_2","volume-title":"Proceedings of the Conference on Neural Information Processing Systems, 1\u201312","author":"Castells T.","unstructured":"T. Castells, P. Weinzaepfel, and J. Revaud, 2020. SuperLoss: A generic loss for robust curriculum learning. In Proceedings of the Conference on Neural Information Processing Systems, 1\u201312."},{"key":"e_1_3_6_16_2","unstructured":"S. Chatterjee and P. Zielinski. 2022. On the generalization mystery in deep learning. Retrieved from https:\/\/arXiv:2203.1003"},{"key":"e_1_3_6_17_2","doi-asserted-by":"publisher","unstructured":"C. Cortes and V. Vapnik. 1993. Support-vector networks. Mach. Learn. (1993). 10.1007\/BF00994018","DOI":"10.1007\/BF00994018"},{"key":"e_1_3_6_18_2","volume-title":"McClelland","author":"Rumelhart E.","year":"1987","unstructured":"David E. Rumelhart and James L. McClelland. 1987. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations, MIT Press, Cambridge, MA, 318\u2013362."},{"key":"e_1_3_6_19_2","doi-asserted-by":"publisher","unstructured":"D. D\u2019souza Z. Nussbaum C. Agarwal and S. Hooker. 2021. A tale of two long tails. IEEE Trans. Neural Netw. Learn. Syst. (2021) 1716\u20131721. 10.1109\/TNNLS.2016.2546956","DOI":"10.1109\/TNNLS.2016.2546956"},{"key":"e_1_3_6_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISCAS.1990.112581"},{"key":"e_1_3_6_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW50498.2020.00010"},{"key":"e_1_3_6_22_2","doi-asserted-by":"publisher","unstructured":"Y. Duan and O. Wu. 2016. Learning with auxiliary less-noisy labels. IEEE Trans. Neural Netw. Learn. Syst. (2016) 1716\u20131721. 10.1109\/TNNLS.2016.2546956","DOI":"10.1109\/TNNLS.2016.2546956"},{"key":"e_1_3_6_23_2","unstructured":"R. Dwivedi C. Singh B. Yu and M. J. Wainwright. 2020. Revisiting complexity and the bias-variance tradeoff. Retrieved from https:\/\/arXiv2006.10189"},{"key":"e_1_3_6_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2020.107339"},{"key":"e_1_3_6_25_2","doi-asserted-by":"publisher","DOI":"10.1006\/jcss.1997.1504"},{"key":"e_1_3_6_26_2","unstructured":"B. Fu X. Zhao Y. Li X. Wang and Y. Reng. 2018. A convolutional neural networks denoising approach for salt and pepper noise. Multimedia Tools Appl. (2018) 1\u201318."},{"key":"e_1_3_6_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patrec.2020.03.008"},{"key":"e_1_3_6_28_2","doi-asserted-by":"publisher","unstructured":"W. Ge W. Huang D. Dong and M. R. Scott. 2018. Deep metric learning with hierarchical triplet loss. In Proceedings of the European Conference on Computer Vision Lecture Notes in Computer Science. 272\u2013288. 10.48550\/arXiv.1810.06951","DOI":"10.48550\/arXiv.1810.06951"},{"key":"e_1_3_6_29_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1992.4.1.1"},{"key":"e_1_3_6_30_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1810.00383"},{"key":"e_1_3_6_31_2","unstructured":"B. Han Q. Yao T. Liu G. Niu I. W. Tsang J. T. Kwo and M. Suigiyama. 2020. A survey of label-noise representation learning. Retrieved from https:\/\/arXiv:2011.04406"},{"key":"e_1_3_6_32_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1804.06872"},{"key":"e_1_3_6_33_2","doi-asserted-by":"publisher","unstructured":"T. Hastie R. Tibshirani and J. H. Friedman. 2009. The Elements of Statistical Learning. Springer Berlin. 10.1007\/978-0-387-21606-5","DOI":"10.1007\/978-0-387-21606-5"},{"key":"e_1_3_6_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_6_35_2","doi-asserted-by":"publisher","DOI":"10.1162\/089976698300017232"},{"key":"e_1_3_6_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2021.3079209"},{"key":"e_1_3_6_37_2","doi-asserted-by":"crossref","unstructured":"H. Huang and Q. Yang. 2020. Large scale analysis of generalization error in learning using margin based classification methods. Retrieved from https:\/\/arXiv:1901.08057","DOI":"10.1088\/1742-5468\/abbed5"},{"key":"e_1_3_6_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00342"},{"key":"e_1_3_6_39_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1712.05055"},{"key":"e_1_3_6_40_2","unstructured":"P. Kaushik A. Gain A. Kortylewski and A. L. Yuille. Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping. Retrieved from https:\/\/arxiv.org\/abs\/2102.11343"},{"key":"e_1_3_6_41_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1703.04977"},{"key":"e_1_3_6_42_2","volume-title":"Learning Multiple Layers of Features from Tiny Images. Technical report","author":"Krizhevsky A.","unstructured":"A. Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto."},{"key":"e_1_3_6_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3065386"},{"key":"e_1_3_6_44_2","doi-asserted-by":"publisher","DOI":"10.14778\/3407790.3407842"},{"key":"e_1_3_6_45_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1811.05181"},{"key":"e_1_3_6_46_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1904.09483"},{"key":"e_1_3_6_47_2","doi-asserted-by":"publisher","unstructured":"M. Li and P. Vit\u00e1nyi. 1997. An Introduction to Kolmogorov Complexity and Its Applications. Springer Berlin. 10.1007\/978-3-030-11298-1","DOI":"10.1007\/978-3-030-11298-1"},{"key":"e_1_3_6_48_2","unstructured":"J. Z. Lin and J. Bradic. 2021. Learning to combat noisy labels via classification margins. Retrieved from https:\/\/arXiv:2102.00751."},{"key":"e_1_3_6_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.324"},{"key":"e_1_3_6_50_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.03500"},{"key":"e_1_3_6_51_2","doi-asserted-by":"crossref","unstructured":"A. Maalouf G. Eini B. Mussay D. Feldman and M. Osadchy. 2021. A unified approach to coreset learning. Retrieved from https:\/\/arXiv:2111.03044","DOI":"10.1109\/TNNLS.2022.3213169"},{"key":"e_1_3_6_52_2","volume-title":"Proceedings of the International Conference on Machine Learning Workshop Deep Phenomena. 16","author":"Mangalam K.","unstructured":"K. Mangalam and V. Prabhu. 2019. Do deep neural networks learn shallow learnable examples first? In Proceedings of the International Conference on Machine Learning Workshop Deep Phenomena. 16. 10.1016"},{"key":"e_1_3_6_53_2","unstructured":"Lawrence Mosley. 2013. A balanced approach to the multi-class imbalance problem."},{"key":"e_1_3_6_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983888"},{"key":"e_1_3_6_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2012.6247954"},{"key":"e_1_3_6_56_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1906.02629"},{"key":"e_1_3_6_57_2","doi-asserted-by":"crossref","unstructured":"C. Oinar B. M. Le and S. S. Woo. 2022. KappaFace: Adaptive additive angular margin loss for deep face recognition. Retrieved from https:\/\/arXiv2201.07394","DOI":"10.1109\/ACCESS.2023.3338648"},{"key":"e_1_3_6_58_2","unstructured":"M. Pagliardini G. O. Manunza M. Jaggi M. I. Jordan and T. Thavdarova. 2022. Increasing the classification margin with uncertainty driven perturbations. Retrieved from https:\/\/arXiv2201:2202.05737"},{"key":"e_1_3_6_59_2","doi-asserted-by":"publisher","DOI":"10.1186\/s40537-021-00447-5"},{"key":"e_1_3_6_60_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2020.107585"},{"key":"e_1_3_6_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"e_1_3_6_62_2","unstructured":"M. Sharma A. Yadav S. Soman and Jayadeva. 2019. Effect of various regularizers on model complexities of neural networks in presence of input noise. Retrieved from https:\/\/arXiv2201:1901.11458"},{"key":"e_1_3_6_63_2","unstructured":"W. Shin J. Ha S. Li Y. Cho H. Song and S. Kwon. 2020. Which strategies matter for noisy label classification? insight into loss and uncertainty. Retrieved from https:\/\/arXiv2201:2008.06218"},{"key":"e_1_3_6_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3517252"},{"key":"e_1_3_6_65_2","unstructured":"M. Toneva A. Sordoni R. T. Combes A. Trischler Y. Bengio and G. J. Gordon. An Empirical Study of Example Forgetting during Deep Neural Network Learning. Retrieved from https:\/\/arxiv.org\/abs\/1812.05159."},{"key":"e_1_3_6_66_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1805.08522"},{"key":"e_1_3_6_67_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1505.07634"},{"key":"e_1_3_6_68_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2108.09335"},{"key":"e_1_3_6_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441800"},{"key":"e_1_3_6_70_2","doi-asserted-by":"publisher","unstructured":"X. Wang Y. Hua E. Kodirov and N. M. Robertson. 2021. Ranked list loss for deep metric learning. IEEE Trans. Pattern Anal. Mach. Intell. (2021) 5207\u20135216. 10.48550\/arXiv.1903.03238","DOI":"10.48550\/arXiv.1903.03238"},{"key":"e_1_3_6_71_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1901.06783"},{"key":"e_1_3_6_72_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.01441"},{"key":"e_1_3_6_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298885"},{"key":"e_1_3_6_74_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1811.07253"},{"key":"e_1_3_6_75_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2103.15209"},{"key":"e_1_3_6_76_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2202.13170"},{"key":"e_1_3_6_77_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2002.11328"},{"key":"e_1_3_6_78_2","volume-title":"SUD: Supervision by denoising for medical image segmentation.","author":"Young S. I.","year":"2022","unstructured":"S. I. Young, A. V. Dalca, E. Ferrante, P. Golland, B. Fischl, and J. E. Iglesias. 2022. SUD: Supervision by denoising for medical image segmentation. Retrieved from https:\/\/arXiv:2202.02952"},{"key":"e_1_3_6_79_2","doi-asserted-by":"publisher","DOI":"10.3233\/JIFS-201758"},{"key":"e_1_3_6_80_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2010.01736"},{"key":"e_1_3_6_81_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1812.06410"},{"key":"e_1_3_6_82_2","unstructured":"X. Zhou and O. Wu. 2021. Which samples should be learnt first? Easy or hard? Retrieved from https:\/\/arXiv:2110.05481"},{"key":"e_1_3_6_83_2","volume-title":"Proceedings of the IEEE 20th International Symposium on Biomedical Imaging (ISBI\u201923)","author":"Zhou Z.","unstructured":"Z. Zhou, J. Luo, D. Arefan, G. Kitamura, and S. Wu. 2023. Human not in the loop: Objective sample difficulty measures for curriculum learning. In Proceedings of the IEEE 20th International Symposium on Biomedical Imaging (ISBI\u201923). 1\u20135."}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3636512","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3636512","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:54:11Z","timestamp":1750287251000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3636512"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,13]]},"references-count":82,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,5,31]]}},"alternative-id":["10.1145\/3636512"],"URL":"https:\/\/doi.org\/10.1145\/3636512","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,13]]},"assertion":[{"value":"2022-07-17","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-29","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}