{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,24]],"date-time":"2026-02-24T15:20:20Z","timestamp":1771946420462,"version":"3.50.1"},"reference-count":103,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2024,6,27]],"date-time":"2024-06-27T00:00:00Z","timestamp":1719446400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62372021"],"award-info":[{"award-number":["62372021"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,7,31]]},"abstract":"<jats:p>Defect prediction is crucial for software quality assurance and has been extensively researched over recent decades. However, prior studies rarely focus on data complexity in defect prediction tasks, and even less on understanding the difficulties of these tasks from the perspective of data complexity. In this article, we conduct an empirical study to estimate the hardness of over 33,000 instances, employing a set of measures to characterize the inherent difficulty of instances and the characteristics of defect datasets. Our findings indicate that: (1) instance hardness in both classes displays a right-skewed distribution, with the defective class exhibiting a more scattered distribution; (2) class overlap is the primary factor influencing instance hardness and can be characterized through feature, structural, and instance-level overlap; (3) no universal preprocessing technique is applicable to all datasets, and it may not consistently reduce data complexity, fortunately, dataset complexity measures can help identify suitable techniques for specific datasets; (4)\u00a0integrating data complexity information into the learning process can enhance an algorithm\u2019s learning capacity. In summary, this empirical study highlights the crucial role of data complexity in defect prediction tasks, and provides a novel perspective for advancing research in defect prediction techniques.<\/jats:p>","DOI":"10.1145\/3649596","type":"journal-article","created":{"date-parts":[[2024,2,26]],"date-time":"2024-02-26T12:35:42Z","timestamp":1708950942000},"page":"1-45","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Data Complexity: A New Perspective for Analyzing the Difficulty of Defect Prediction Tasks"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6498-8570","authenticated-orcid":false,"given":"Xiaohui","family":"Wan","sequence":"first","affiliation":[{"name":"Beihang University, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7922-9067","authenticated-orcid":false,"given":"Zheng","family":"Zheng","sequence":"additional","affiliation":[{"name":"Beihang University, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3407-5161","authenticated-orcid":false,"given":"Fangyun","family":"Qin","sequence":"additional","affiliation":[{"name":"Capital Normal University, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6839-5633","authenticated-orcid":false,"given":"Xuhui","family":"Lu","sequence":"additional","affiliation":[{"name":"Beihang University, Beijing, China"}]}],"member":"320","published-online":{"date-parts":[[2024,6,27]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180197"},{"key":"e_1_3_3_3_2","volume-title":"Data Complexity in Supervised Learning: A Far-Reaching Implication","author":"Antol\u00ednez N\u00faria Maci\u00e0","year":"2011","unstructured":"N\u00faria Maci\u00e0 Antol\u00ednez. 2011. Data Complexity in Supervised Learning: A Far-Reaching Implication. Ph. D. Dissertation. Universitat Ramon Llull."},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-61380-8_33"},{"key":"e_1_3_3_5_2","first-page":"10","volume-title":"Proceedings of the WOB","author":"Batista Gustavo EAPA","year":"2003","unstructured":"Gustavo EAPA Batista, Ana L. C. Bazzan, and Maria Carolina Monard. 2003. Balancing training data for automated annotation of keywords: A case study. In Proceedings of the WOB. 10\u201318."},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/1007730.1007735"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2731766"},{"key":"e_1_3_3_8_2","first-page":"115","volume-title":"Proceedings of the 30th International Conference on International Conference on Machine Learning","author":"Bergstra James","year":"2013","unstructured":"James Bergstra, Daniel Yamins, and David Cox. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the 30th International Conference on International Conference on Machine Learning. 115\u2013123."},{"key":"e_1_3_3_9_2","article-title":"An overview of data mining techniques","author":"Berson Alex","year":"2004","unstructured":"Alex Berson, Stephen Smith, and Kurt Thearling. 2004. An overview of data mining techniques. Building Data Mining Application for CRM (2004).","journal-title":"Building Data Mining Application for CRM"},{"key":"e_1_3_3_10_2","doi-asserted-by":"crossref","first-page":"3665","DOI":"10.1016\/j.patcog.2014.05.003","article-title":"Dynamic selection of classifiers\u2013a comprehensive review","author":"Jr Alceu S. Britto","year":"2014","unstructured":"Alceu S. Britto Jr, Robert Sabourin, and Luiz ES Oliveira. 2014. Dynamic selection of classifiers\u2013a comprehensive review. Pattern Recognition 47, 11 (2014), 3665\u20133680.","journal-title":"Pattern Recognition"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2017.10.038"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0378-3758(02)00269-0"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2008.10.027"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2968520"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-020-05570-7"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2010.5463279"},{"key":"e_1_3_3_18_2","volume-title":"Straightforward Statistics for the Behavioral Sciences.","author":"Evans James D.","year":"1996","unstructured":"James D. Evans. 1996. Straightforward Statistics for the Behavioral Sciences.Thomson Brooks\/Cole Publishing Co."},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2016.04.017"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.3233\/IDA-194803"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.91"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2017.18"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2019.2895462"},{"key":"e_1_3_3_24_2","first-page":"698","volume-title":"Proceedings of the 34th IEEE\/ACM International Conference on Automated Software Engineering","author":"Gong Lina","year":"2020","unstructured":"Lina Gong, Shujuan Jiang, Rongcun Wang, and Li Jiang. 2020. Empirical evaluation of the impact of class overlap on software defect prediction. In Proceedings of the 34th IEEE\/ACM International Conference on Automated Software Engineering. 698\u2013709."},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3220740"},{"key":"e_1_3_3_26_2","doi-asserted-by":"crossref","unstructured":"Arnulf B. A. Graf and Silvio Borer. 2001. Normalization in support vector machines. In Proceedings of the 23rd DAGM-Symposium on Pattern Recognition. 277\u2013282.","DOI":"10.1007\/3-540-45404-7_37"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.5555\/3305381.3305518"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2011.103"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070510"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2008.239"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3182542"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/34.990132"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/1868328.1868342"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2018.00137"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2016.56"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2007.70773"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9437-5"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2022.11.056"},{"issue":"1","key":"e_1_3_3_39_2","first-page":"3","article-title":"Supervised machine learning: A review of classification techniques","volume":"160","author":"Kotsiantis Sotiris B.","year":"2007","unstructured":"Sotiris B. Kotsiantis, Ioannis Zaharakis, and P. Pintelas. 2007. Supervised machine learning: A review of classification techniques. Emerging Artificial Intelligence Applications in Computer Engineering 160, 1 (2007), 3\u201324.","journal-title":"Emerging Artificial Intelligence Applications in Computer Engineering"},{"key":"e_1_3_3_40_2","first-page":"3474","volume-title":"Proceedings of the 28th Conference on Neural Information Processing Systems","author":"Kuleshov Volodymyr","year":"2015","unstructured":"Volodymyr Kuleshov and Percy Liang. 2015. Calibrated structured prediction. In Proceedings of the 28th Conference on Neural Information Processing Systems. 3474\u20133482."},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.5555\/2595490.2595492"},{"issue":"17","key":"e_1_3_3_42_2","first-page":"1","article-title":"Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning","volume":"18","author":"Lema\u00eetre Guillaume","year":"2017","unstructured":"Guillaume Lema\u00eetre, Fernando Nogueira, and Christos K. Aridas. 2017. Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning. Journal of Machine Learning Research 18, 17 (2017), 1\u20135.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2014.2327034"},{"issue":"6","key":"e_1_3_3_44_2","first-page":"1","article-title":"Feature selection: A data perspective","volume":"50","year":"2017","unstructured":"Jundong Li, Kewei Cheng, Suhang Wang, Fred Morstatter, Robert P. Trevino, Jiliang Tang, and Huan Liu. 2017. Feature selection: A data perspective. ACM Computing Surveys 50, 6 (2017), 1\u201345.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380360"},{"key":"e_1_3_3_46_2","volume-title":"Data Complexity in Machine Learning","author":"Li Ling","year":"2006","unstructured":"Ling Li and Yaser S. Abu-Mostafa. 2006. Data Complexity in Machine Learning. Technical Report CaltechCSTR:2006.004. California Institute of Technology."},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1049\/iet-sen.2017.0148"},{"key":"e_1_3_3_48_2","first-page":"841","volume-title":"Proceedings of the 36th IEEE International Conference on Data Engineering","year":"2020","unstructured":"Zhining Liu, Wei Cao, Zhifeng Gao, Jiang Bian, Hechang Chen, Yi Chang, and Tieyan Liu. 2020. Self-paced ensemble for highly imbalanced massive data classification. In Proceedings of the 36th IEEE International Conference on Data Engineering. 841\u2013852."},{"key":"e_1_3_3_49_2","first-page":"14463","volume-title":"Proceedings of the 34th Conference on Neural Information Processing Systems","author":"Liu Zhining","year":"2020","unstructured":"Zhining Liu, Pengfei Wei, Jing Jiang, Wei Cao, Jiang Bian, and Yi Chang. 2020. MESA: Boost ensemble imbalanced learning with meta-sampler. In Proceedings of the 34th Conference on Neural Information Processing Systems. 14463\u201314474."},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3347711"},{"key":"e_1_3_3_51_2","article-title":"Trusting my predictions: On the value of Instance-Level analysis","author":"Lorena Ana C.","year":"2023","unstructured":"Ana C. Lorena, Pedro Y. A. Paiva, and Ricardo B. C. Prud\u00eancio. 2023. Trusting my predictions: On the value of Instance-Level analysis. ACM Computing Surveys (2023).","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2019.2944962"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-013-0700-4"},{"issue":"8","key":"e_1_3_3_54_2","first-page":"1695","article-title":"Data complexity analysis for software defect detection","volume":"14","author":"Ma Ying","year":"2018","unstructured":"Ying Ma, Yichang Li, Junwen Lu, Peng Sun, Yu Sun, and Xiatian Zhu. 2018. Data complexity analysis for software defect detection. International Journal of Performability Engineering 14, 8 (2018), 1695.","journal-title":"International Journal of Performability Engineering"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/3477.938265"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2007.256941"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-010-0069-5"},{"key":"e_1_3_3_58_2","volume-title":"Interpretable Machine Learning","author":"Molnar Christoph","year":"2020","unstructured":"Christoph Molnar. 2020. Interpretable Machine Learning. Lulu. com."},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534405"},{"key":"e_1_3_3_60_2","unstructured":"Daniel M\u00fcllner. 2011. Modern hierarchical agglomerative clustering algorithms. arXiv:1109.2378. Retrieved from https:\/\/arxiv.org\/abs\/1109.2378"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/1134285.1134349"},{"key":"e_1_3_3_62_2","volume-title":"Survey on Software Defect Prediction","author":"Nam Jaechang","year":"2014","unstructured":"Jaechang Nam. 2014. Survey on Software Defect Prediction. Technical Report. The Hong Kong University of Science and Technology."},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2017.2720603"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2013.6606584"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/BRACIS.2017.66"},{"issue":"1","key":"e_1_3_3_66_2","first-page":"12","article-title":"Documentation for the data complexity library in c++","volume":"196","author":"Orriols-Puig Albert","year":"2010","unstructured":"Albert Orriols-Puig, N\u00faria Macia, and Tin Kam Ho. 2010. Documentation for the data complexity library in c++. Universitat Ramon Llull, La Salle 196, 1-40 (2010), 12.","journal-title":"Universitat Ramon Llull, La Salle"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-022-06205-9"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_3_3_69_2","first-page":"68","volume-title":"Proceedings of the ICML Workshop on Meta-Learning","author":"Peterson Adam H.","year":"2005","unstructured":"Adam H. Peterson and Tony R. Martinez. 2005. Estimating the potential for combining learning models. In Proceedings of the ICML Workshop on Meta-Learning. 68\u201375."},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2006.365"},{"issue":"6","key":"e_1_3_3_71_2","first-page":"1291","article-title":"Impact of hyper parameter optimization for cross-project software defect prediction","volume":"14","author":"Qu Yubin","year":"2018","unstructured":"Yubin Qu, Xiang Chen, Yingquan Zhao, and Xiaolin Ju. 2018. Impact of hyper parameter optimization for cross-project software defect prediction. International Journal of Performability Engineering 14, 6 (2018), 1291.","journal-title":"International Journal of Performability Engineering"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2013.6606589"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-022-10150-3"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.11"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.43"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00050"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.24"},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2003.1251146"},{"key":"e_1_3_3_79_2","article-title":"Neural networks in data mining","author":"Singh Yashpal","year":"2009","unstructured":"Yashpal Singh and Alok Singh Chauhan. 2009. Neural networks in data mining. Journal of Theoretical and Applied Information Technology 5, 6 (2009), 36\u201342.","journal-title":"Journal of Theoretical and Applied Information Technology"},{"key":"e_1_3_3_80_2","volume-title":"An empirical Study of Instance Hardness","author":"Smith Michael R.","year":"2009","unstructured":"Michael R. Smith. 2009. An empirical Study of Instance Hardness. Master\u2019s thesis. Brigham Young University-Provo."},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1111\/coin.12047"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-013-5422-z"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2018.08.004"},{"key":"e_1_3_3_84_2","doi-asserted-by":"crossref","unstructured":"Charles Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology 15 (1904) 72\u2013101.","DOI":"10.2307\/1412159"},{"key":"e_1_3_3_85_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2019.01.047"},{"key":"e_1_3_3_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2876537"},{"key":"e_1_3_3_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884857"},{"key":"e_1_3_3_88_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2794977"},{"key":"e_1_3_3_89_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9103-7"},{"key":"e_1_3_3_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2018.8489269"},{"key":"e_1_3_3_91_2","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2022.3155183"},{"key":"e_1_3_3_92_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2877612"},{"key":"e_1_3_3_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884804"},{"key":"e_1_3_3_94_2","doi-asserted-by":"publisher","DOI":"10.1109\/CIDM.2009.4938667"},{"key":"e_1_3_3_95_2","doi-asserted-by":"publisher","DOI":"10.1093\/jee\/39.2.269"},{"key":"e_1_3_3_96_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1996.8.7.1341"},{"key":"e_1_3_3_97_2","doi-asserted-by":"publisher","DOI":"10.1145\/2025113.2025120"},{"key":"e_1_3_3_98_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2016.13"},{"key":"e_1_3_3_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/ESEM.2017.48"},{"key":"e_1_3_3_100_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383219.3383232"},{"key":"e_1_3_3_101_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-022-00838-z"},{"key":"e_1_3_3_102_2","unstructured":"Tianyi Zhou Shengjie Wang and Jeffrey Bilmes. 2020. Curriculum Learning by Dynamic Instance Hardness. In Proceedings of the 34th Conference on Neural Information Processing Systems. 8602\u20138613."},{"key":"e_1_3_3_103_2","doi-asserted-by":"publisher","DOI":"10.1201\/b12207"},{"key":"e_1_3_3_104_2","doi-asserted-by":"publisher","DOI":"10.1145\/1368088.1368161"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649596","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3649596","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T23:56:54Z","timestamp":1750291014000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3649596"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,27]]},"references-count":103,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,7,31]]}},"alternative-id":["10.1145\/3649596"],"URL":"https:\/\/doi.org\/10.1145\/3649596","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,27]]},"assertion":[{"value":"2023-05-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-15","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}