{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,20]],"date-time":"2026-05-20T14:23:24Z","timestamp":1779287004391,"version":"3.51.4"},"reference-count":151,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,12,26]],"date-time":"2024-12-26T00:00:00Z","timestamp":1735171200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,1,31]]},"abstract":"<jats:p>\n            <jats:italic>Background<\/jats:italic>\n            : Recently, machine and deep learning (ML\/DL) algorithms have been increasingly adopted in many software systems. Due to their inductive nature, ensuring the quality of these systems remains a significant challenge for the research community. Traditionally, software systems were constructed deductively, by writing explicit rules that govern the behavior of the system as program code. However, ML\/DL systems infer rules from training data i.e., they are generated inductively. Recent research in ML\/DL quality assurance has adapted concepts from traditional software testing, such as mutation testing, to improve reliability. However, it is unclear if these proposed testing techniques are adopted in practice, or if new testing strategies have emerged from real-world ML deployments. There is little empirical evidence about the testing strategies.\n          <\/jats:p>\n          <jats:p>\n            <jats:italic>Aims<\/jats:italic>\n            : To fill this gap, we perform the first fine-grained empirical study on ML testing in the wild to identify the ML properties being tested, the testing strategies, and their implementation throughout the ML workflow.\n          <\/jats:p>\n          <jats:p>\n            <jats:italic>Method<\/jats:italic>\n            : We conducted a mixed-methods study to understand ML software testing practices. We analyzed test files and cases from 11 open-source ML\/DL projects on GitHub. Using open coding, we manually examined the testing strategies, tested ML properties, and implemented testing methods to understand their practical application in building and releasing ML\/DL software systems.\n          <\/jats:p>\n          <jats:p>\n            <jats:italic>Results<\/jats:italic>\n            : Our findings reveal several key insights: (1) The most common testing strategies, accounting for less than 40%, are Grey-box and White-box methods, such as\n            <jats:italic>Negative Testing<\/jats:italic>\n            ,\n            <jats:italic>Oracle Approximation<\/jats:italic>\n            , and\n            <jats:italic>Statistical Testing<\/jats:italic>\n            . (2) A wide range of\n            <jats:inline-formula content-type=\"math\/tex\">\n              <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(17\\)<\/jats:tex-math>\n            <\/jats:inline-formula>\n            ML properties are tested, out of which only 20% to 30% are frequently tested, including\n            <jats:italic>Consistency<\/jats:italic>\n            ,\n            <jats:italic>Correctness<\/jats:italic>\n            , and\n            <jats:italic>Efficiency<\/jats:italic>\n            . (3)\n            <jats:italic>Bias and Fairness<\/jats:italic>\n            is more tested in Recommendation (6%) and Computer Vision (CV) (3.9%) systems, while\n            <jats:italic>Security and Privacy<\/jats:italic>\n            is tested in CV (2%), Application Platforms (0.9%), and NLP (0.5%). (4) We identified 13 types of testing methods, such as\n            <jats:italic>Unit Testing<\/jats:italic>\n            ,\n            <jats:italic>Input Testing<\/jats:italic>\n            , and\n            <jats:italic>Model Testing<\/jats:italic>\n            .\n          <\/jats:p>\n          <jats:p>\n            <jats:italic>Conclusions<\/jats:italic>\n            : This study sheds light on the current adoption of software testing techniques and highlights gaps and limitations in existing ML testing practices.\n          <\/jats:p>","DOI":"10.1145\/3680463","type":"journal-article","created":{"date-parts":[[2024,7,24]],"date-time":"2024-07-24T15:34:17Z","timestamp":1721835257000},"page":"1-63","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["An Empirical Study of Testing Machine Learning in the Wild"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4115-3834","authenticated-orcid":false,"given":"Moses","family":"Openja","sequence":"first","affiliation":[{"name":"Polytechnique Montr\u00e9al, Montreal, Queb\u00e9c, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5704-4173","authenticated-orcid":false,"given":"Foutse","family":"Khomh","sequence":"additional","affiliation":[{"name":"Polytechnique Montr\u00e9al, Montreal, Queb\u00e9c, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6356-1938","authenticated-orcid":false,"given":"Armstrong","family":"Foundjem","sequence":"additional","affiliation":[{"name":"Polytechnique Montr\u00e9al, Montreal, Queb\u00e9c, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3063-3197","authenticated-orcid":false,"given":"Zhen Ming (Jack)","family":"Jiang","sequence":"additional","affiliation":[{"name":"York University, Toronto, Ontario, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9058-8519","authenticated-orcid":false,"given":"Mouna","family":"Abidi","sequence":"additional","affiliation":[{"name":"Polytechnique Montr\u00e9al, Montreal, Queb\u00e9c, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7749-5513","authenticated-orcid":false,"given":"Ahmed E.","family":"Hassan","sequence":"additional","affiliation":[{"name":"Queen's University, Kingston, Ontario, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,12,26]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"2019. Continuous Delivery for Machine Learning. Retrieved from https:\/\/martinfowler.com\/articles\/cd4ml.html."},{"key":"e_1_3_2_3_2","unstructured":"2021. GitHub REST API. Retrieved January 5 2021 from https:\/\/developer.github.com\/v3\/"},{"key":"e_1_3_2_4_2","unstructured":"2021. TOSEM-2021-Replication; \u201cStudying the Practices of Testing Machine Learning Software in the Wild\u201d. Retrieved from https:\/\/github.com\/SWATLab-git\/TOSEM-2021-Replication"},{"key":"e_1_3_2_5_2","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1145\/3338906.3338937","volume-title":"Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Aggarwal Aniya","year":"2019","unstructured":"Aniya Aggarwal, Pranay Lohia, Seema Nagar, Kuntal Dey, and Diptikalyan Saha. 2019. Black box fairness testing of machine learning models. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 625\u2013635."},{"key":"e_1_3_2_6_2","first-page":"291","volume-title":"Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)","author":"Amershi Saleema","year":"2019","unstructured":"Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 291\u2013300. https:\/\/doi.org\/10.1109\/ICSE-SEIP.2019.00042"},{"key":"e_1_3_2_7_2","unstructured":"Maryam Ashoori and Justin D. Weisz. 2019. In AI we trust? Factors that influence trustworthiness of AI-infused decision-making processes. arXiv:1912.02675."},{"key":"e_1_3_2_8_2","unstructured":"auto-sklearn automl. 2021a. Test Automl. Retrieved from https:\/\/github.com\/automl\/auto-sklearn\/blob\/master\/test\/test_automl\/test_automl.py."},{"key":"e_1_3_2_9_2","unstructured":"auto-sklearn automl. 2021b. Test Metrics. Retrieved from https:\/\/github.com\/automl\/auto-sklearn\/blob\/master\/test\/test_metric\/test_metrics.py."},{"issue":"12","key":"e_1_3_2_10_2","doi-asserted-by":"crossref","first-page":"4374","DOI":"10.3390\/s22124374","article-title":"Test case prioritization, selection, and reduction using improved quantum-behaved particle swarm optimization","volume":"22","author":"Bajaj Anu","year":"2022","unstructured":"Anu Bajaj, Ajith Abraham, Saroj Ratnoo, and Lubna Abdelkareim Gabralla. 2022. Test case prioritization, selection, and reduction using improved quantum-behaved particle swarm optimization. Sensors 22, 12 (2022), 4374.","journal-title":"Sensors"},{"key":"e_1_3_2_11_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s11334-021-00384-9","article-title":"Tri-level regression testing using nature-inspired algorithms","volume":"17","author":"Bajaj Anu","year":"2021","unstructured":"Anu Bajaj and Om Prakash Sangwan. 2021. Tri-level regression testing using nature-inspired algorithms. Innovations in Systems and Software Engineering 17 (2021), 1\u201316.","journal-title":"Innovations in Systems and Software Engineering"},{"key":"e_1_3_2_12_2","first-page":"2429","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"33","author":"Bansal Gagan","year":"2019","unstructured":"Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Weld, Walter S Lasecki, and Eric Horvitz. 2019. Updates in human-ai teams: Understanding and addressing the performance\/compatibility tradeoff. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2429\u20132437."},{"issue":"1","key":"e_1_3_2_13_2","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.entcs.2005.12.014","article-title":"An introduction to software testing","volume":"148","author":"Baresi Luciano","year":"2006","unstructured":"Luciano Baresi and Mauro Pezze. 2006. An introduction to software testing. Electronic Notes in Theoretical Computer Science 148, 1 (2006), 89\u2013111.","journal-title":"Electronic Notes in Theoretical Computer Science"},{"key":"e_1_3_2_14_2","volume-title":"Fairness and Machine Learning: Limitations and Opportunities","author":"Barocas Solon","year":"2023","unstructured":"Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2023. Fairness and Machine Learning: Limitations and Opportunities. MIT Press."},{"issue":"3","key":"e_1_3_2_15_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1541880.1541883","article-title":"Methodologies for data quality assessment and improvement","volume":"41","author":"Batini Carlo","year":"2009","unstructured":"Carlo Batini, Cinzia Cappiello, Chiara Francalanci, and Andrea Maurino. 2009. Methodologies for data quality assessment and improvement. ACM computing surveys (CSUR) 41, 3 (2009), 1\u201352.","journal-title":"ACM computing surveys (CSUR)"},{"key":"e_1_3_2_16_2","unstructured":"Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine translation. arXiv:1711.02173."},{"issue":"1","key":"e_1_3_2_17_2","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1109\/MS.1984.233702","article-title":"Verifying and validating software requirements and design specifications","volume":"1","author":"Boehm Barry W.","year":"1984","unstructured":"Barry W. Boehm. 1984. Verifying and validating software requirements and design specifications. IEEE Software 1, 1 (1984), 75.","journal-title":"IEEE Software"},{"key":"e_1_3_2_18_2","volume-title":"Information Quality: A Conceptual Framework and Empirical Validation","author":"Bovee Matthew Wamsley","year":"2004","unstructured":"Matthew Wamsley Bovee. 2004. Information Quality: A Conceptual Framework and Empirical Validation. Ph.D. Dissertation. University of Kansas."},{"key":"e_1_3_2_19_2","first-page":"426","volume-title":"Proceedings of the 2019 IEEE 19th International Conference on Software Quality, Reliability and Security (QRS).","author":"Braiek Houssem Ben","year":"2019","unstructured":"Houssem Ben Braiek and Foutse Khomh. 2019. TFCheck: A TensorFlow library for detecting training issues in neural network programs. In Proceedings of the 2019 IEEE 19th International Conference on Software Quality, Reliability and Security (QRS). IEEE, 426\u2013433."},{"key":"e_1_3_2_20_2","article-title":"On testing machine learning programs","volume":"164","author":"Braiek Houssem Ben","year":"2020","unstructured":"Houssem Ben Braiek and Foutse Khomh. 2020. On testing machine learning programs. Journal of Systems and Software 164 (2020), Article 110542. DOI: https:\/\/doi.org\/10.1016\/j.jss.2020.110542","journal-title":"Journal of Systems and Software"},{"key":"e_1_3_2_21_2","first-page":"1123","volume-title":"Proceedings of the 2017 IEEE International Conference on Big Data (Big Data).","author":"Breck Eric","year":"2017","unstructured":"Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D Sculley. 2017. The ML test score: A rubric for ML production readiness and technical debt reduction. In Proceedings of the 2017 IEEE International Conference on Big Data (Big Data). IEEE, 1123\u20131132."},{"key":"e_1_3_2_22_2","first-page":"287","volume-title":"Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)","author":"Businge J.","year":"2019","unstructured":"J. Businge, M. Openja, D. Kavaler, E. Bainomugisha, F. Khomh, and V. Filkov. 2019. Studying Android App Popularity by Cross-Linking GitHub and Google Play Store. In Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), 287\u2013297. DOI: 10.1109\/SANER.2019.8667998"},{"key":"e_1_3_2_23_2","first-page":"625","volume-title":"Proceedings of the 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)","author":"Businge J.","year":"2018","unstructured":"J. Businge, M. Openja, S. Nadi, E. Bainomugisha, and T. Berger. 2018. Clone-based variability management in the android ecosystem. In Proceedings of the 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). 625\u2013634. DOI: 10.1109\/ICSME.2018.00072"},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1007\/978-3-540-73589-2_8","volume-title":"Proceedings of the ECOOP 2007\u2013Object-Oriented Programming: 21st European Conference","author":"Cabral Bruno","year":"2007","unstructured":"Bruno Cabral and Paulo Marques. 2007. Exception handling: A field study in Java and. net. In Proceedings of the ECOOP 2007\u2013Object-Oriented Programming: 21st European Conference. Springer, 151\u2013175."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.17016\/FEDS.2005.21"},{"key":"e_1_3_2_26_2","first-page":"2267","volume-title":"Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security","author":"Cao Yulong","year":"2019","unstructured":"Yulong Cao, Chaowei Xiao, Benjamin Cyr, Yimeng Zhou, Won Park, Sara Rampazzi, Qi Alfred Chen, Kevin Fu, and Z Morley Mao. 2019. Adversarial sensor attack on lidar-based perception in autonomous driving. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, 2267\u20132281."},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1109\/SP.2017.49","volume-title":"Proceedings of the 2017 IEEE Symposium on Security and Privacy (SP)","author":"Carlini Nicholas","year":"2017","unstructured":"Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In Proceedings of the 2017 IEEE Symposium on Security and Privacy (SP). IEEE, 39\u201357."},{"key":"e_1_3_2_28_2","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1007\/978-3-642-38314-4_7","volume-title":"Agile Processes in Software Engineering and Extreme Programming","author":"Causevic Adnan","year":"2013","unstructured":"Adnan Causevic, Rakesh Shukla, Sasikumar Punnekkat, and Daniel Sundmark. 2013. Effects of negative testing on TDD: An industrial experiment. In Agile Processes in Software Engineering and Extreme Programming, Hubert Baumeister and Barbara Weber (Eds.). Springer, Berlin, 91\u2013105."},{"key":"e_1_3_2_29_2","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1145\/3468264.3468537","volume-title":"Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Chakraborty Joymallya","year":"2021","unstructured":"Joymallya Chakraborty, Suvodeep Majumder, and Tim Menzies. 2021. Bias in machine learning software: Why? how? what to do? In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 429\u2013440."},{"issue":"3","key":"e_1_3_2_30_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1541880.1541882","article-title":"Anomaly detection: A survey","volume":"41","author":"Chandola Varun","year":"2009","unstructured":"Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly detection: A survey. ACM Computing Surveys (CSUR) 41, 3 (2009), 1\u201358.","journal-title":"ACM Computing Surveys (CSUR)"},{"issue":"1","key":"e_1_3_2_31_2","first-page":"2250","article-title":"Smoke testing","volume":"4","author":"Chauhan Vinod Kumar","year":"2014","unstructured":"Vinod Kumar Chauhan. 2014. Smoke testing. International Journal of Scientific and Research Publication 4, 1 (2014), 2250\u20133153.","journal-title":"International Journal of Scientific and Research Publication"},{"key":"e_1_3_2_32_2","first-page":"2722","volume-title":"Proceedings of the IEEE International Conference on CV","author":"Chen Chenyi","year":"2015","unstructured":"Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. 2015. Deepdriving: Learning affordance for direct perception in autonomous driving. In Proceedings of the IEEE International Conference on CV, 2722\u20132730."},{"key":"e_1_3_2_33_2","unstructured":"Xinyun Chen Chang Liu Bo Li Kimberly Lu and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv:1712.05526. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.1712.05526"},{"key":"e_1_3_2_34_2","first-page":"156","volume-title":"Proceedings of the 2019 IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS)","author":"Chen Xuexin","year":"2019","unstructured":"Xuexin Chen, Pu Miao, and Qingkai Bu. 2019. Image segmentation algorithm based on particle swarm optimization with k-means optimization. In Proceedings of the 2019 IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS). IEEE, 156\u2013159."},{"key":"e_1_3_2_35_2","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1109\/QRS.2018.00044","volume-title":"Proceedings of the 2018 IEEE International Conference on Software Quality, Reliability and Security (QRS)","author":"Cheng Dawei","year":"2018","unstructured":"Dawei Cheng, Chun Cao, Chang Xu, and Xiaoxing Ma. 2018. Manifesting bugs in machine learning code: An explorative study with mutation testing. In Proceedings of the 2018 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 313\u2013324."},{"issue":"5","key":"e_1_3_2_36_2","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1049\/sej.1994.0025","article-title":"Applicability of modified condition\/decision coverage to software testing","volume":"9","author":"Chilenski John Joseph","year":"1994","unstructured":"John Joseph Chilenski and Steven P. Miller. 1994. Applicability of modified condition\/decision coverage to software testing. Software Engineering Journal 9, 5 (1994), 193\u2013200.","journal-title":"Software Engineering Journal"},{"key":"e_1_3_2_37_2","unstructured":"Voskoglou Christina. 2017. What is the best programming language for machine learning? Retrieved from https:\/\/towardsdatascience.com\/what-is-the-best-programming-language-for-machine-learning-a745c156d6b7"},{"issue":"1","key":"e_1_3_2_38_2","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1093\/jjfinec\/nbh004","article-title":"Backtesting value-at-risk: A duration-based approach","volume":"2","author":"Christoffersen Peter","year":"2004","unstructured":"Peter Christoffersen and Denis Pelletier. 2004. Backtesting value-at-risk: A duration-based approach. Journal of Financial Econometrics 2, 1 (2004), 84\u2013108.","journal-title":"Journal of Financial Econometrics"},{"key":"e_1_3_2_39_2","first-page":"1","article-title":"Pearson correlation coefficient","author":"Cohen Israel","year":"2009","unstructured":"Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. In Noise Reduction in Speech Processing, 1\u20134.","journal-title":"Noise Reduction in Speech Processing"},{"key":"e_1_3_2_40_2","volume-title":"Practical Machine Learning with H2O: Powerful, Scalable Techniques for Deep Learning and AI","author":"Cook Darren","year":"2016","unstructured":"Darren Cook. 2016. Practical Machine Learning with H2O: Powerful, Scalable Techniques for Deep Learning and AI. O\u2019Reilly Media, Inc."},{"key":"e_1_3_2_41_2","unstructured":"cpputest. 2021. CppUTest: CppUTest unit testing and mocking framework for C\/C++. Retrieved from https:\/\/cpputest.github.io\/"},{"issue":"2","key":"e_1_3_2_42_2","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1089\/big.2016.0048","article-title":"Conscientious classification: A data scientist's guide to discrimination-aware classification","volume":"5","author":"d\u2019Alessandro Brian","year":"2017","unstructured":"Brian d\u2019Alessandro, Cathy O\u2019Neil, and Tom LaGatta. 2017. Conscientious classification: A data scientist's guide to discrimination-aware classification. Big Data 5, 2 (2017), 120\u2013134.","journal-title":"Big Data"},{"key":"e_1_3_2_43_2","unstructured":"Alexander D\u2019Amour Katherine Heller Dan Moldovan Ben Adlam Babak Alipanahi Alex Beutel Christina Chen Jonathan Deaton Jacob Eisenstein Matthew D Hoffman et al. 2020. Underspecification presents challenges for credibility in modern machine learning. arXiv:2011.03395."},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aal1950"},{"key":"e_1_3_2_45_2","unstructured":"Rosie Dunford Quanrong Su and Ekraj Tamang. 2014. The pareto principle. DOI: https:\/\/api.semanticscholar.org\/CorpusID:15925174"},{"key":"e_1_3_2_46_2","unstructured":"Anna Fariha Ashish Tiwari Arjun Radhakrishna Sumit Gulwani and Alexandra Meliou. 2020. Data invariants: On trust in data-driven systems. arXiv:2003.01289."},{"key":"e_1_3_2_47_2","doi-asserted-by":"crossref","first-page":"113233","DOI":"10.1016\/j.eswa.2020.113233","article-title":"A multimodal particle swarm optimization-based approach for image segmentation","volume":"149","author":"Farshi Taymaz Rahkar","year":"2020","unstructured":"Taymaz Rahkar Farshi, John H Drake, and Ender \u00d6zcan. 2020. A multimodal particle swarm optimization-based approach for image segmentation. Expert Systems with Applications 149 (2020), 113233.","journal-title":"Expert Systems with Applications"},{"key":"e_1_3_2_48_2","article-title":"Data set quality in machine learning: Consistency measure based on group decision making","volume":"106","author":"Fenza Giuseppe","year":"2021","unstructured":"Giuseppe Fenza, Mariacristina Gallo, Vincenzo Loia, Francesco Orciuoli, and Enrique Herrera-Viedma. 2021. Data set quality in machine learning: Consistency measure based on group decision making. Applied Soft Computing 106 (2021), 107366. DOI: https:\/\/doi.org\/10.1016\/j.asoc.2021.107366","journal-title":"Applied Soft Computing"},{"key":"e_1_3_2_49_2","article-title":"The EU Artificial Intelligence Act and Access to Justice","author":"Fink Melanie","year":"2021","unstructured":"Melanie Fink. 2021. The EU Artificial Intelligence Act and Access to Justice. EU Law Live (2021).","journal-title":"EU Law Live"},{"issue":"10","key":"e_1_3_2_50_2","doi-asserted-by":"crossref","first-page":"4731","DOI":"10.1109\/TSE.2023.3313875","article-title":"A Grounded Theory of Cross-Community SECOs: Feedback Diversity Versus Synchronization","volume":"49","author":"Foundjem A.","year":"2023","unstructured":"A. Foundjem, E. E. Eghan, and B. Adams. 2023. A Grounded Theory of Cross-Community SECOs: Feedback Diversity Versus Synchronization. IEEE Transactions on Software Engineering 49, 10 (Oct. 2023), 4731\u20134750. DOI: https:\/\/doi.org\/10.1109\/TSE.2023.3313875","journal-title":"IEEE Transactions on Software Engineering"},{"issue":"1","key":"e_1_3_2_51_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2594473.2594475","article-title":"Comprehensible classification models: a position paper","volume":"15","author":"Freitas Alex A","year":"2014","unstructured":"Alex A Freitas. 2014. Comprehensible classification models: a position paper. ACM SIGKDD Explorations Newsletter 15, 1 (2014), 1\u201310.","journal-title":"ACM SIGKDD Explorations Newsletter"},{"key":"e_1_3_2_52_2","doi-asserted-by":"crossref","first-page":"498","DOI":"10.1145\/3106237.3106277","volume-title":"Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering","author":"Galhotra Sainyam","year":"2017","unstructured":"Sainyam Galhotra, Yuriy Brun, and Alexandra Meliou. 2017. Fairness testing: testing software for discrimination. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, 498\u2013510."},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2020.3016143"},{"key":"e_1_3_2_54_2","unstructured":"Rozental Gennadiy and Enficiaud Raffi. 2020. Boost C++ Libraries. Retrieved from https:\/\/www.boost.org\/"},{"key":"e_1_3_2_55_2","doi-asserted-by":"crossref","first-page":"272","DOI":"10.1016\/j.ins.2016.01.090","article-title":"BNC-PSO: Structure learning of Bayesian networks by Particle Swarm Optimization","volume":"348","author":"Gheisari S.","year":"2016","unstructured":"S. Gheisari and M.R. Meybodi. 2016. BNC-PSO: Structure learning of Bayesian networks by Particle Swarm Optimization. Information Sciences 348 (2016), 272\u2013289. DOI: https:\/\/doi.org\/10.1016\/j.ins.2016.01.090","journal-title":"Information Sciences"},{"key":"e_1_3_2_56_2","unstructured":"Inc GitHub. 2021. The GitHub Search API Lets You to Search for the Specific Item Efficiently. Retrieved from https:\/\/docs.github.com\/en\/rest\/reference\/search"},{"key":"e_1_3_2_57_2","first-page":"55","volume-title":"Proceedings of the 2nd World Congress on Software Quality","volume":"9","author":"Glinz Martin","year":"2000","unstructured":"Martin Glinz. 2000. Improving the quality of requirements with scenarios. In Proceedings of the 2nd World Congress on Software Quality, Vol. 9, 55\u201360."},{"key":"e_1_3_2_58_2","unstructured":"Ian J. Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv:1412.6572."},{"key":"e_1_3_2_59_2","unstructured":"Cloud Google. 2021. MLOps: Continuous Delivery and Automation Pipelines in Machine Learning. Retrieved from https:\/\/cloud.google.com\/architecture\/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning"},{"key":"e_1_3_2_60_2","unstructured":"Googletest. 2021. Googletest Primer. Retrieved from http:\/\/google.github.io\/googletest\/primer.html"},{"key":"e_1_3_2_61_2","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1016\/j.knosys.2018.03.022","article-title":"Graph embedding techniques, applications, and performance: A survey","volume":"151","author":"Goyal Palash","year":"2018","unstructured":"Palash Goyal and Emilio Ferrara. 2018. Graph embedding techniques, applications, and performance: A survey. Knowledge-Based Systems 151 (2018), 78\u201394. DOI: https:\/\/doi.org\/10.1016\/j.knosys.2018.03.022","journal-title":"Knowledge-Based Systems"},{"issue":"2","key":"e_1_3_2_62_2","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1145\/367008.367020","article-title":"An empirical study of regression test selection techniques","volume":"10","author":"Graves Todd L.","year":"2001","unstructured":"Todd L. Graves, Mary Jean Harrold, Jung-Min Kim, Adam Porter, and Gregg Rothermel. 2001. An empirical study of regression test selection techniques. ACM Transactions on Software Engineering and Methodology (TOSEM) 10, 2 (2001), 184\u2013208.","journal-title":"ACM Transactions on Software Engineering and Methodology (TOSEM)"},{"key":"e_1_3_2_63_2","first-page":"78","volume-title":"Proceedings of the 2012 International Symposium on Software Testing and Analysis (ISSTA \u201912)","author":"Groce Alex","year":"2012","unstructured":"Alex Groce, Chaoqiang Zhang, Eric Eide, Yang Chen, and John Regehr. 2012. Swarm Testing. In Proceedings of the 2012 International Symposium on Software Testing and Analysis (ISSTA \u201912). ACM, New York, NY, 78\u201388. DOI: 10.1145\/2338965.2336763"},{"issue":"5","key":"e_1_3_2_64_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3236009","article-title":"A survey of methods for explaining black box models","volume":"51","author":"Guidotti Riccardo","year":"2018","unstructured":"Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM Computing Surveys (CSUR) 51, 5 (2018), 1\u201342.","journal-title":"ACM Computing Surveys (CSUR)"},{"issue":"1","key":"e_1_3_2_65_2","first-page":"3790","article-title":"Monotonic calibrated interpolated look-up tables","volume":"17","author":"Gupta Maya","year":"2016","unstructured":"Maya Gupta, Andrew Cotter, Jan Pfeifer, Konstantin Voevodski, Kevin Canini, Alexander Mangylov, Wojciech Moczydlowski, and Alexander Van Esbroeck. 2016. Monotonic calibrated interpolated look-up tables. The Journal of Machine Learning Research 17, 1 (2016), 3790\u20133836.","journal-title":"The Journal of Machine Learning Research"},{"key":"e_1_3_2_66_2","unstructured":"Naveen Gv. 2019. Memory Errors in C++. Retrieved from https:\/\/www.cprogramming.com\/tutorial\/memory_debugging_parallel_inspector.html"},{"key":"e_1_3_2_67_2","unstructured":"Patrick Hall Navdeep Gill and Nicholas Schmidt. 2019. Proposed guidelines for the responsible use of explainable machine learning. arXiv:1906.03533."},{"issue":"3","key":"e_1_3_2_68_2","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1109\/TSE.1977.231133","article-title":"Compiler Analysis of the Value Ranges for Variables","volume":"3","author":"Harrison W. H.","year":"1977","unstructured":"W. H. Harrison. 1977. Compiler Analysis of the Value Ranges for Variables. IEEE Transactions on Software Engineering 3, 3 (May 1977), 243\u2013250. DOI: https:\/\/doi.org\/10.1109\/TSE.1977.231133","journal-title":"IEEE Transactions on Software Engineering"},{"issue":"1","key":"e_1_3_2_69_2","doi-asserted-by":"crossref","first-page":"13","DOI":"10.3905\/jpm.2015.42.1.013","article-title":"Backtesting","volume":"42","author":"Harvey Campbell R","year":"2015","unstructured":"Campbell R Harvey and Yan Liu. 2015. Backtesting. The Journal of Portfolio Management 42, 1 (2015), 13\u201328.","journal-title":"The Journal of Portfolio Management"},{"key":"e_1_3_2_70_2","first-page":"187","volume-title":"Proceedings of the Sixth Workshop on Statistical Machine Translation","author":"Heafield Kenneth","year":"2011","unstructured":"Kenneth Heafield. 2011. KenLM: Faster and Smaller Language Model Queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation. Association for Computational Linguistics, 187\u2013197. Retrieved from https:\/\/www.aclweb.org\/anthology\/W11-2123"},{"key":"e_1_3_2_71_2","first-page":"1018","volume-title":"Inductive Bias","author":"H\u00fcllermeier Eyke","year":"2013","unstructured":"Eyke H\u00fcllermeier, Thomas Fober, and Marco Mernberger. 2013. Inductive Bias. Springer, New York, NY, 1018\u20131018. DOI: https:\/\/doi.org\/10.1007\/978-1-4419-9863-7_927"},{"issue":"1","key":"e_1_3_2_72_2","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1016\/j.dss.2010.12.003","article-title":"An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models","volume":"51","author":"Huysmans Johan","year":"2011","unstructured":"Johan Huysmans, Karel Dejaeger, Christophe Mues, Jan Vanthienen, and Bart Baesens. 2011. An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models. Decision Support Systems 51, 1 (2011), 141\u2013154.","journal-title":"Decision Support Systems"},{"key":"e_1_3_2_73_2","unstructured":"IBM. 2020. The Machine Learning Development and Operations. Retrieved from https:\/\/ibm-cloud-architecture.github.io\/refarch-data-ai-analytics\/methodology\/MLops\/"},{"key":"e_1_3_2_74_2","unstructured":"ICS-33. 2021. Complexity of Python Operations. Retrieved from https:\/\/www.ics.uci.edu\/pattis\/ICS-33\/lectures\/complexitypython.txt"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.372146"},{"key":"e_1_3_2_76_2","first-page":"1341","volume-title":"Statistical Tests","author":"Ille Tatjana","year":"2008","unstructured":"Tatjana Ille and Natasa Milic. 2008. Statistical Tests. Springer Netherlands, Dordrecht, 1341\u20131344. DOI: https:\/\/doi.org\/10.1007\/978-1-4020-5614-7_3349"},{"key":"e_1_3_2_77_2","first-page":"788","volume-title":"Proceedings of the 2018 3rd International Conference on Inventive Computation Technologies (ICICT)","year":"2018","unstructured":"Isha, Abhinav Sharma, and M. Revathi. 2018. Automated API testing. In Proceedings of the 2018 3rd International Conference on Inventive Computation Technologies (ICICT), 788\u2013791. DOI: 10.1109\/ICICT43934.2018.9034254"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1177\/1525822X05282260"},{"key":"e_1_3_2_79_2","doi-asserted-by":"crossref","unstructured":"Mohit Iyyer John Wieting Kevin Gimpel and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. arXiv:1804.06059.","DOI":"10.18653\/v1\/N18-1170"},{"key":"e_1_3_2_80_2","doi-asserted-by":"crossref","DOI":"10.1201\/9781439889503","volume-title":"Software Testing: A Craftsman'S Approach","author":"Jorgensen Paul C","year":"2013","unstructured":"Paul C Jorgensen. 2013. Software Testing: A Craftsman'S Approach. Auerbach Publications."},{"key":"e_1_3_2_81_2","first-page":"1885","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Koh Pang Wei","year":"2017","unstructured":"Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proceedings of the International Conference on Machine Learning. PMLR, 1885\u20131894."},{"key":"e_1_3_2_82_2","first-page":"14","volume-title":"Proceedings of the 2nd Workshop on New Trends on Content-Based Recommender Systems co-located with 9th ACM Conference on Recommender Systems (RecSys \u201915)","volume":"1448","author":"Kula Maciej","year":"2015","unstructured":"Maciej Kula. 2015. Metadata Embeddings for User and Item Cold-start Recommendations. In Proceedings of the 2nd Workshop on New Trends on Content-Based Recommender Systems co-located with 9th ACM Conference on Recommender Systems (RecSys \u201915). (CEUR Workshop Proceedings, Vol. 1448), Toine Bogers and Marijn Koolen (Eds.). CEUR-WS.org, 14\u201321. Retrieved from http:\/\/ceur-ws.org\/Vol-1448\/paper4.pdf"},{"key":"e_1_3_2_83_2","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1109\/VLHCC.2005.44","volume-title":"Proceedings of the 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC\u201905)","author":"Lawrence J","year":"2005","unstructured":"J Lawrence, Steven Clarke, Margaret Burnett, and Gregg Rothermel. 2005. How well do professional developers test with code coverage visualizations? An empirical study. In Proceedings of the 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC\u201905). IEEE, 53\u201360."},{"key":"e_1_3_2_84_2","first-page":"21002","article-title":"Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection","volume":"33","author":"Li Xiang","year":"2020","unstructured":"Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. 2020. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. Advances in Neural Information Processing Systems 33 (2020), 21002\u201321012.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_85_2","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","article-title":"A survey on deep learning in medical image analysis","volume":"42","author":"Litjens Geert","year":"2017","unstructured":"Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I S\u00e1nchez. 2017. A survey on deep learning in medical image analysis. Medical Image Analysis 42 (2017), 60\u201388.","journal-title":"Medical Image Analysis"},{"key":"e_1_3_2_86_2","first-page":"292","volume-title":"Proceedings of the ICIQ","author":"Liu Liping","year":"2002","unstructured":"Liping Liu and Lauren Chi. 2002. Evolutional data quality: A theory-specific view. In Proceedings of the ICIQ. 292\u2013304."},{"key":"e_1_3_2_87_2","first-page":"2736","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Liu Zhuang","year":"2017","unstructured":"Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, 2736\u20132744."},{"key":"e_1_3_2_88_2","first-page":"623","volume-title":"Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Lou Yin","year":"2013","unstructured":"Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. 2013. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 623\u2013631."},{"key":"e_1_3_2_89_2","first-page":"100","volume-title":"Proceedings of the 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE)","author":"Ma Lei","year":"2018","unstructured":"Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, et al. 2018. Deepmutation: Mutation testing of deep learning systems. In Proceedings of the 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 100\u2013111."},{"key":"e_1_3_2_90_2","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1109\/ICSME55016.2022.00014","volume-title":"Proceedings of the 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME)","author":"Majidi Forough","year":"2022","unstructured":"Forough Majidi, Moses Openja, Foutse Khomh, and Heng Li. 2022. An Empirical Study on the Usage of Automated Machine Learning Tools. In Proceedings of the 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). 59\u201370. DOI: 10.1109\/ICSME55016.2022.00014"},{"key":"e_1_3_2_91_2","unstructured":"VanderVoord Mark Karlesky Mike and Williams Greg. 2015. Unity Unit Testing for C (Especially Embedded Software). Retrieved from http:\/\/www.throwtheswitch.org\/unity"},{"issue":"3","key":"e_1_3_2_92_2","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1079\/PHN2002429","article-title":"Statistical approaches for assessing the relative validity of a food-frequency questionnaire: use of correlation coefficients and the kappa statistic","volume":"6","author":"Masson Lindsey Fiona","year":"2003","unstructured":"Lindsey Fiona Masson, Geraldine McNeill, J. O. Tomany, J. A. Simpson, Heather Sinclair Peace, L. Wei, D. A. Grubb, and C. Bolton-Smith. 2003. Statistical approaches for assessing the relative validity of a food-frequency questionnaire: use of correlation coefficients and the kappa statistic. Public Health Nutrition 6, 3 (2003), 313\u2013321.","journal-title":"Public Health Nutrition"},{"issue":"6","key":"e_1_3_2_93_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3457607","article-title":"A survey on bias and fairness in machine learning","volume":"54","author":"Mehrabi Ninareh","year":"2021","unstructured":"Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54, 6 (2021), 1\u201335.","journal-title":"ACM Computing Surveys (CSUR)"},{"key":"e_1_3_2_94_2","unstructured":"Cohn Mike. 2009. The Forgotten Layer of the Test Automation Pyramid. Retrieved from https:\/\/www.mountaingoatsoftware.com\/blog\/the-forgotten-layer-of-the-test-automation-pyramid"},{"key":"e_1_3_2_95_2","first-page":"11264","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Molchanov Pavlo","year":"2019","unstructured":"Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. 2019. Importance estimation for neural network pruning. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, 11264\u201311272."},{"key":"e_1_3_2_96_2","volume-title":"Interval Analysis","author":"Moore Ramon E","year":"1966","unstructured":"Ramon E Moore. 1966. Interval Analysis. Vol. 4. Prentice-Hall Englewood Cliffs."},{"issue":"6","key":"e_1_3_2_97_2","doi-asserted-by":"crossref","first-page":"3219","DOI":"10.1007\/s10664-017-9512-6","article-title":"Curating github for engineered software projects","volume":"22","author":"Munaiah Nuthan","year":"2017","unstructured":"Nuthan Munaiah, Steven Kroh, Craig Cabrey, and Meiyappan Nagappan. 2017. Curating github for engineered software projects. Empirical Software Engineering 22, 6 (2017), 3219\u20133253.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_2_98_2","unstructured":"W James Murdoch Chandan Singh Karl Kumbier Reza Abbasi-Asl and Bin Yu. 2019. Interpretable machine learning: definitions methods and applications. arXiv:1901.04592."},{"key":"e_1_3_2_99_2","first-page":"785","volume-title":"Proceedings of the 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Nejadgholi Mahdi","year":"2019","unstructured":"Mahdi Nejadgholi and Jinqiu Yang. 2019. A study of oracle approximations in testing deep learning libraries. In Proceedings of the 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 785\u2013796."},{"key":"e_1_3_2_100_2","unstructured":"NumPy. 2021. NumPy: The fundamental package for scientific computing with Python. Retrieved from https:\/\/numpy.org\/"},{"key":"e_1_3_2_101_2","doi-asserted-by":"crossref","unstructured":"Moses Openja Gabriel Laberge and Foutse Khomh. 2023. Detection and evaluation of bias-inducing features in machine learning. arXiv:2310.12805.","DOI":"10.1007\/s10664-023-10409-5"},{"key":"e_1_3_2_102_2","first-page":"190","volume-title":"Proceedings of the International Conference on Evaluation and Assessment in Software Engineering 2022 (EASE \u201922)","author":"Openja Moses","year":"2022","unstructured":"Moses Openja, Forough Majidi, Foutse Khomh, Bhagya Chembakottu, and Heng Li. 2022a. Studying the Practices of Deploying Machine Learning Projects on Docker. In Proceedings of the International Conference on Evaluation and Assessment in Software Engineering 2022 (EASE \u201922). ACM, New York, NY, 190\u2013200. DOI: 10.1145\/3530019.3530039"},{"key":"e_1_3_2_103_2","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1109\/ICSME55016.2022.00010","volume-title":"Proceedings of the 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME)","author":"Openja Moses","year":"2022","unstructured":"Moses Openja, Amin Nikanjam, Ahmed Haj Yahmed, Foutse Khomh, and Zhen Ming Jack Jiang. 2022b. An empirical study of challenges in converting deep learning models. In Proceedings of the 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 13\u201323."},{"key":"e_1_3_2_104_2","first-page":"506","volume-title":"Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security (ASIA CCS \u201917)","author":"Papernot Nicolas","year":"2017","unstructured":"Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. 2017. Practical Black-Box Attacks against Machine Learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security (ASIA CCS \u201917). ACM, New York, NY, 506\u2013519. DOI: 10.1145\/3052973.3053009"},{"key":"e_1_3_2_105_2","first-page":"1","volume-title":"Proceedings of the 26th Symposium on Operating Systems Principles","author":"Pei Kexin","year":"2017","unstructured":"Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana. 2017. Deepxplore: Automated whitebox testing of deep learning systems. In Proceedings of the 26th Symposium on Operating Systems Principles, 1\u201318."},{"issue":"3","key":"e_1_3_2_106_2","first-page":"1","article-title":"Search-based fairness testing for regression-based machine learning systems","volume":"27","author":"Perera Anjana","year":"2022","unstructured":"Anjana Perera, Aldeida Aleti, Chakkrit Tantithamthavorn, Jirayus Jiarpakdee, Burak Turhan, Lisa Kuhn, and Katie Walker. 2022. Search-based fairness testing for regression-based machine learning systems. Empirical Software Engineering 27, 3 (2022), 1\u201336.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_2_107_2","first-page":"193","volume-title":"Proceedings of the 29th Annual International Conference on Computer Science and Software Engineering (CASCON \u201919)","author":"Peruma Anthony","year":"2019","unstructured":"Anthony Peruma, Khalid Almalki, Christian D. Newman, Mohamed Wiem Mkaouer, Ali Ouni, and Fabio Palomba. 2019. On the distribution of test smells in open source android applications: An exploratory study. In Proceedings of the 29th Annual International Conference on Computer Science and Software Engineering (CASCON \u201919). IBM Corp., 193\u2013202."},{"issue":"4","key":"e_1_3_2_108_2","first-page":"481","article-title":"The permutation testing approach: A review","volume":"70","author":"Pesarin Fortunato","year":"2010","unstructured":"Fortunato Pesarin and Luigi Salmaso. 2010. The permutation testing approach: A review. Statistica 70, 4 (2010), 481\u2013509.","journal-title":"Statistica"},{"key":"e_1_3_2_109_2","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1007\/978-1-4471-0721-7_2","article-title":"Genetic Algorithms","author":"Pham D. T.","year":"2000","unstructured":"D. T. Pham, Dervis Karaboga, D. T. Pham, and D. Karaboga. 2000. Genetic Algorithms. In Intelligent Optimisation Techniques: Genetic Algorithms, Tabu Search, Simulated Annealing and Neural Networks, 51\u2013147.","journal-title":"Intelligent Optimisation Techniques: Genetic Algorithms, Tabu Search, Simulated Annealing and Neural Networks"},{"key":"e_1_3_2_110_2","doi-asserted-by":"crossref","unstructured":"Vinodkumar Prabhakaran Ben Hutchinson and Margaret Mitchell. 2019. Perturbation sensitivity analysis to detect unintended model biases. arXiv:1910.04210.","DOI":"10.18653\/v1\/D19-1578"},{"key":"e_1_3_2_111_2","unstructured":"Pytest. 2021. Pytest About fixtures. Retrieved from https:\/\/docs.pytest.org\/en\/latest\/explanation\/fixtures.html"},{"issue":"1","key":"e_1_3_2_112_2","doi-asserted-by":"crossref","first-page":"e0262838","DOI":"10.1371\/journal.pone.0262838","article-title":"Deep learning model calibration for improving performance in class-imbalanced medical image classification tasks","volume":"17","author":"Rajaraman Sivaramakrishnan","year":"2022","unstructured":"Sivaramakrishnan Rajaraman, Prasanth Ganesan, and Sameer Antani. 2022. Deep learning model calibration for improving performance in class-imbalanced medical image classification tasks. PloS One 17, 1 (2022), e0262838.","journal-title":"PloS One"},{"key":"e_1_3_2_113_2","unstructured":"Bharath Ramsundar Peter Eastman Patrick Walters Vijay Pande Karl Leswing and Zhenqin Wu. 2019. Deep Learning for the Life Sciences. O\u2019Reilly Media. Retrieved from https:\/\/www.amazon.com\/Deep-Learning-Life-Sciences-Microscopy\/dp\/1492039837"},{"key":"e_1_3_2_114_2","first-page":"530","volume-title":"Proceedings of the 2017 International Conference on Computing Methodologies and Communication (ICCMC)","author":"Reddy T. Namratha","year":"2017","unstructured":"T. Namratha Reddy and K. P. Supreethi. 2017. Optimization of K-means algorithm: Ant colony optimization. In Proceedings of the 2017 International Conference on Computing Methodologies and Communication (ICCMC). IEEE, 530\u2013535."},{"key":"e_1_3_2_115_2","doi-asserted-by":"crossref","first-page":"1135","DOI":"10.1145\/2939672.2939778","volume-title":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Ribeiro Marco Tulio","year":"2016","unstructured":"Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. \u201cWhy should i trust you?\u201d Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135\u20131144."},{"key":"e_1_3_2_116_2","doi-asserted-by":"crossref","first-page":"856","DOI":"10.18653\/v1\/P18-1079","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Ribeiro Marco Tulio","year":"2018","unstructured":"Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 856\u2013865."},{"key":"e_1_3_2_117_2","doi-asserted-by":"crossref","first-page":"5193","DOI":"10.1007\/s10664-020-09881-0","article-title":"Testing machine learning based systems: A systematic mapping","volume":"25","author":"Riccio Vincenzo","year":"2020","unstructured":"Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, and Paolo Tonella. 2020. Testing machine learning based systems: A systematic mapping. Empirical Software Engineering 25 (2020), 5193\u20135254.","journal-title":"Empirical Software Engineering"},{"issue":"1","key":"e_1_3_2_118_2","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1109\/TII.2012.2198665","article-title":"Comparison of parallel genetic algorithm and particle swarm optimization for real-time UAV path planning","volume":"9","author":"Roberge Vincent","year":"2012","unstructured":"Vincent Roberge, Mohammed Tarbouchi, and Gilles Labont\u00e9. 2012. Comparison of parallel genetic algorithm and particle swarm optimization for real-time UAV path planning. IEEE Transactions on Industrial Informatics 9, 1 (2012), 132\u2013141.","journal-title":"IEEE Transactions on Industrial Informatics"},{"key":"e_1_3_2_119_2","volume-title":"Mastering the Requirements Process: Getting Requirements Right","author":"Robertson Suzanne","year":"2012","unstructured":"Suzanne Robertson and James Robertson. 2012. Mastering the Requirements Process: Getting Requirements Right. Addison-wesley."},{"key":"e_1_3_2_120_2","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/3007.001.0001","volume-title":"Finite-State Language Processing","author":"Roche Emmanuel","year":"1997","unstructured":"Emmanuel Roche and Yves Schabes. 1997. Finite-State Language Processing. MIT press."},{"key":"e_1_3_2_121_2","volume-title":"Swarm Planning: The Development of a Planning Methodology to Deal with Climate Adaptation","author":"Roggema Rob","year":"2013","unstructured":"Rob Roggema. 2013. Swarm Planning: The Development of a Planning Methodology to Deal with Climate Adaptation. Springer Science & Business Media."},{"issue":"5","key":"e_1_3_2_122_2","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1080\/09613218.2012.710047","article-title":"Swarm planning for climate change: an alternative pathway for resilience","volume":"40","author":"Roggema Rob","year":"2012","unstructured":"Rob Roggema and Andy Van den Dobbelsteen. 2012. Swarm planning for climate change: an alternative pathway for resilience. Building Research & Information 40, 5 (2012), 606\u2013624.","journal-title":"Building Research & Information"},{"key":"e_1_3_2_123_2","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1007\/978-3-030-36718-3_20","volume-title":"Proceedings of the International Conference on Neural Information Processing","author":"Rychalska Barbara","year":"2019","unstructured":"Barbara Rychalska, Dominika Basaj, Alicja Gosiewska, and Przemysaw Biecek. 2019. Models in the wild: On corruption robustness of neural nlp systems. In Proceedings of the International Conference on Neural Information Processing. Springer, 235\u2013247."},{"key":"e_1_3_2_124_2","first-page":"1831","article-title":"Reliable decisions with threshold calibration","volume":"34","author":"Sahoo Roshni","year":"2021","unstructured":"Roshni Sahoo, Shengjia Zhao, Alyssa Chen, and Stefano Ermon. 2021. Reliable decisions with threshold calibration. Advances in Neural Information Processing Systems 34 (2021), 1831\u20131844.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_125_2","first-page":"27374","volume-title":"Advances in Neural Information Processing SystemsCurran Associates, Inc.","volume":"35","author":"Sandoval-Segura Pedro","year":"2022","unstructured":"Pedro Sandoval-Segura, Vasu Singla, Jonas Geiping, Micah Goldblum, Tom Goldstein, and David Jacobs. 2022. Autoregressive Perturbations for Data Poisoning. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 27374\u201327386. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2022\/file\/af66ac99716a64476c07ae8b089d59f8-Paper-Conference.pdf"},{"key":"e_1_3_2_126_2","first-page":"1","article-title":"Data quality under a computer science perspective","volume":"2","author":"Scannapieco Monica","year":"2002","unstructured":"Monica Scannapieco and Tiziana Catarci. 2002. Data quality under a computer science perspective. Archivi & Computer 2 (2002), 1\u201315.","journal-title":"Archivi & Computer"},{"issue":"4","key":"e_1_3_2_127_2","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1109\/32.799955","article-title":"Qualitative methods in empirical studies of software engineering","volume":"25","author":"Seaman Carolyn B.","year":"1999","unstructured":"Carolyn B. Seaman. 1999. Qualitative methods in empirical studies of software engineering. IEEE Transactions on Software Engineering 25, 4 (1999), 557\u2013572.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_128_2","unstructured":"Witowski Sebastian. 2021. Membership Testing. Retrieved from https:\/\/switowski.com\/blog\/membership-testing"},{"key":"e_1_3_2_129_2","first-page":"85","volume-title":"Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)","author":"Sekhon Jasmine","year":"2019","unstructured":"Jasmine Sekhon and Cody Fleming. 2019. Towards improved testing for deep learning. In Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). IEEE, 85\u201388."},{"key":"e_1_3_2_130_2","first-page":"458","volume-title":"Proceedings of the Computer Safety, Reliability, and Security: SAFECOMP 2018 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE","author":"Shafaei Sina","year":"2018","unstructured":"Sina Shafaei, Stefan Kugele, Mohd Hafeez Osman, and Alois Knoll. 2018. Uncertainty in machine learning: A safety perspective on autonomous driving. In Proceedings of the Computer Safety, Reliability, and Security: SAFECOMP 2018 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE. Springer, 458\u2013464."},{"issue":"3","key":"e_1_3_2_131_2","doi-asserted-by":"crossref","first-page":"675","DOI":"10.1080\/02564602.2021.1894250","article-title":"Path planning for multiple targets interception by the swarm of UAVs based on swarm intelligence algorithms: A review","volume":"39","author":"Sharma Abhishek","year":"2022","unstructured":"Abhishek Sharma, Shraga Shoval, Abhinav Sharma, and Jitendra Kumar Pandey. 2022. Path planning for multiple targets interception by the swarm of UAVs based on swarm intelligence algorithms: A review. IETE Technical Review 39, 3 (2022), 675\u2013697.","journal-title":"IETE Technical Review"},{"key":"e_1_3_2_132_2","first-page":"445","volume-title":"Proceedings of the International Conference on Cryptographic Hardware and Embedded Systems","author":"Shin Hocheol","year":"2017","unstructured":"Hocheol Shin, Dohyun Kim, Yujin Kwon, and Yongdae Kim. 2017. Illusion and dazzle: Adversarial optical channel exploits against lidars for automotive applications. In Proceedings of the International Conference on Cryptographic Hardware and Embedded Systems. Springer, 445\u2013467."},{"issue":"2","key":"e_1_3_2_133_2","first-page":"1","article-title":"Construction of a quality model for machine learning systems","volume":"30","author":"Siebert Julien","year":"2021","unstructured":"Julien Siebert, Lisa Joeckel, Jens Heidrich, Adam Trendowicz, Koji Nakamichi, Kyoko Ohashi, Isao Namba, Rieko Yamamoto, and Mikio Aoyama. 2021. Construction of a quality model for machine learning systems. Software Quality Journal 30, 2 (2021), 1\u201329.","journal-title":"Software Quality Journal"},{"key":"e_1_3_2_134_2","unstructured":"Amanpreet Singh Vedanuj Goswami Vivek Natarajan Yu Jiang Xinlei Chen Meet Shah Marcus Rohrbach Dhruv Batra and Devi Parikh. 2020. Mmf: A multimodal framework for vision and language research."},{"key":"e_1_3_2_135_2","unstructured":"Aman Sinha Hongseok Namkoong Riccardo Volpi and John Duchi. 2017. Certifying some distributional robustness with principled adversarial training. arXiv:1710.10571."},{"key":"e_1_3_2_136_2","unstructured":"British Computer Society. 1998. Glossary of terms used in software testing (Version 6.3). Retrieved from http:\/\/www.testingstandards.co.uk\/bs_7925-1_online.htm"},{"key":"e_1_3_2_137_2","volume-title":"Requirements Engineering: A Good Practice Guide","author":"Sommerville Iain","year":"1997","unstructured":"Iain Sommerville and Peter Sawyer. 1997. Requirements Engineering: A Good Practice Guide. John Wiley & Sons, Inc."},{"key":"e_1_3_2_138_2","doi-asserted-by":"crossref","first-page":"3272","DOI":"10.1145\/3394486.3403379","volume-title":"Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD \u201920)","author":"Srivastava Megha","year":"2020","unstructured":"Megha Srivastava, Besmira Nushi, Ece Kamar, Shital Shah, and Eric Horvitz. 2020. An empirical analysis of backward compatibility in machine learning systems. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD \u201920). ACM, New York, NY, 3272\u20133280. DOI: 10.1145\/3394486.3403379"},{"key":"e_1_3_2_139_2","unstructured":"StackExchange. 2017. \u201cGitHub Stars\u201d Is a Very Useful Metric. But for *What*? Retrieved from https:\/\/opensource.stackexchange.com\/questions\/5110\/github-stars-is-a-very-useful-metric-but-for-what"},{"key":"e_1_3_2_140_2","doi-asserted-by":"crossref","first-page":"108868","DOI":"10.1016\/j.patcog.2022.108868","article-title":"DSLA: Dynamic smooth label assignment for efficient anchor-free object detection","volume":"131","author":"Su Hu","year":"2022","unstructured":"Hu Su, Yonghao He, Rui Jiang, Jiabin Zhang, Wei Zou, and Bin Fan. 2022. DSLA: Dynamic smooth label assignment for efficient anchor-free object detection. Pattern Recognition 131 (2022), Article 108868.","journal-title":"Pattern Recognition"},{"key":"e_1_3_2_141_2","unstructured":"Youcheng Sun Xiaowei Huang Daniel Kroening James Sharp Matthew Hill and Rob Ashmore. 2018. Testing deep neural networks. arXiv:1803.04792."},{"key":"e_1_3_2_142_2","unstructured":"Vincent Tjeng Kai Xiao and Russ Tedrake. 2017. Evaluating robustness of neural networks with mixed integer programming. arXiv:1711.07356."},{"issue":"2","key":"e_1_3_2_143_2","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1137\/1015034","article-title":"Matrix calculus operations and Taylor expansions","volume":"15","author":"Vetter William J","year":"1973","unstructured":"William J Vetter. 1973. Matrix calculus operations and Taylor expansions. SIAM Review 15, 2 (1973), 352\u2013369.","journal-title":"SIAM Review"},{"key":"e_1_3_2_144_2","first-page":"1548","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE)","author":"Wang Song","year":"2021","unstructured":"Song Wang, Nishtha Shrestha, Abarna Kucheri Subburaman, Junjie Wang, Moshi Wei, and Nachiappan Nagappan. 2021. Automatic unit test generation for machine learning libraries: How far are we? In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1548\u20131560."},{"key":"e_1_3_2_145_2","first-page":"1","volume-title":"Proceedings of the IEEE Congress on Evolutionary Computation","author":"Wu Yanghui","year":"2010","unstructured":"Yanghui Wu, John McCall, and David Corne. 2010. Two novel ant colony optimization approaches for Bayesian network structure learning. In Proceedings of the IEEE Congress on Evolutionary Computation. IEEE, 1\u20137."},{"key":"e_1_3_2_146_2","first-page":"10585","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"35","author":"Yan Zhicong","year":"2021","unstructured":"Zhicong Yan, Gaolei Li, Yuan TIan, Jun Wu, Shenghong Li, Mingzhe Chen, and H Vincent Poor. 2021. Dehib: Deep hidden backdoor attack on semi-supervised learning via adversarial perturbation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, 10585\u201310593."},{"issue":"5","key":"e_1_3_2_147_2","doi-asserted-by":"crossref","first-page":"480","DOI":"10.1038\/s42256-022-00482-8","article-title":"Autonomous environment-adaptive microrobot swarm navigation enabled by deep learning-based real-time distribution planning","volume":"4","author":"Yang Lidong","year":"2022","unstructured":"Lidong Yang, Jialin Jiang, Xiaojie Gao, Qinglong Wang, Qi Dou, and Li Zhang. 2022. Autonomous environment-adaptive microrobot swarm navigation enabled by deep learning-based real-time distribution planning. Nature Machine Intelligence 4, 5 (2022), 480\u2013493.","journal-title":"Nature Machine Intelligence"},{"key":"e_1_3_2_148_2","unstructured":"Xiao Yang Weiqing Liu Dong Zhou Jiang Bian and Tie-Yan Liu. 2020. Qlib: An AI-oriented quantitative investment platform. arXiv:2009.11189."},{"issue":"2","key":"e_1_3_2_149_2","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1002\/stvr.430","article-title":"Regression testing minimization, selection and prioritization: A survey","volume":"22","author":"Yoo Shin","year":"2012","unstructured":"Shin Yoo and Mark Harman. 2012. Regression testing minimization, selection and prioritization: A survey. Software testing, verification and reliability 22, 2 (2012), 67\u2013120.","journal-title":"Software testing, verification and reliability"},{"key":"e_1_3_2_150_2","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1109\/ICST.2008.47","volume-title":"Proceedings of the 2008 1st International Conference on Software Testing, Verification, and Validation","author":"Zaidman Andy","year":"2008","unstructured":"Andy Zaidman, Bart Van Rompaey, Serge Demeyer, and Arie van Deursen. 2008. Mining software repositories to study co-evolution of production & test code. In Proceedings of the 2008 1st International Conference on Software Testing, Verification, and Validation, 220\u2013229. DOI: 10.1109\/ICST.2008.47"},{"issue":"1","key":"e_1_3_2_151_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TSE.2019.2962027","article-title":"Machine learning testing: Survey, landscapes and horizons","volume":"48","author":"Zhang Jie M","year":"2020","unstructured":"Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering 48, 1 (2020), 1\u201336.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_152_2","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1016\/j.bdr.2015.12.001","article-title":"Finding the best classification threshold in imbalanced classification","volume":"5","author":"Zou Quan","year":"2016","unstructured":"Quan Zou, Sifa Xie, Ziyu Lin, Meihong Wu, and Ying Ju. 2016. Finding the best classification threshold in imbalanced classification. Big Data Research 5 (2016), 2\u20138.","journal-title":"Big Data Research"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3680463","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3680463","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:58:26Z","timestamp":1750294706000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3680463"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,26]]},"references-count":151,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1,31]]}},"alternative-id":["10.1145\/3680463"],"URL":"https:\/\/doi.org\/10.1145\/3680463","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,26]]},"assertion":[{"value":"2021-09-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-11","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}