{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,19]],"date-time":"2026-09-19T13:42:38Z","timestamp":1789825358553,"version":"4.0.1"},"reference-count":83,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2025,2,28]],"date-time":"2025-02-28T00:00:00Z","timestamp":1740700800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62325201"],"award-info":[{"award-number":["62325201"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,3,31]]},"abstract":"<jats:p>This article conducts fairness testing of automated pedestrian detection, a crucial but under-explored issue in autonomous driving systems. We evaluate eight state-of-the-art deep learning-based pedestrian detectors across demographic groups on large-scale real-world datasets. To enable thorough fairness testing, we provide extensive annotations for the datasets, resulting in 8,311 images with 16,070 gender labels, 20,115 age labels, and 3,513 skin tone labels. Our findings reveal significant fairness issues, particularly related to age. The proportion of undetected children is 20.14% higher compared to adults. Furthermore, we explore how various driving scenarios affect the fairness of pedestrian detectors. We find that pedestrian detectors demonstrate significant gender biases during night time, potentially exacerbating the prevalent societal issue of female safety concerns during nighttime out. Moreover, we observe that pedestrian detectors can demonstrate both enhanced fairness and superior performance under specific driving conditions, which challenges the fairness-performance tradeoff theory widely acknowledged in the fairness literature. We publicly release the code, data, and results to support future research on fairness in autonomous driving.<\/jats:p>","DOI":"10.1145\/3702989","type":"journal-article","created":{"date-parts":[[2024,11,2]],"date-time":"2024-11-02T11:26:49Z","timestamp":1730546809000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":19,"title":["Bias behind the Wheel: Fairness Testing of Autonomous Driving Systems"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-4058-6733","authenticated-orcid":false,"given":"Xinyue","family":"Li","sequence":"first","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4765-1893","authenticated-orcid":false,"given":"Zhenpeng","family":"Chen","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0481-7264","authenticated-orcid":false,"given":"Jie M.","family":"Zhang","sequence":"additional","affiliation":[{"name":"King\u2019s College London, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9146-442X","authenticated-orcid":false,"given":"Federica","family":"Sarro","sequence":"additional","affiliation":[{"name":"University College London, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-6924-2319","authenticated-orcid":false,"given":"Ying","family":"Zhang","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7908-8484","authenticated-orcid":false,"given":"Xuanzhe","family":"Liu","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,2,28]]},"reference":[{"key":"e_1_3_1_2_2","first-page":"5187","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Liu Wei","year":"2019","unstructured":"Wei Liu, Shengcai Liao, Weiqiang Ren, Weidong Hu, and Yinan Yu. 2019. High-level semantic feature detection: A new perspective for pedestrian detection. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, 5187\u20135196."},{"key":"e_1_3_1_3_2","unstructured":"Daisuke Wakabayashi. 2018. Self-Driving Uber Car Kills Pedestrian in Arizona Where Robots Roam. Retrieved from https:\/\/www.nytimes.com\/2018\/03\/19\/technology\/uber-driverless-fatality.html"},{"key":"e_1_3_1_4_2","first-page":"359","volume-title":"Proceedings of the 42nd International Conference on Software Engineering (ICSE \u201920)","author":"Stocco Andrea","year":"2020","unstructured":"Andrea Stocco, Michael Weiss, Marco Calzana, and Paolo Tonella. 2020. Misbehaviour prediction for autonomous driving systems. In Proceedings of the 42nd International Conference on Software Engineering (ICSE \u201920), 359\u2013371."},{"key":"e_1_3_1_5_2","unstructured":"Irtiza Hasan Shengcai Liao Jinpeng Li Saad Ullah Akram and Ling Shao. 2022. Pedestrian detection: Domain generalization CNNs transformers and beyond. arXiv:2201.03176. Retrieved from https:\/\/arxiv.org\/abs\/2201.03176"},{"key":"e_1_3_1_6_2","unstructured":"GitHub. 2023. Apollo. Retrieved from https:\/\/github.com\/ApolloAuto\/apollo-model-yolox"},{"key":"e_1_3_1_7_2","unstructured":"Wikipedia. 2024. Afro-Germans. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Afro-Germans"},{"key":"e_1_3_1_8_2","unstructured":"Wikipedia. 2024. Demographics-of-Germany. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Demographics_of_Germany"},{"key":"e_1_3_1_9_2","unstructured":"Mohamad Moslimani Christine Tamir Abby Budiman Luis Noe-Bustamante and Lauren Mora. 2024. Facts about the U.S. Black Population. Retrieved from https:\/\/www.pewresearch.org\/social-trends\/fact-sheet\/facts-about-the-us-black-population\/"},{"key":"e_1_3_1_10_2","unstructured":"GitHub. 2024. Replication Package. Retrieved from https:\/\/github.com\/xinyuelxy\/Bias-Behind-the-Wheel-Fairness-Testing-of-Autonomous-Driving-Systems"},{"key":"e_1_3_1_11_2","first-page":"750","volume-title":"Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201920)","author":"Chen Zhenpeng","year":"2020","unstructured":"Zhenpeng Chen, Yanbin Cao, Yuanqiang Liu, Haoyu Wang, Tao Xie, and Xuanzhe Liu. 2020. A comprehensive study on challenges in deploying deep learning based software. In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201920), 750\u2013762."},{"key":"e_1_3_1_12_2","first-page":"671","article-title":"Big Data's disparate impact","volume":"104","author":"Barocas Solon","year":"2016","unstructured":"Solon Barocas and Andrew D. Selbst. 2016. Big Data's disparate impact. California Law Review 104 (2016), 671.","journal-title":"California Law Review"},{"issue":"6","key":"e_1_3_1_13_2","first-page":"983","article-title":"Image quality assessment employing RMS contrast and histogram similarity","volume":"15","author":"Bhuiyan Md. Al-Amin","year":"2018","unstructured":"Md. Al-Amin Bhuiyan and Abdul Raouf Khan. 2018. Image quality assessment employing RMS contrast and histogram similarity. The International Arab Journal of Information Technology 15, 6 (2018), 983\u2013989.","journal-title":"The International Arab Journal of Information Technology"},{"key":"e_1_3_1_14_2","volume-title":"Adverse Impact and Test Validation: A Practitioner's Guide to Valid and Defensible Employment Testing","author":"Biddle D.","year":"2005","unstructured":"D. Biddle. 2005. Adverse Impact and Test Validation: A Practitioner's Guide to Valid and Defensible Employment Testing. Gower."},{"key":"e_1_3_1_15_2","first-page":"11679","volume-title":"Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR \u201920)","author":"Bijelic Mario","year":"2020","unstructured":"Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, and Felix Heide. 2020. Seeing through fog without seeing fog: Deep multimodal sensor fusion in unseen adverse weather. In Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR \u201920), 11679\u201311689."},{"key":"e_1_3_1_16_2","first-page":"642","volume-title":"Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201920)","author":"Biswas Sumon","year":"2020","unstructured":"Sumon Biswas and Hridesh Rajan. 2020. Do the machine learning models on a crowd sourced platform exhibit bias? An empirical study on model fairness. In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201920), 642\u2013653."},{"key":"e_1_3_1_17_2","first-page":"981","volume-title":"Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201921)","author":"Biswas Sumon","year":"2021","unstructured":"Sumon Biswas and Hridesh Rajan. 2021. Fair preprocessing: Towards understanding compositional fairness of data transformers in machine learning pipeline. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201921), 981\u2013993."},{"key":"e_1_3_1_18_2","unstructured":"Martim Brandao. 2019. Age and gender bias in pedestrian detection algorithms. arXiv: 1906.10490. Retrieved from https:\/\/arxiv.org\/abs\/1906.10490"},{"issue":"8","key":"e_1_3_1_19_2","doi-asserted-by":"crossref","first-page":"1844","DOI":"10.1109\/TPAMI.2019.2897684","article-title":"EuroCity Persons: A novel benchmark for person detection in traffic scenes","volume":"41","author":"Braun Markus","year":"2019","unstructured":"Markus Braun, Sebastian Krebs, Fabian Flohr, and Dariu M. Gavrila. 2019. EuroCity Persons: A novel benchmark for person detection in traffic scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 8 (2019), 1844\u20131861.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_1_20_2","first-page":"6154","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Cai Zhaowei","year":"2018","unstructured":"Zhaowei Cai and Nuno Vasconcelos. 2018. Cascade R-CNN: Delving into high quality object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6154\u20136162."},{"key":"e_1_3_1_21_2","first-page":"4913","article-title":"From handcrafted to deep features for pedestrian detection: A survey","volume":"44","author":"Cao Jiale","year":"2020","unstructured":"Jiale Cao, Yanwei Pang, Jin Xie, Fahad Shahbaz Khan, and Ling Shao. 2020. From handcrafted to deep features for pedestrian detection: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 (2020), 4913\u20134934.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_1_22_2","volume-title":"Statistical Inference","author":"Casella G.","year":"2007","unstructured":"G. Casella and R. L. Berger. 2007. Statistical Inference (2nd. ed.). Duxbury Press."},{"key":"e_1_3_1_23_2","first-page":"429","volume-title":"Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201921)","author":"Chakraborty Joymallya","year":"2021","unstructured":"Joymallya Chakraborty, Suvodeep Majumder, and Tim Menzies. 2021. Bias in machine learning software: Why? How? What to do? In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201921), 429\u2013440."},{"key":"e_1_3_1_24_2","first-page":"654","volume-title":"Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201920)","author":"Chakraborty Joymallya","year":"2020","unstructured":"Joymallya Chakraborty, Suvodeep Majumder, Zhe Yu, and Tim Menzies. 2020. Fairway: A way to build fair ML software. In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201920), 654\u2013665."},{"key":"e_1_3_1_25_2","unstructured":"Kai Chen Jiaqi Wang Jiangmiao Pang Yuhang Cao Yu Xiong Xiaoxiao Li Shuyang Sun Wansen Feng Ziwei Liu Jiarui Xu et\u00a0al. 2019. MMDetection: Open MMLab detection toolbox and benchmark. arXiv:1906.07155. Retrieved from https:\/\/arxiv.org\/abs\/1906.07155"},{"key":"e_1_3_1_26_2","first-page":"674","volume-title":"Proceedings of the 43rd IEEE\/ACM International Conference on Software Engineering (ICSE \u201921)","author":"Chen Zhenpeng","year":"2021","unstructured":"Zhenpeng Chen, Huihan Yao, Yiling Lou, Yanbin Cao, Yuanqiang Liu, Haoyu Wang, and Xuanzhe Liu. 2021. An empirical study on deployment faults of deep learning based mobile applications. In Proceedings of the 43rd IEEE\/ACM International Conference on Software Engineering (ICSE \u201921), 674\u2013685."},{"issue":"5","key":"e_1_3_1_27_2","first-page":"137:1","article-title":"Fairness testing: A comprehensive survey and analysis of trends","volume":"33","author":"Chen Zhenpeng","year":"2024","unstructured":"Zhenpeng Chen, Jie M. Zhang, Max Hort, Mark Harman, and Federica Sarro. 2024. Fairness testing: A comprehensive survey and analysis of trends. ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 137:1\u2013137:59.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_28_2","first-page":"1122","volume-title":"Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201922)","author":"Chen Zhenpeng","year":"2022","unstructured":"Zhenpeng Chen, Jie M. Zhang, Federica Sarro, and Mark Harman. 2022. MAAT: A novel ensemble approach to addressing fairness and performance bugs for machine learning software. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201922), 1122\u20131134."},{"issue":"4","key":"e_1_3_1_29_2","first-page":"106:1","article-title":"A comprehensive empirical study of bias mitigation methods for machine learning classifiers","volume":"32","author":"Chen Zhenpeng","year":"2023","unstructured":"Zhenpeng Chen, Jie M. Zhang, Federica Sarro, and Mark Harman. 2023. A comprehensive empirical study of bias mitigation methods for machine learning classifiers. ACM Transactions on Software Engineering and Methodology 32, 4 (2023), 106:1\u2013106:30.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_30_2","volume-title":"Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering (ICSE \u201924)","volume":"160","author":"Chen Zhenpeng","year":"2024","unstructured":"Zhenpeng Chen, Jie M. Zhang, Federica Sarro, and Mark Harman. 2024. Fairness improvement with multiple protected attributes: How far are we? In Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering (ICSE \u201924), 160:1\u2013160:13."},{"key":"e_1_3_1_31_2","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen Jacob","year":"1960","unstructured":"Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20 (1960), 37\u201346.","journal-title":"Educational and Psychological Measurement"},{"key":"e_1_3_1_32_2","unstructured":"Sam Corbett-Davies and Sharad Goel. 2018. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv: 1808.00023. Retrieved from https:\/\/arxiv.org\/abs\/1808.00023"},{"key":"e_1_3_1_33_2","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1109\/CVPR.2009.5206631","volume-title":"Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition","author":"Dollar Piotr","year":"2009","unstructured":"Piotr Dollar, Christian Wojek, Bernt Schiele, and Pietro Perona. 2009. Pedestrian detection: A benchmark. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, 304\u2013311."},{"key":"e_1_3_1_34_2","doi-asserted-by":"crossref","first-page":"743","DOI":"10.1109\/TPAMI.2011.155","article-title":"Pedestrian detection: An evaluation of the state of the art","volume":"34","author":"Doll\u00e1r Piotr","year":"2012","unstructured":"Piotr Doll\u00e1r, Christian Wojek, Bernt Schiele, and Pietro Perona. 2012. Pedestrian detection: An evaluation of the state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (2012), 743\u2013761.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_1_36_2","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1145\/2783258.2783311","volume-title":"Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD \u201915)","author":"Feldman Michael","year":"2015","unstructured":"Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD \u201915), 259\u2013268."},{"key":"e_1_3_1_37_2","first-page":"115","volume-title":"Proceedings of the 16th IEEE International Requirements Engineering Conference (RE \u201908)","author":"Finkelstein Anthony","year":"2008","unstructured":"Anthony Finkelstein, Mark Harman, S. Afshin Mansouri, Jian Ren, and Yuanyuan Zhang. 2008. \u201cFairness analysis\u201d in requirements assignments. In Proceedings of the 16th IEEE International Requirements Engineering Conference (RE \u201908), 115\u2013124."},{"key":"e_1_3_1_38_2","first-page":"385","volume-title":"Proceedings of the 42nd International Conference on Software Engineering (ICSE \u201920)","author":"Garcia Joshua","year":"2020","unstructured":"Joshua Garcia, Yang Feng, Junjie Shen, Sumaya Almanee, Yuan Xia, and Qi Alfred Chen. 2020. A comprehensive study of autonomous vehicle bugs. In Proceedings of the 42nd International Conference on Software Engineering (ICSE \u201920), 385\u2013396."},{"key":"e_1_3_1_39_2","first-page":"316","volume-title":"Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM \u201922)","author":"Ge Yingqiang","year":"2022","unstructured":"Yingqiang Ge, Xiaoting Zhao, Lucia Yu, Saurabh Paul, Diane Hu, Chu-Cheng Hsieh, and Yongfeng Zhang. 2022. Toward Pareto efficient fairness-utility trade-off in recommendation through reinforcement learning. In Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM \u201922), 316\u2013324."},{"key":"e_1_3_1_40_2","unstructured":"Zheng Ge Songtao Liu Feng Wang Zeming Li and Jian Sun. 2021. Yolox: Exceeding yolo series in 2021. arXiv:2107.08430. Retrieved from https:\/\/arxiv.org\/abs\/2107.08430"},{"key":"e_1_3_1_41_2","first-page":"1533","volume-title":"Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923)","author":"Gohar Usman","year":"2023","unstructured":"Usman Gohar, Sumon Biswas, and Hridesh Rajan. 2023. Towards understanding fairness and its composition in ensemble machine learning. In Proceedings of the 45th IEEE\/ACM International Conference on Software Engineering (ICSE \u201923), 1533\u20131545."},{"key":"e_1_3_1_42_2","doi-asserted-by":"crossref","first-page":"480","DOI":"10.1145\/3533767.3534397","volume-title":"Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201922)","author":"Guo An","year":"2022","unstructured":"An Guo, Yang Feng, and Zhenyu Chen. 2022. LiRTest: Augmenting LiDAR point clouds for automated testing of autonomous driving systems. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201922), 480\u2013492."},{"key":"e_1_3_1_43_2","first-page":"11323","volume-title":"The elephant in the roomProceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR \u201920)","author":"Hasan Irtiza","year":"2020","unstructured":"Irtiza Hasan, Shengcai Liao, Jinpeng Li, Saad Ullah Akram, and Ling Shao. 2020. Generalizable pedestrian detection: The elephant in the room. In Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR \u201920), 11323\u201311332."},{"issue":"2","key":"e_1_3_1_44_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3631326","article-title":"Bias mitigation for machine learning classifiers: A comprehensive survey","volume":"1","author":"Hort Max","year":"2024","unstructured":"Max Hort, Zhenpeng Chen, Jie M. Zhang, Mark Harman, and Federica Sarro. 2024. Bias mitigation for machine learning classifiers: A comprehensive survey. ACM Journal on Responsible Computing 1, 2 (2024), 1\u201352.","journal-title":"ACM Journal on Responsible Computing"},{"key":"e_1_3_1_45_2","first-page":"994","volume-title":"Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201921)","author":"Hort Max","year":"2021","unstructured":"Max Hort, Jie M. Zhang, Federica Sarro, and Mark Harman. 2021. Fairea: A model behaviour mutation approach to benchmarking bias mitigation methods. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201921), 994\u20131006."},{"key":"e_1_3_1_46_2","first-page":"366","volume-title":"Neurocomputing","author":"Jakhetiya Vinit","year":"2018","unstructured":"Vinit Jakhetiya, Weisi Lin, Sunil Prasad Jaiswal, Ke Gu, and Sharath Chandra Guntuku. 2018. Just noticeable difference for natural images using RMS contrast and feed-back mechanism. Neurocomputing 275 (2018), 366\u2013376."},{"key":"e_1_3_1_47_2","volume-title":"Proceedings of the AAAI Workshop on Artificial Intelligence with Biased or Scarce Data (AIBSD \u201922)","volume":"33","author":"Kogure Shunsuke","year":"2022","unstructured":"Shunsuke Kogure, Kai Watabe, Ryosuke Yamada, Yoshimitsu Aoki, Akio Nakamura, and Hirokatsu Kataoka. 2022. Age should not matter: Towards more accurate pedestrian detection via self-training. In Proceedings of the AAAI Workshop on Artificial Intelligence with Biased or Scarce Data (AIBSD \u201922), 33, 1 (2022). Retrieved from https:\/\/www.mdpi.com\/2813-0324\/3\/1\/11"},{"issue":"1","key":"e_1_3_1_48_2","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis J. Richard","year":"1977","unstructured":"J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics 33, 1 (1977), 159\u2013174.","journal-title":"Biometrics"},{"key":"e_1_3_1_49_2","first-page":"2980","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Lin Tsung-Yi","year":"2017","unstructured":"Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll\u00e1r. 2017. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, 2980\u20132988."},{"key":"e_1_3_1_50_2","first-page":"21","volume-title":"Proceedings of the 14th European Conference on Computer Vision (ECCV \u201916)","author":"Liu Wei","year":"2016","unstructured":"Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. 2016. SSD: Single shot multibox detector. In Proceedings of the 14th European Conference on Computer Vision (ECCV \u201916), 21\u201337."},{"key":"e_1_3_1_51_2","first-page":"618","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV \u201918)","author":"Liu Wei","year":"2018","unstructured":"Wei Liu, Shengcai Liao, Weidong Hu, Xuezhi Liang, and Xiao Chen. 2018. Learning efficient single-stage pedestrian detectors by asymptotic localization fitting. In Proceedings of the European Conference on Computer Vision (ECCV \u201918), 618\u2013634."},{"issue":"6","key":"e_1_3_1_52_2","first-page":"156:1","article-title":"Rise of distributed deep learning training in the big model era: From a software engineering perspective","volume":"32","author":"Liu Xuanzhe","year":"2023","unstructured":"Xuanzhe Liu, Diandian Gu, Zhenpeng Chen, Jinfeng Wen, Zili Zhang, Yun Ma, Haoyu Wang, and Xin Jin. 2023. Rise of distributed deep learning training in the big model era: From a software engineering perspective. ACM Transactions on Software Engineering and Methodology 32, 6 (2023), 156:1\u2013156:26.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_53_2","first-page":"502","volume-title":"Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201923)","author":"Nguyen Giang","year":"2023","unstructured":"Giang Nguyen, Sumon Biswas, and Hridesh Rajan. 2023. Fix fairness, don\u2019t ruin accuracy: Performance aware fairness repair using AutoML. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201923), 502\u2013514."},{"key":"e_1_3_1_54_2","first-page":"384","volume-title":"Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE \u201916)","author":"Niu Nan","year":"2016","unstructured":"Nan Niu, Wentao Wang, and Arushi Gupta. 2016. Gray links in the use of requirements traceability. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE \u201916), 384\u2013395."},{"key":"e_1_3_1_55_2","first-page":"4967","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Pang Yanwei","year":"2019","unstructured":"Yanwei Pang, Jin Xie, Muhammad Haris Khan, Rao Muhammad Anwer, Fahad Shahbaz Khan, and Ling Shao. 2019. Mask-guided attention network for occluded pedestrian detection. In Proceedings of the IEEE\/CVF International Conference on Computer Vision, 4967\u20134975."},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3361566"},{"issue":"10","key":"e_1_3_1_57_2","doi-asserted-by":"crossref","first-page":"2032","DOI":"10.1364\/JOSAA.7.002032","article-title":"Contrast in complex images","volume":"7","author":"Peli Eli","year":"1990","unstructured":"Eli Peli. 1990. Contrast in complex images. Journal of the Optical Society of America A 7, 10 (Oct. 1990), 2032\u20132040.","journal-title":"Journal of the Optical Society of America A"},{"key":"e_1_3_1_58_2","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.visres.2013.04.015","article-title":"Measuring contrast sensitivity","volume":"90","author":"Pelli Denis G.","year":"2013","unstructured":"Denis G. Pelli and Peter Bex. 2013. Measuring contrast sensitivity. Vision Research 90 (2013), 10\u201314.","journal-title":"Vision Research"},{"key":"e_1_3_1_59_2","first-page":"126","volume-title":"Proceedings of the 2009 ICSE Workshop on Software Engineering for Adaptive and Self-Managing Systems (SEAMS \u201909)","author":"Qureshi Nauman A.","year":"2009","unstructured":"Nauman A. Qureshi and Anna Perini. 2009. Engineering adaptive requirements. In Proceedings of the 2009 ICSE Workshop on Software Engineering for Adaptive and Self-Managing Systems (SEAMS \u201909), 126\u2013131."},{"key":"e_1_3_1_60_2","first-page":"779","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Redmon Joseph","year":"2016","unstructured":"Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 779\u2013788."},{"issue":"6","key":"e_1_3_1_61_2","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards real-time object detection with region proposal networks","volume":"39","author":"Ren Shaoqing","year":"2017","unstructured":"Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2017. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 6 (2017), 1137\u20131149.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_1_62_2","first-page":"32","volume-title":"Proceedings of the 16th European Conference on Computer Vision (ECCV \u201920)","author":"Song Xiaolin","year":"2020","unstructured":"Xiaolin Song, Kaili Zhao, Wen-Sheng Chu, Honggang Zhang, and Jun Guo. 2020. Progressive refinement network for occluded pedestrian detection. In Proceedings of the 16th European Conference on Computer Vision (ECCV \u201920), 32\u201348."},{"issue":"6","key":"e_1_3_1_63_2","first-page":"156","article-title":"Fairness testing of machine translation systems","volume":"33","author":"Sun Zeyu","year":"2024","unstructured":"Zeyu Sun, Zhenpeng Chen, Jie Zhang, and Dan Hao. 2024. Fairness testing of machine translation systems. ACM Transactions on Software Engineering and Methodology 33, 6 (2024), 156.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_64_2","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1145\/3180155.3180220","volume-title":"Proceedings of the 40th International Conference on Software Engineering (ICSE \u201918)","author":"Tian Yuchi","year":"2018","unstructured":"Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. DeepTest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering (ICSE \u201918), 303\u2013314."},{"issue":"4","key":"e_1_3_1_65_2","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1007\/s10664-021-09956-6","article-title":"On systematically building a controlled natural language for functional requirements","volume":"26","author":"Veizaga Alvaro","year":"2021","unstructured":"Alvaro Veizaga, Mauricio Alf\u00e9rez, Damiano Torre, Mehrdad Sabetzadeh, and Lionel C. Briand. 2021. On systematically building a controlled natural language for functional requirements. Empirical Software Engineering 26, 4 (2021), 79.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_1_66_2","first-page":"515","volume-title":"Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201923)","author":"Wan Yuxuan","year":"2023","unstructured":"Yuxuan Wan, Wenxuan Wang, Pinjia He, Jiazhen Gu, Haonan Bai, and Michael R. Lyu. 2023. BiasAsker: Measuring the bias in conversational AI system. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201923), 515\u2013527."},{"key":"e_1_3_1_67_2","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1109\/MSR59073.2023.00019","volume-title":"Proceedings of the 20th IEEE\/ACM International Conference on Mining Software Repositories (MSR \u201923)","author":"Wang Chao","year":"2023","unstructured":"Chao Wang, Zhenpeng Chen, and Minghui Zhou. 2023. AutoML from software engineering perspective: Landscapes and challenges. In Proceedings of the 20th IEEE\/ACM International Conference on Mining Software Repositories (MSR \u201923), 39\u201351."},{"issue":"1","key":"e_1_3_1_68_2","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1108\/TR-12-2021-0563","article-title":"Do young women travellers really consider the risk of sexual harassment during night travel? Evening travel vs midnight travel","volume":"78","author":"Wang Xin","year":"2023","unstructured":"Xin Wang, Ivan Ka Wai Lai, and Kun Wang. 2023. Do young women travellers really consider the risk of sexual harassment during night travel? Evening travel vs midnight travel. Tourism Review 78, 1 (2023), 58\u201371.","journal-title":"Tourism Review"},{"key":"e_1_3_1_69_2","first-page":"416","volume-title":"Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201921)","author":"Wen Jinfeng","year":"2021","unstructured":"Jinfeng Wen, Zhenpeng Chen, Yi Liu, Yiling Lou, Yun Ma, Gang Huang, Xin Jin, and Xuanzhe Liu. 2021. An empirical study on challenges of application development in serverless computing. In Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201921), 416\u2013428."},{"key":"e_1_3_1_70_2","first-page":"8780","volume-title":"Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019 (NeurIPS \u201919)","author":"Wick Michael L.","year":"2019","unstructured":"Michael L. Wick, Swetasudha Panda, and Jean-Baptiste Tristan. 2019. Unlocking fairness: A trade-off revisited. In Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019 (NeurIPS \u201919), 8780\u20138789."},{"key":"e_1_3_1_71_2","unstructured":"Benjamin Wilson Judy Hoffman and Jamie H. Morgenstern. 2019. Predictive inequity in object detection. arXiv:1902.11097. Retrieved from https:\/\/arxiv.org\/abs\/1902.11097"},{"key":"e_1_3_1_72_2","first-page":"2633","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR \u201920)","author":"Yu Fisher","year":"2020","unstructured":"Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. 2020. BDD100K: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR \u201920), 2633\u20132642."},{"key":"e_1_3_1_73_2","first-page":"962","volume-title":"Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS \u201917)","author":"Zafar Muhammad Bilal","year":"2017","unstructured":"Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi. 2017. Fairness constraints: Mechanisms for fair classification. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS \u201917), 962\u2013970."},{"key":"e_1_3_1_74_2","first-page":"1436","volume-title":"Proceedings of 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE \u201921)","author":"Zhang Jie M.","year":"2021","unstructured":"Jie M. Zhang and Mark Harman. 2021. \u201cIgnorance and prejudice\u201d in software fairness. In Proceedings of 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE \u201921), 1436\u20131447."},{"issue":"2","key":"e_1_3_1_75_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TSE.2019.2962027","article-title":"Machine learning testing: Survey, landscapes and horizons","volume":"48","author":"Zhang Jie M.","year":"2022","unstructured":"Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2022. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering 48, 2 (2022), 1\u201336.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_1_76_2","first-page":"6","volume-title":"Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201922)","author":"Zhang Mengdi","year":"2022","unstructured":"Mengdi Zhang and Jun Sun. 2022. Adaptive fairness improvement based on causality analysis. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE \u201922), 6\u201317."},{"issue":"6","key":"e_1_3_1_77_2","first-page":"137:1","article-title":"TestSGD: Interpretable testing of neural networks against subtle group discrimination","volume":"32","author":"Zhang Mengdi","year":"2023","unstructured":"Mengdi Zhang, Jun Sun, Jingyi Wang, and Bing Sun. 2023. TestSGD: Interpretable testing of neural networks against subtle group discrimination. ACM Transactions on Software Engineering and Methodology 32, 6 (2023), 137:1\u2013137:24.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_78_2","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1145\/3238147.3238187","volume-title":"Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering (ASE \u201918)","author":"Zhang Mengshi","year":"2018","unstructured":"Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. 2018. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering (ASE \u201918), 132\u2013142."},{"key":"e_1_3_1_79_2","doi-asserted-by":"crossref","first-page":"1259","DOI":"10.1109\/CVPR.2016.141","volume-title":"Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR \u201916)","author":"Zhang Shanshan","year":"2016","unstructured":"Shanshan Zhang, Rodrigo Benenson, Mohamed Omran, Jan Hosang, and Bernt Schiele. 2016. How far are we from solving pedestrian detection? In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR \u201916), 1259\u20131267."},{"key":"e_1_3_1_80_2","doi-asserted-by":"crossref","first-page":"4457","DOI":"10.1109\/CVPR.2017.474","volume-title":"Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR \u201917)","author":"Zhang Shanshan","year":"2017","unstructured":"Shanshan Zhang, Rodrigo Benenson, and Bernt Schiele. 2017. CityPersons: A diverse dataset for pedestrian detection. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR \u201917), 4457\u20134465."},{"issue":"2","key":"e_1_3_1_81_2","first-page":"37:1","article-title":"Characterizing and detecting WebAssembly runtime bugs","volume":"33","author":"Zhang Yixuan","year":"2024","unstructured":"Yixuan Zhang, Shangtong Cao, Haoyu Wang, Zhenpeng Chen, Xiapu Luo, Dongliang Mu, Yun Ma, Gang Huang, and Xuanzhe Liu. 2024. Characterizing and detecting WebAssembly runtime bugs. ACM Transactions on Software Engineering and Methodology 33, 2 (2024), 37:1\u201337:29.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_82_2","doi-asserted-by":"crossref","first-page":"347","DOI":"10.1145\/3377811.3380422","volume-title":"Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering (ICSE \u201920)","author":"Zhou Husheng","year":"2020","unstructured":"Husheng Zhou, Wei Li, Zelun Kong, Junfeng Guo, Yuqun Zhang, Bei Yu, Lingming Zhang, and Cong Liu. 2020. DeepBillboard: Systematic physical-world testing of autonomous driving systems. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering (ICSE \u201920), 347\u2013358."},{"key":"e_1_3_1_83_2","first-page":"257","volume-title":"Proceedings of the IEEE","volume":"111","author":"Zou Zhengxia","year":"2023","unstructured":"Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. 2023. Object detection in 20 years: A survey. Proceedings of the IEEE 111, 3 (2023), 257\u2013276."},{"key":"e_1_3_1_84_2","first-page":"823","volume-title":"Proceedings of the 58th ACM\/IEEE Design Automation Conference (DAC \u201921)","author":"Zuo Feilong","year":"2021","unstructured":"Feilong Zuo, Zhengxiong Luo, Junze Yu, Zhe Liu, and Yu Jiang. 2021. PAVFuzz: State-sensitive fuzz testing of protocols in autonomous vehicles. In Proceedings of the 58th ACM\/IEEE Design Automation Conference (DAC \u201921), 823\u2013828."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3702989","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3702989","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T21:18:04Z","timestamp":1750281484000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3702989"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,28]]},"references-count":83,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,3,31]]}},"alternative-id":["10.1145\/3702989"],"URL":"https:\/\/doi.org\/10.1145\/3702989","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,28]]},"assertion":[{"value":"2024-04-04","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-10-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}