{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T10:05:33Z","timestamp":1775815533011,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":73,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,11,30]],"date-time":"2023-11-30T00:00:00Z","timestamp":1701302400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Research Grants Council of the Hong Kong Special Administrative Region, China","award":["CUHK 14206921"],"award-info":[{"award-number":["CUHK 14206921"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62102340"],"award-info":[{"award-number":["62102340"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,11,30]]},"DOI":"10.1145\/3611643.3616310","type":"proceedings-article","created":{"date-parts":[[2023,11,30]],"date-time":"2023-11-30T23:14:38Z","timestamp":1701386078000},"page":"515-527","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":40,"title":["BiasAsker: Measuring the Bias in Conversational AI System"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-6739-4675","authenticated-orcid":false,"given":"Yuxuan","family":"Wan","sequence":"first","affiliation":[{"name":"Chinese University of Hong Kong, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9803-8204","authenticated-orcid":false,"given":"Wenxuan","family":"Wang","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3377-8129","authenticated-orcid":false,"given":"Pinjia","family":"He","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, Shenzhen, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5831-9474","authenticated-orcid":false,"given":"Jiazhen","family":"Gu","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-6061-107X","authenticated-orcid":false,"given":"Haonan","family":"Bai","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3666-5798","authenticated-orcid":false,"given":"Michael R.","family":"Lyu","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, Hong Kong, China"}]}],"member":"320","published-online":{"date-parts":[[2023,11,30]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"2021. Coverage-Guided Testing for Recurrent Neural Networks. IEEE Transactions on Reliability."},{"key":"e_1_3_2_2_2_1","unstructured":"Adelaide A.. 2023. Main types of questions in English (with examples). https:\/\/preply.com\/en\/blog\/types-of-questions-in-english\/"},{"key":"e_1_3_2_2_3_1","volume-title":"Neural Machine Translation by Jointly Learning to Align and Translate. ICLR, abs\/1409.0473","author":"Bahdanau Dzmitry","year":"2015","unstructured":"Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. ICLR, abs\/1409.0473 (2015)."},{"key":"e_1_3_2_2_4_1","volume-title":"Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts. In Conference on Empirical Methods in Natural Language Processing.","author":"Baheti Ashutosh","unstructured":"Ashutosh Baheti, Maarten Sap, Alan Ritter, and Mark O. Riedl. 2021. Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts. In Conference on Empirical Methods in Natural Language Processing."},{"key":"e_1_3_2_2_5_1","unstructured":"Newsbeat BBC. 2019. Taylor Swift \u2019tried to sue\u2019 Microsoft over racist chatbot Tay. https:\/\/www.bbc.com\/news\/newsbeat-49645508 Accessed: 2022-08-01"},{"key":"e_1_3_2_2_6_1","unstructured":"Nicola Bleu. 2022. 29 Top Chatbot Statistics For 2022: Usage Demographics Trends. https:\/\/bloggingwizard.com\/chatbot-statistics\/ Accessed: 2022-08-01"},{"key":"e_1_3_2_2_7_1","volume-title":"Learning End-to-End Goal-Oriented Dialog. ICLR, abs\/1605.07683","author":"Bordes Antoine","year":"2017","unstructured":"Antoine Bordes and Jason Weston. 2017. Learning End-to-End Goal-Oriented Dialog. ICLR, abs\/1605.07683 (2017)."},{"key":"e_1_3_2_2_8_1","volume-title":"Bowman","author":"Bordia Shikha","year":"2019","unstructured":"Shikha Bordia and Samuel R. Bowman. 2019. Identifying and Reducing Gender Bias in Word-Level Language Models. In North American Chapter of the Association for Computational Linguistics."},{"key":"e_1_3_2_2_9_1","unstructured":"Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger T. J. Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeff Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. NeurIPS."},{"key":"e_1_3_2_2_10_1","volume-title":"Hidden Voice Commands. In USENIX Security Symposium.","author":"Carlini Nicholas","year":"2016","unstructured":"Nicholas Carlini, Pratyush Mishra, Tavish Vaidya, Yuankai Zhang, Michael E. Sherr, Clay Shields, David A. Wagner, and Wenchao Zhou. 2016. Hidden Voice Commands. In USENIX Security Symposium."},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468537"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678670"},{"key":"e_1_3_2_2_13_1","unstructured":"Christopher Clark Kenton Lee Ming-Wei Chang Tom Kwiatkowski Michael Collins and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes\/No Questions. In North American Chapter of the Association for Computational Linguistics."},{"key":"e_1_3_2_2_14_1","unstructured":"David Curry. 2022. Apple Statistics. https:\/\/www.businessofapps.com\/data\/apple-statistics\/ Accessed: 2022-08-01"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445924"},{"key":"e_1_3_2_2_16_1","volume-title":"Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation. In Conference on Empirical Methods in Natural Language Processing.","author":"Dinan Emily","year":"2019","unstructured":"Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2019. Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation. In Conference on Empirical Methods in Natural Language Processing."},{"key":"e_1_3_2_2_17_1","volume-title":"Unified Language Model Pre-training for Natural Language Understanding and Generation. CoRR, abs\/1905.03197","author":"Dong Li","year":"2019","unstructured":"Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified Language Model Pre-training for Natural Language Understanding and Generation. CoRR, abs\/1905.03197 (2019), arXiv:1905.03197. arxiv:1905.03197"},{"key":"e_1_3_2_2_18_1","unstructured":"EF. [n. d.]. The comparative and the superlative. https:\/\/www.ef.edu\/english-resources\/english-grammar\/comparative-and-superlative\/"},{"key":"e_1_3_2_2_19_1","volume-title":"Le","author":"Freitas Daniel De","year":"2020","unstructured":"Daniel De Freitas, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. Towards a Human-like Open-Domain Chatbot. ArXiv, abs\/2001.09977 (2020)."},{"key":"e_1_3_2_2_20_1","volume-title":"Smith","author":"Gehman Samuel","year":"2020","unstructured":"Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. ArXiv, abs\/2009.11462 (2020)."},{"key":"e_1_3_2_2_21_1","volume-title":"2020 IEEE\/ACM 42nd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 107\u2013109","author":"Gupta Shashij","year":"2020","unstructured":"Shashij Gupta. 2020. Machine Translation Testing via Pathological Invariance. 2020 IEEE\/ACM 42nd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 107\u2013109."},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00047"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.3115\/992133.992154"},{"key":"e_1_3_2_2_24_1","unstructured":"Will Heaven. 2020. How to make a chatbot that isn\u2019t racist or sexist. https:\/\/thegoodai.co\/2020\/10\/24\/how-to-make-a-chatbot-that-isnt-racist-or-sexist\/ Accessed: 2022-08-01"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464825"},{"key":"e_1_3_2_2_26_1","volume-title":"Tencent\u2019s Multilingual Machine Translation System for WMT22 Large-Scale African Languages. In Conference on Machine Translation.","author":"Jiao Wenxiang","year":"2022","unstructured":"Wenxiang Jiao, Zhaopeng Tu, Jiarui Li, Wenxuan Wang, Jen tse Huang, and Shuming Shi. 2022. Tencent\u2019s Multilingual Machine Translation System for WMT22 Large-Scale African Languages. In Conference on Machine Translation."},{"key":"e_1_3_2_2_27_1","volume-title":"Xing Wang, and Zhaopeng Tu.","author":"Jiao Wenxiang","year":"2023","unstructured":"Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT A Good Translator? A Preliminary Study. ArXiv, abs\/2301.08745 (2023)."},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"crossref","unstructured":"Daniel Khashabi Snigdha Chaturvedi Michael Roth Shyam Upadhyay and Dan Roth. 2018. Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences. In North American Chapter of the Association for Computational Linguistics.","DOI":"10.18653\/v1\/N18-1023"},{"key":"e_1_3_2_2_29_1","volume-title":"Tesla fatal crash: \u2019autopilot","author":"Levin Sam","year":"2018","unstructured":"Sam Levin. 2018. Tesla fatal crash: \u2019autopilot\u2019 mode sped up car before driver killed, report finds [Online]. https:\/\/www.theguardian.com\/technology\/2018\/jun\/07\/tesla-fatal-crash-silicon-valley-autopilot-mode-report Accessed: 2018-06"},{"key":"e_1_3_2_2_30_1","volume-title":"ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain. 74\u201381. https:\/\/aclanthology.org\/W04-1013"},{"key":"e_1_3_2_2_31_1","volume-title":"Towards Fairness in Dialogue Systems. In International Conference on Computational Linguistics.","author":"Liu Haochen","year":"2020","unstructured":"Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2020. Does Gender Matter? Towards Fairness in Dialogue Systems. In International Conference on Computational Linguistics."},{"key":"e_1_3_2_2_32_1","volume-title":"Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learning. In Conference on Empirical Methods in Natural Language Processing.","author":"Liu Haochen","year":"2020","unstructured":"Haochen Liu, Wentao Wang, Yiqi Wang, Hui Liu, Zitao Liu, and Jiliang Tang. 2020. Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learning. In Conference on Empirical Methods in Natural Language Processing."},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556929"},{"key":"e_1_3_2_2_34_1","volume-title":"Interactive Planning for Autonomous Urban Driving in Adversarial Scenarios. 2021 IEEE International Conference on Robotics and Automation (ICRA), 5261\u20135267","author":"Luo Yuanfu","year":"2021","unstructured":"Yuanfu Luo, Malika Meghjani, Qi Heng Ho, David Hsu, and Daniela Rus. 2021. Interactive Planning for Autonomous Urban Driving in Adversarial Scenarios. 2021 IEEE International Conference on Robotics and Automation (ICRA), 5261\u20135267."},{"key":"e_1_3_2_2_35_1","unstructured":"Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards Deep Learning Models Resistant to Adversarial Attacks. ICLR."},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.416"},{"key":"e_1_3_2_2_37_1","volume-title":"Annual Meeting of the Association for Computational Linguistics.","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Annual Meeting of the Association for Computational Linguistics."},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_2_39_1","volume-title":"DEVIATE: A Deep Learning Variance Testing Framework. 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE), 1286\u20131290","author":"Pham Hung Viet","year":"2021","unstructured":"Hung Viet Pham, Mijung Kim, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. 2021. DEVIATE: A Deep Learning Variance Testing Framework. 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE), 1286\u20131290."},{"key":"e_1_3_2_2_40_1","unstructured":"Jack W. Rae Sebastian Borgeaud Trevor Cai Katie Millican Jordan Hoffmann Francis Song John Aslanides Sarah Henderson Roman Ring Susannah Young Eliza Rutherford Tom Hennigan Jacob Menick Albin Cassirer Richard Powell George van den Driessche Lisa Anne Hendricks Maribeth Rauh Po-Sen Huang Amelia Glaese Johannes Welbl Sumanth Dathathri Saffron Huang Jonathan Uesato John F. J. Mellor Irina Higgins Antonia Creswell Nathan McAleese Amy Wu Erich Elsen Siddhant M. Jayakumar Elena Buchatskaya David Budden Esme Sutherland Karen Simonyan Michela Paganini L. Sifre Lena Martens Xiang Lorraine Li Adhiguna Kuncoro Aida Nematzadeh Elena Gribovskaya Domenic Donato Angeliki Lazaridou Arthur Mensch Jean-Baptiste Lespiau Maria Tsimpoukelli N. K. Grigorev Doug Fritz Thibault Sottiaux Mantas Pajarskas Tobias Pohlen Zhitao Gong Daniel Toyama Cyprien de Masson d\u2019Autume Yujia Li Tayfun Terzi Vladimir Mikulik Igor Babuschkin Aidan Clark Diego de Las Casas Aurelia Guy Chris Jones James Bradbury Matthew G. Johnson Blake A. Hechtman Laura Weidinger Iason Gabriel William S. Isaac Edward Lockhart Simon Osindero Laura Rimell Chris Dyer Oriol Vinyals Kareem W. Ayoub Jeff Stanway L. L. Bennett Demis Hassabis Koray Kavukcuoglu and Geoffrey Irving. 2021. Scaling Language Models: Methods Analysis & Insights from Training Gopher. ArXiv abs\/2112.11446 (2021)."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1264"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"crossref","unstructured":"Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP.","DOI":"10.18653\/v1\/D19-1410"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09881-0"},{"key":"e_1_3_2_2_44_1","volume-title":"Y-Lan Boureau, and Jason Weston.","author":"Roller Stephen","year":"2020","unstructured":"Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2020. Recipes for building an open-domain chatbot. CoRR, abs\/2004.13637 (2020), arXiv:2004.13637. arxiv:2004.13637"},{"key":"e_1_3_2_2_45_1","volume-title":"Liubov Kovriguina, Debanjan Chaudhuri, Ricardo Usbeck, and Jens Lehmann.","year":"2022","unstructured":"Md. Rashad Al Hasan Rony, Liubov Kovriguina, Debanjan Chaudhuri, Ricardo Usbeck, and Jens Lehmann. 2022. RoMe: A Robust Metric for Evaluating Natural Language Generation. In Annual Meeting of the Association for Computational Linguistics."},{"key":"e_1_3_2_2_46_1","unstructured":"Maarten Sap Saadia Gabriel Lianhui Qin Dan Jurafsky Noah A. Smith and Yejin Choi. 2020. Social Bias Frames: Reasoning about Social and Power Implications of Language. ACL."},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556953"},{"key":"e_1_3_2_2_48_1","volume-title":"Revealing Persona Biases in Dialogue Systems. ArXiv, abs\/2104.08728","author":"Sheng Emily","year":"2021","unstructured":"Emily Sheng, Josh Arnold, Zhou Yu, Kai-Wei Chang, and Nanyun Peng. 2021. Revealing Persona Biases in Dialogue Systems. ArXiv, abs\/2104.08728 (2021)."},{"key":"e_1_3_2_2_49_1","volume-title":"Nice Try","author":"Sheng Emily","unstructured":"Emily Sheng, Kai-Wei Chang, P. Natarajan, and Nanyun Peng. 2021. \u201cNice Try, Kiddo\u201d: Investigating Ad Hominems in Dialogue Responses. In NAACL."},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3548606.3560599"},{"key":"e_1_3_2_2_51_1","volume-title":"Eleonora Presani, and Adina Williams.","author":"Smith Eric Michael","year":"2022","unstructured":"Eric Michael Smith, Melissa Hall Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. \"I\u2019m sorry to hear that\": finding bias in language models with a holistic descriptor dataset. ArXiv, abs\/2205.09209 (2022)."},{"key":"e_1_3_2_2_52_1","volume-title":"On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark. Findings of ACL, abs\/2110.08466","author":"Sun Hao","year":"2022","unstructured":"Hao Sun, Guangxuan Xu, Deng Jiawen, Jiale Cheng, Chujie Zheng, Hao Zhou, Nanyun Peng, Xiaoyan Zhu, and Minlie Huang. 2022. On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark. Findings of ACL, abs\/2110.08466 (2022)."},{"key":"e_1_3_2_2_53_1","unstructured":"Romal Thoppilan Daniel De Freitas Jamie Hall Noam M. Shazeer Apoorv Kulshreshtha Heng-Tze Cheng Alicia Jin Taylor Bos Leslie Baker Yu Du Yaguang Li Hongrae Lee Huaixiu Zheng Amin Ghafouri Marcelo Menegali Yanping Huang Maxim Krikun Dmitry Lepikhin James Qin Dehao Chen Yuanzhong Xu Zhifeng Chen Adam Roberts Maarten Bosma Yanqi Zhou Chung-Ching Chang I. A. Krivokon Willard James Rusch Marc Pickett Kathleen S. Meier-Hellstern Meredith Ringel Morris Tulsee Doshi Renelito Delos Santos Toju Duke Johnny Hartz S\u00f8raker Ben Zevenbergen Vinodkumar Prabhakaran Mark D\u00edaz Ben Hutchinson Kristen Olson Alejandra Molina Erin Hoffman-John Josh Lee Lora Aroyo Ravindran Rajakumar Alena Butryna Matthew Lamm V. O. Kuzmina Joseph Fenton Aaron Cohen Rachel Bernstein Ray Kurzweil Blaise Aguera-Arcas Claire Cui Marian Croak Ed Huai hsin Chi and Quoc Le. 2022. LaMDA: Language Models for Dialog Applications. ArXiv abs\/2201.08239 (2022)."},{"key":"e_1_3_2_2_54_1","volume-title":"Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis.","author":"Huang Jen","unstructured":"Jen tse Huang, Jianping Zhang, Wenxuan Wang, Pinjia He, Yuxin Su, and Michael R. Lyu. 2022. AEON: a method for automatic evaluation of NLP test cases. Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis."},{"key":"e_1_3_2_2_55_1","volume-title":"Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving. ArXiv, abs\/2101.06784","author":"Tu James","year":"2021","unstructured":"James Tu, Huichen Li, Xinchen Yan, Mengye Ren, Yun Chen, Ming Liang, Eilyan Bitar, Ersin Yumer, and Raquel Urtasun. 2021. Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving. ArXiv, abs\/2101.06784 (2021)."},{"key":"e_1_3_2_2_56_1","volume-title":"Automated Directed Fairness Testing. 2018 33rd IEEE\/ACM International Conference on Automated Software Engineering (ASE), 98\u2013108","author":"Udeshi Sakshi","year":"2018","unstructured":"Sakshi Udeshi, Pryanshu Arora, and Sudipta Chattopadhyay. 2018. Automated Directed Fairness Testing. 2018 33rd IEEE\/ACM International Conference on Automated Software Engineering (ASE), 98\u2013108."},{"key":"e_1_3_2_2_57_1","volume-title":"Learning to Speak and Act in a Fantasy Text Adventure Game. EMNLP, abs\/1903.03094","author":"Urbanek Jack","year":"2019","unstructured":"Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rockt\u00e4schel, Douwe Kiela, Arthur D. Szlam, and Jason Weston. 2019. Learning to Speak and Act in a Fantasy Text Adventure Game. EMNLP, abs\/1903.03094 (2019)."},{"key":"e_1_3_2_2_58_1","volume-title":"RobOT: Robustness-Oriented Testing for Deep Learning Systems. 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE), 300\u2013311","author":"Wang Jingyi","year":"2021","unstructured":"Jingyi Wang, Jialuo Chen, Youcheng Sun, Xingjun Ma, Dongxia Wang, Jun Sun, and Peng Cheng. 2021. RobOT: Robustness-Oriented Testing for Deep Learning Systems. 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE), 300\u2013311."},{"key":"e_1_3_2_2_59_1","volume-title":"Semantics-Enhanced Task-Oriented Dialogue Translation: A Case Study on Hotel Booking. In International Joint Conference on Natural Language Processing.","author":"Wang Longyue","year":"2017","unstructured":"Longyue Wang, Jinhua Du, Liangyou Li, Zhaopeng Tu, Andy Way, and Qun Liu. 2017. Semantics-Enhanced Task-Oriented Dialogue Translation: A Case Study on Hotel Booking. In International Joint Conference on Natural Language Processing."},{"key":"e_1_3_2_2_60_1","volume-title":"Weibin Wu, Jianping Zhang, Yizhan Huang, Shuqing Li, Pinjia He, and Michael R. Lyu.","author":"Wang Wenxuan","year":"2023","unstructured":"Wenxuan Wang, Jen tse Huang, Weibin Wu, Jianping Zhang, Yizhan Huang, Shuqing Li, Pinjia He, and Michael R. Lyu. 2023. MTTM: Metamorphic Testing for Textual Content Moderation Software. ArXiv, abs\/2302.05706 (2023)."},{"key":"e_1_3_2_2_61_1","unstructured":"Josh Wardini. 2022. Voice Search Statistics: Smart Speakers Voice Assistants and Users in 2022. https:\/\/serpwatch.io\/blog\/voice-search-statistics\/ Accessed: 2022-08-01"},{"key":"e_1_3_2_2_62_1","volume-title":"Courtney Anne De Thomas, and Jennifer M Weller","author":"Webster Craig S.","year":"2022","unstructured":"Craig S. Webster, S Taylor, Courtney Anne De Thomas, and Jennifer M Weller. 2022. Social bias, discrimination and inequity in healthcare: mechanisms, implications and recommendations.. BJA education."},{"key":"e_1_3_2_2_63_1","volume-title":"Lyu","author":"Wu Hao","year":"2023","unstructured":"Hao Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael R. Lyu. 2023. ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark. ArXiv, abs\/2303.13648 (2023)."},{"key":"e_1_3_2_2_64_1","unstructured":"Jing Xu Da Ju Margaret Li Y-Lan Boureau Jason Weston and Emily Dinan. 2021. Bot-Adversarial Dialogue for Safe Conversational Agents. In North American Chapter of the Association for Computational Linguistics."},{"key":"e_1_3_2_2_65_1","volume-title":"Building Task-Oriented Dialogue Systems for Online Shopping. In AAAI Conference on Artificial Intelligence.","author":"Yan Zhao","year":"2017","unstructured":"Zhao Yan, Nan Duan, Peng Chen, M. Zhou, Jianshe Zhou, and Zhoujun Li. 2017. Building Task-Oriented Dialogue Systems for Online Shopping. In AAAI Conference on Artificial Intelligence."},{"key":"e_1_3_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00129"},{"key":"e_1_3_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_2_2_68_1","volume-title":"Wenxuan Wang, Yichen Li, Weibin Wu, Xiaosen Wang, Yuxin Su, and Michael R. Lyu.","author":"Zhang Jianping","year":"2023","unstructured":"Jianping Zhang, Jen tse Huang, Wenxuan Wang, Yichen Li, Weibin Wu, Xiaosen Wang, Yuxin Su, and Michael R. Lyu. 2023. Improving the Transferability of Adversarial Samples by Path-Augmented Method. ArXiv, abs\/2303.15735 (2023)."},{"key":"e_1_3_2_2_69_1","volume-title":"2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14973\u201314982","author":"Zhang Jianping","unstructured":"Jianping Zhang, Weibin Wu, Jen tse Huang, Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael R. Lyu. 2022. Improving Adversarial Transferability via Neuron Attribution-based Attacks. 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14973\u201314982."},{"key":"e_1_3_2_2_70_1","volume-title":"Dolan","author":"Zhang Yizhe","year":"2019","unstructured":"Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B. Dolan. 2019. DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation. In Annual Meeting of the Association for Computational Linguistics."},{"key":"e_1_3_2_2_71_1","volume-title":"Hao Sun, Xiaocong Yang, Bosi Wen, Xiaoyan Zhu, Minlie Huang, and Jie Tang.","author":"Zhou Hao","year":"2021","unstructured":"Hao Zhou, Pei Ke, Zheng Zhang, Yuxian Gu, Yinhe Zheng, Chujie Zheng, Yida Wang, Chen Henry Wu, Hao Sun, Xiaocong Yang, Bosi Wen, Xiaoyan Zhu, Minlie Huang, and Jie Tang. 2021. EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Training. ArXiv, abs\/2108.01547 (2021)."},{"key":"e_1_3_2_2_72_1","unstructured":"Chris Ziegler. 2016. A google self-driving car caused a crash for the first time. [Online]. https:\/\/www.theverge.com\/2016\/2\/29\/11134344\/google-self-driving-car-crash-report Accessed: 2016-09"},{"key":"e_1_3_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13278-020-00665-4"}],"event":{"name":"ESEC\/FSE '23: 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"San Francisco CA USA","acronym":"ESEC\/FSE '23","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3611643.3616310","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3611643.3616310","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:36:04Z","timestamp":1750178164000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3611643.3616310"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,30]]},"references-count":73,"alternative-id":["10.1145\/3611643.3616310","10.1145\/3611643"],"URL":"https:\/\/doi.org\/10.1145\/3611643.3616310","relation":{},"subject":[],"published":{"date-parts":[[2023,11,30]]},"assertion":[{"value":"2023-11-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}