{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T14:50:57Z","timestamp":1776783057374,"version":"3.51.2"},"publisher-location":"New York, NY, USA","reference-count":93,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,7,18]],"date-time":"2022-07-18T00:00:00Z","timestamp":1658102400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,7,18]]},"DOI":"10.1145\/3533767.3534389","type":"proceedings-article","created":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T14:28:50Z","timestamp":1657895330000},"page":"467-479","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":24,"title":["Automated testing of image captioning systems"],"prefix":"10.1145","author":[{"given":"Boxi","family":"Yu","sequence":"first","affiliation":[{"name":"Chinese University of Hong Kong at Shenzhen, China"}]},{"given":"Zhiqing","family":"Zhong","sequence":"additional","affiliation":[{"name":"South China University of Technology, China"}]},{"given":"Xinran","family":"Qin","sequence":"additional","affiliation":[{"name":"South China University of Technology, China"}]},{"given":"Jiayi","family":"Yao","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong at Shenzhen, China"}]},{"given":"Yuancheng","family":"Wang","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong at Shenzhen, China"}]},{"given":"Pinjia","family":"He","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong at Shenzhen, China"}]}],"member":"320","published-online":{"date-parts":[[2022,7,18]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"2018. Image Caption Generator. https:\/\/developer.ibm.com\/exchanges\/models\/all\/max-image-caption-generator \t\t\t\t\t  2018. Image Caption Generator. https:\/\/developer.ibm.com\/exchanges\/models\/all\/max-image-caption-generator"},{"key":"e_1_3_2_2_2_1","unstructured":"2020. Auto Image Captioning. https:\/\/medium.com\/ai-techsystems\/auto-image-captioning-8efcfa517402 \t\t\t\t\t  2020. Auto Image Captioning. https:\/\/medium.com\/ai-techsystems\/auto-image-captioning-8efcfa517402"},{"key":"e_1_3_2_2_3_1","unstructured":"2020. Automatic Image Captioning Using Neural Networks. https:\/\/evergreen.team\/articles\/automatic-image-captioning.html \t\t\t\t\t  2020. Automatic Image Captioning Using Neural Networks. https:\/\/evergreen.team\/articles\/automatic-image-captioning.html"},{"key":"e_1_3_2_2_4_1","unstructured":"2021. Automated Image Captions and Descriptions. https:\/\/cloud.google.com\/ai-workshop\/experiments\/automated-image-captions-and-descriptions \t\t\t\t\t  2021. Automated Image Captions and Descriptions. https:\/\/cloud.google.com\/ai-workshop\/experiments\/automated-image-captions-and-descriptions"},{"key":"e_1_3_2_2_5_1","unstructured":"2021. Azure Cognitive Services. https:\/\/azure.microsoft.com\/en-us\/services\/cognitive-services \t\t\t\t\t  2021. Azure Cognitive Services. https:\/\/azure.microsoft.com\/en-us\/services\/cognitive-services"},{"key":"e_1_3_2_2_6_1","unstructured":"2021. Flickr: Find your inspiration. https:\/\/www.flickr.com\/ \t\t\t\t\t  2021. Flickr: Find your inspiration. https:\/\/www.flickr.com\/"},{"key":"e_1_3_2_2_7_1","unstructured":"2022. MetaIC: An Automated Testing Toolkit for Image Captioning. https:\/\/github.com\/RobustNLP\/TestIC \t\t\t\t\t  2022. MetaIC: An Automated Testing Toolkit for Image Captioning. https:\/\/github.com\/RobustNLP\/TestIC"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00904"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46454-1_24"},{"key":"e_1_3_2_2_10_1","unstructured":"Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. \t\t\t\t\t  Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473."},{"key":"e_1_3_2_2_11_1","volume-title":"Multimodal machine learning: A survey and taxonomy","author":"Baltru\u0161aitis Tadas","year":"2018","unstructured":"Tadas Baltru\u0161aitis , Chaitanya Ahuja , and Louis-Philippe Morency . 2018. Multimodal machine learning: A survey and taxonomy . IEEE transactions on pattern analysis and machine intelligence (TPAMI) , 41, 2 ( 2018 ), 423\u2013443. Tadas Baltru\u0161aitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence (TPAMI), 41, 2 (2018), 423\u2013443."},{"key":"e_1_3_2_2_12_1","article-title":"YOLACT++: Better Real-time Instance Segmentation","author":"Bolya D","year":"2020","unstructured":"D Bolya , C Zhou , F Xiao , and YJ Lee . 2020 . YOLACT++: Better Real-time Instance Segmentation .. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). D Bolya, C Zhou, F Xiao, and YJ Lee. 2020. YOLACT++: Better Real-time Instance Segmentation.. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)."},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00925"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/SPW.2018.00009"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9248-x"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2005.67"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.4018\/jwsr.2007040103"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1241"},{"key":"e_1_3_2_2_19_1","unstructured":"Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543. \t\t\t\t\t  Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543."},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3143561","article-title":"Metamorphic testing: A review of challenges and opportunities","volume":"51","author":"Chen Tsong Yueh","year":"2018","unstructured":"Tsong Yueh Chen , Fei-Ching Kuo , Huai Liu , Pak-Lok Poon , Dave Towey , TH Tse , and Zhi Quan Zhou . 2018 . Metamorphic testing: A review of challenges and opportunities . ACM Computing Surveys (CSUR) , 51 , 1 (2018), 1 \u2013 27 . Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, TH Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys (CSUR), 51, 1 (2018), 1\u201327.","journal-title":"ACM Computing Surveys (CSUR)"},{"key":"e_1_3_2_2_21_1","unstructured":"Xinlei Chen Hao Fang Tsung-Yi Lin Ramakrishna Vedantam Saurabh Gupta Piotr Doll\u00e1r and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. \t\t\t\t\t  Xinlei Chen Hao Fang Tsung-Yi Lin Ramakrishna Vedantam Saurabh Gupta Piotr Doll\u00e1r and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325."},{"key":"e_1_3_2_2_22_1","volume-title":"Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.","author":"Cho Kyunghyun","year":"2014","unstructured":"Kyunghyun Cho , Bart Van Merri\u00ebnboer , Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 . Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. Kyunghyun Cho, Bart Van Merri\u00ebnboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2012.2211477"},{"key":"e_1_3_2_2_24_1","volume-title":"Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2018 . Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338954"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330566"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401430"},{"key":"e_1_3_2_2_28_1","unstructured":"Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. \t\t\t\t\t  Ian J Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572."},{"key":"e_1_3_2_2_29_1","unstructured":"Shuangchi Gu Ping Yi Ting Zhu Yao Yao and Wei Wang. 2019. Detecting adversarial examples in deep neural networks using normalizing filters. UMBC Student Collection. \t\t\t\t\t  Shuangchi Gu Ping Yi Ting Zhu Yao Yao and Wei Wang. 2019. Detecting adversarial examples in deep neural networks using normalizing filters. UMBC Student Collection."},{"key":"e_1_3_2_2_30_1","unstructured":"Awni Hannun Carl Case Jared Casper Bryan Catanzaro Greg Diamos Erich Elsen Ryan Prenger Sanjeev Satheesh Shubho Sengupta and Adam Coates. 2014. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567. \t\t\t\t\t  Awni Hannun Carl Case Jared Casper Bryan Catanzaro Greg Diamos Erich Elsen Ryan Prenger Sanjeev Satheesh Shubho Sengupta and Adam Coates. 2014. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567."},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.322"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380339"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00047"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/AITest.2019.00-12"},{"key":"e_1_3_2_2_36_1","volume-title":"Long short-term memory. Neural computation, 9, 8","author":"Hochreiter Sepp","year":"1997","unstructured":"Sepp Hochreiter and J\u00fcrgen Schmidhuber . 1997. Long short-term memory. Neural computation, 9, 8 ( 1997 ), 1735\u20131780. Sepp Hochreiter and J\u00fcrgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9, 8 (1997), 1735\u20131780."},{"key":"e_1_3_2_2_37_1","volume-title":"Vivo: Surpassing human performance in novel object captioning with visual vocabulary pre-training. arXiv preprint arXiv:2009.13682.","author":"Hu Xiaowei","year":"2020","unstructured":"Xiaowei Hu , Xi Yin , Kevin Lin , Lijuan Wang , Lei Zhang , Jianfeng Gao , and Zicheng Liu . 2020 . Vivo: Surpassing human performance in novel object captioning with visual vocabulary pre-training. arXiv preprint arXiv:2009.13682. Xiaowei Hu, Xi Yin, Kevin Lin, Lijuan Wang, Lei Zhang, Jianfeng Gao, and Zicheng Liu. 2020. Vivo: Surpassing human performance in novel object captioning with visual vocabulary pre-training. arXiv preprint arXiv:2009.13682."},{"key":"e_1_3_2_2_38_1","volume-title":"International Conference on Machine Learning (ICML). 2410\u20132419","author":"Kalchbrenner Nal","year":"2018","unstructured":"Nal Kalchbrenner , Erich Elsen , Karen Simonyan , Seb Noury , Norman Casagrande , Edward Lockhart , Florian Stimberg , Aaron Oord , Sander Dieleman , and Koray Kavukcuoglu . 2018 . Efficient neural audio synthesis . In International Conference on Machine Learning (ICML). 2410\u20132419 . Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. 2018. Efficient neural audio synthesis. In International Conference on Machine Learning (ICML). 2410\u20132419."},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3387940.3391456"},{"key":"e_1_3_2_2_40_1","unstructured":"Harini Kannan Alexey Kurakin and Ian Goodfellow. 2018. Adversarial logit pairing. arXiv preprint arXiv:1803.06373. \t\t\t\t\t  Harini Kannan Alexey Kurakin and Ian Goodfellow. 2018. Adversarial logit pairing. arXiv preprint arXiv:1803.06373."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298932"},{"key":"e_1_3_2_2_42_1","unstructured":"Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images. \t\t\t\t\t  Alex Krizhevsky and Geoffrey Hinton. 2009. Learning multiple layers of features from tiny images."},{"key":"e_1_3_2_2_43_1","volume-title":"Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems (NIPS), 25","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky , Ilya Sutskever , and Geoffrey E Hinton . 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems (NIPS), 25 ( 2012 ), 1097\u20131105. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems (NIPS), 25 (2012), 1097\u20131105."},{"key":"e_1_3_2_2_44_1","unstructured":"Fred. Lambert. 2016. Understanding the fatal Tesla accident on Autopilot and the NHTSA probe.. https:\/\/electrek.co\/2016\/07\/01\/understanding-fatal-tesla-accident-autopilot-nhtsa-probe\/ \t\t\t\t\t  Fred. Lambert. 2016. Understanding the fatal Tesla accident on Autopilot and the NHTSA probe.. https:\/\/electrek.co\/2016\/07\/01\/understanding-fatal-tesla-accident-autopilot-nhtsa-probe\/"},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2666356.2594334"},{"key":"e_1_3_2_2_46_1","volume-title":"Tesla fatal crash: \u2019autopilot","author":"Levin Sam","year":"2018","unstructured":"Sam Levin . 2018. Tesla fatal crash: \u2019autopilot \u2019 mode sped up car before driver killed, report finds.. https:\/\/www.theguardian.com\/technology\/ 2018 \/jun\/07\/tesla-fatal-crash-silicon-valley-autopilot-mode-report Sam Levin. 2018. Tesla fatal crash: \u2019autopilot\u2019 mode sped up car before driver killed, report finds.. https:\/\/www.theguardian.com\/technology\/2018\/jun\/07\/tesla-fatal-crash-silicon-valley-autopilot-mode-report"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58577-8_8"},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.472"},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2813885.2737986"},{"key":"e_1_3_2_2_50_1","unstructured":"Ji Lin Chuang Gan and Song Han. 2019. Defensive quantization: When efficiency meets robustness. arXiv preprint arXiv:1904.08444. \t\t\t\t\t  Ji Lin Chuang Gan and Song Han. 2019. Defensive quantization: When efficiency meets robustness. arXiv preprint arXiv:1904.08444."},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"crossref","unstructured":"Bo Luo Yannan Liu Lingxiao Wei and Qiang Xu. 2018. Towards imperceptible and robust adversarial example attacks against neural networks. In Thirty-second aaai conference on artificial intelligence (AAAI). \t\t\t\t\t  Bo Luo Yannan Liu Lingxiao Wei and Qiang Xu. 2018. Towards imperceptible and robust adversarial example attacks against neural networks. In Thirty-second aaai conference on artificial intelligence (AAAI).","DOI":"10.1609\/aaai.v32i1.11499"},{"key":"e_1_3_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2019.23415"},{"key":"e_1_3_2_2_54_1","unstructured":"Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083. \t\t\t\t\t  Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083."},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468573"},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"crossref","unstructured":"Mitchell Marcus Beatrice Santorini and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. \t\t\t\t\t  Mitchell Marcus Beatrice Santorini and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank.","DOI":"10.21236\/ADA273556"},{"key":"e_1_3_2_2_57_1","volume-title":"Interrater reliability: the kappa statistic. Biochemia medica, 22, 3","author":"McHugh Mary L","year":"2012","unstructured":"Mary L McHugh . 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22, 3 ( 2012 ), 276\u2013282. Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22, 3 (2012), 276\u2013282."},{"key":"e_1_3_2_2_58_1","unstructured":"Youssef Mroueh. 2020. Image Captioning as an Assistive Technology. https:\/\/www.ibm.com\/blogs\/research\/2020\/07\/image-captioning-assistive-technology \t\t\t\t\t  Youssef Mroueh. 2020. Image Captioning as an Assistive Technology. https:\/\/www.ibm.com\/blogs\/research\/2020\/07\/image-captioning-assistive-technology"},{"key":"e_1_3_2_2_59_1","unstructured":"CAROL VAN NATTA. 2020. AI Fails at Photo Captions. https:\/\/author.carolvannatta.com\/ai-fails-at-photo-captions\/ \t\t\t\t\t  CAROL VAN NATTA. 2020. AI Fails at Photo Captions. https:\/\/author.carolvannatta.com\/ai-fails-at-photo-captions\/"},{"key":"e_1_3_2_2_60_1","unstructured":"World Health Organization. 2019. World report on vision. ISBN: 9241516577 Publisher: World Health Organization. \t\t\t\t\t  World Health Organization. 2019. World report on vision. ISBN: 9241516577 Publisher: World Health Organization."},{"key":"e_1_3_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2016.41"},{"key":"e_1_3_2_2_62_1","volume-title":"International conference on machine learning. 5231\u20135240","author":"Qin Yao","year":"2019","unstructured":"Yao Qin , Nicholas Carlini , Garrison Cottrell , Ian Goodfellow , and Colin Raffel . 2019 . Imperceptible, robust, and targeted adversarial examples for automatic speech recognition . In International conference on machine learning. 5231\u20135240 . Yao Qin, Nicholas Carlini, Garrison Cottrell, Ian Goodfellow, and Colin Raffel. 2019. Imperceptible, robust, and targeted adversarial examples for automatic speech recognition. In International conference on machine learning. 5231\u20135240."},{"key":"e_1_3_2_2_63_1","volume-title":"Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems (NIPS), 28","author":"Ren Shaoqing","year":"2015","unstructured":"Shaoqing Ren , Kaiming He , Ross Girshick , and Jian Sun . 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems (NIPS), 28 ( 2015 ), 91\u201399. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems (NIPS), 28 (2015), 91\u201399."},{"key":"e_1_3_2_2_64_1","volume-title":"COSMO: Code Coverage Made Easier for Android. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). 417\u2013423","author":"Romdhana Andrea","year":"2021","unstructured":"Andrea Romdhana , Mariano Ceccato , Gabriel Claudiu Georgiu , Alessio Merlo , and Paolo Tonella . 2021 . COSMO: Code Coverage Made Easier for Android. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). 417\u2013423 . Andrea Romdhana, Mariano Ceccato, Gabriel Claudiu Georgiu, Alessio Merlo, and Paolo Tonella. 2021. COSMO: Code Coverage Made Easier for Android. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). 417\u2013423."},{"key":"e_1_3_2_2_65_1","volume-title":"Data Cascades in High-Stakes AI. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI). 1\u201315","author":"Sambasivan Nithya","year":"2021","unstructured":"Nithya Sambasivan , Shivani Kapania , Hannah Highfill , Diana Akrong , Praveen Paritosh , and Lora M Aroyo . 2021 . \u201c Everyone wants to do the model work, not the data work \u201d: Data Cascades in High-Stakes AI. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI). 1\u201315 . Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 2021. \u201cEveryone wants to do the model work, not the data work\u201d: Data Cascades in High-Stakes AI. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI). 1\u201315."},{"key":"e_1_3_2_2_66_1","unstructured":"Stan Schroeder. 2016. Microsoft created a bot to auto-caption photos and it\u2019s going hilariously wrong. https:\/\/mashable.com\/article\/microsoft-captionbot Section: Life. \t\t\t\t\t  Stan Schroeder. 2016. Microsoft created a bot to auto-caption photos and it\u2019s going hilariously wrong. https:\/\/mashable.com\/article\/microsoft-captionbot Section: Life."},{"key":"e_1_3_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_2_68_1","volume-title":"Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229.","author":"Sermanet Pierre","year":"2013","unstructured":"Pierre Sermanet , David Eigen , Xiang Zhang , Micha\u00ebl Mathieu , Rob Fergus , and Yann LeCun . 2013 . Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229. Pierre Sermanet, David Eigen, Xiang Zhang, Micha\u00ebl Mathieu, Rob Fergus, and Yann LeCun. 2013. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229."},{"key":"e_1_3_2_2_69_1","unstructured":"Matteo Stefanini Marcella Cornia Lorenzo Baraldi Silvia Cascianelli Giuseppe Fiameni and Rita Cucchiara. 2021. From show to tell: A survey on image captioning. arXiv preprint arXiv:2107.06912. \t\t\t\t\t  Matteo Stefanini Marcella Cornia Lorenzo Baraldi Silvia Cascianelli Giuseppe Fiameni and Rita Cucchiara. 2021. From show to tell: A survey on image captioning. arXiv preprint arXiv:2107.06912."},{"key":"e_1_3_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_2_2_71_1","unstructured":"Ilya Sutskever Oriol Vinyals and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems (NIPS). 3104\u20133112. \t\t\t\t\t  Ilya Sutskever Oriol Vinyals and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems (NIPS). 3104\u20133112."},{"key":"e_1_3_2_2_72_1","volume-title":"Attacks meet interpretability: Attribute-steered detection of adversarial samples. arXiv preprint arXiv:1810","author":"Tao Guanhong","year":"2018","unstructured":"Guanhong Tao , Shiqing Ma , Yingqi Liu , and Xiangyu Zhang . 2018 . Attacks meet interpretability: Attribute-steered detection of adversarial samples. arXiv preprint arXiv:1810 .11580. Guanhong Tao, Shiqing Ma, Yingqi Liu, and Xiangyu Zhang. 2018. Attacks meet interpretability: Attribute-steered detection of adversarial samples. arXiv preprint arXiv:1810.11580."},{"key":"e_1_3_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_2_74_1","volume-title":"Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering (ICSE). 1122\u20131134","author":"Tian Yuchi","year":"2020","unstructured":"Yuchi Tian , Ziyuan Zhong , Vicente Ordonez , Gail Kaiser , and Baishakhi Ray . 2020 . Testing DNN image classifiers for confusion & bias errors . In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering (ICSE). 1122\u20131134 . Yuchi Tian, Ziyuan Zhong, Vicente Ordonez, Gail Kaiser, and Baishakhi Ray. 2020. Testing DNN image classifiers for confusion & bias errors. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering (ICSE). 1122\u20131134."},{"key":"e_1_3_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2018.8500621"},{"key":"e_1_3_2_2_76_1","volume-title":"\u0141 ukasz Kaiser, and Illia Polosukhin","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N Gomez , \u0141 ukasz Kaiser, and Illia Polosukhin . 2017 . Attention is all you need. In Advances in neural information processing systems (NIPS) . 5998\u20136008. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141 ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems (NIPS). 5998\u20136008."},{"key":"e_1_3_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"e_1_3_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298935"},{"key":"e_1_3_2_2_79_1","volume-title":"Assessment of source code obfuscation techniques. In 2016 IEEE 16th international working conference on source code analysis and manipulation (SCAM). 11\u201320","author":"Viticchi\u00e9 Alessio","unstructured":"Alessio Viticchi\u00e9 , Leonardo Regano , Marco Torchiano , Cataldo Basile , Mariano Ceccato , Paolo Tonella , and Roberto Tiella . 2016. Assessment of source code obfuscation techniques. In 2016 IEEE 16th international working conference on source code analysis and manipulation (SCAM). 11\u201320 . Alessio Viticchi\u00e9, Leonardo Regano, Marco Torchiano, Cataldo Basile, Mariano Ceccato, Paolo Tonella, and Roberto Tiella. 2016. Assessment of source code obfuscation techniques. In 2016 IEEE 16th international working conference on source code analysis and manipulation (SCAM). 11\u201320."},{"key":"e_1_3_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00660"},{"key":"e_1_3_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00126"},{"key":"e_1_3_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416584"},{"key":"e_1_3_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2009.26"},{"key":"e_1_3_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.11.920"},{"key":"e_1_3_2_2_85_1","volume-title":"International conference on machine learning (ICML). PMLR","author":"Xu Kelvin","year":"2015","unstructured":"Kelvin Xu , Jimmy Ba , Ryan Kiros , Kyunghyun Cho , Aaron Courville , Ruslan Salakhudinov , Rich Zemel , and Yoshua Bengio . 2015 . Show, attend and tell: Neural image caption generation with visual attention . In International conference on machine learning (ICML). PMLR , 2048\u20132057. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning (ICML). PMLR, 2048\u20132057."},{"key":"e_1_3_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00520"},{"key":"e_1_3_2_2_87_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00426"},{"key":"e_1_3_2_2_88_1","volume-title":"Improving crowdsourced label quality using noise correction","author":"Zhang Jing","year":"2017","unstructured":"Jing Zhang , Victor S Sheng , Tao Li , and Xindong Wu. 2017. Improving crowdsourced label quality using noise correction . IEEE transactions on neural networks and learning systems (TNNLS) , 29, 5 ( 2017 ), 1675\u20131688. Jing Zhang, Victor S Sheng, Tao Li, and Xindong Wu. 2017. Improving crowdsourced label quality using noise correction. IEEE transactions on neural networks and learning systems (TNNLS), 29, 5 (2017), 1675\u20131688."},{"key":"e_1_3_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_2_2_90_1","doi-asserted-by":"crossref","unstructured":"Pengchuan Zhang Xiujun Li Xiaowei Hu Jianwei Yang Lei Zhang Lijuan Wang Yejin Choi and Jianfeng Gao. 2021. VinVL: Making Visual Representations Matter in Vision-Language Models. arXiv preprint arXiv:2101.00529. \t\t\t\t\t  Pengchuan Zhang Xiujun Li Xiaowei Hu Jianwei Yang Lei Zhang Lijuan Wang Yejin Choi and Jianfeng Gao. 2021. VinVL: Making Visual Representations Matter in Vision-Language Models. arXiv preprint arXiv:2101.00529.","DOI":"10.1109\/CVPR46437.2021.00553"},{"key":"e_1_3_2_2_91_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00112"},{"key":"e_1_3_2_2_92_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.7005"},{"key":"e_1_3_2_2_93_1","unstructured":"Chris. Ziegler. 2016. A Google self-driving car caused a crash for the first time.. https:\/\/www.theverge.com\/2016\/2\/29\/11134344\/google-self-driving-car-crash-report \t\t\t\t\t  Chris. Ziegler. 2016. A Google self-driving car caused a crash for the first time.. https:\/\/www.theverge.com\/2016\/2\/29\/11134344\/google-self-driving-car-crash-report"}],"event":{"name":"ISSTA '22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis","location":"Virtual South Korea","acronym":"ISSTA '22","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533767.3534389","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3533767.3534389","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T18:43:41Z","timestamp":1750272221000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533767.3534389"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,18]]},"references-count":93,"alternative-id":["10.1145\/3533767.3534389","10.1145\/3533767"],"URL":"https:\/\/doi.org\/10.1145\/3533767.3534389","relation":{},"subject":[],"published":{"date-parts":[[2022,7,18]]},"assertion":[{"value":"2022-07-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}