{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,11]],"date-time":"2025-12-11T21:02:12Z","timestamp":1765486932683,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":85,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T00:00:00Z","timestamp":1689120000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,7,12]]},"DOI":"10.1145\/3597926.3598094","type":"proceedings-article","created":{"date-parts":[[2023,7,13]],"date-time":"2023-07-13T20:12:53Z","timestamp":1689279173000},"page":"766-778","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["ROME: Testing Image Captioning Systems via Recursive Object Melting"],"prefix":"10.1145","author":[{"given":"Boxi","family":"Yu","sequence":"first","affiliation":[{"name":"Chinese University of Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhiqing","family":"Zhong","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiaqi","family":"Li","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yixing","family":"Yang","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shilin","family":"He","sequence":"additional","affiliation":[{"name":"Microsoft Research, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pinjia","family":"He","sequence":"additional","affiliation":[{"name":"Chinese University of Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,7,13]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2020. Auto Image Captioning. https:\/\/medium.com\/ai-techsystems\/auto-image-captioning-8efcfa517402 \t\t\t\t  2020. Auto Image Captioning. https:\/\/medium.com\/ai-techsystems\/auto-image-captioning-8efcfa517402"},{"key":"e_1_3_2_1_2_1","unstructured":"2021. Azure Cognitive Services. https:\/\/docs.microsoft.com\/en-us\/azure\/cognitive-services\/computer-vision\/concept-describing-images \t\t\t\t  2021. Azure Cognitive Services. https:\/\/docs.microsoft.com\/en-us\/azure\/cognitive-services\/computer-vision\/concept-describing-images"},{"key":"e_1_3_2_1_3_1","unstructured":"2021. Flickr: Find your inspiration. https:\/\/www.flickr.com\/ \t\t\t\t  2021. Flickr: Find your inspiration. https:\/\/www.flickr.com\/"},{"key":"e_1_3_2_1_4_1","unstructured":"2022. Alternative text for Facebook. https:\/\/www.facebook.com\/help\/216219865403298 \t\t\t\t  2022. Alternative text for Facebook. https:\/\/www.facebook.com\/help\/216219865403298"},{"key":"e_1_3_2_1_5_1","unstructured":"2022. Alternative text for Microsoft Edge Explorer. https:\/\/microsoftedge.microsoft.com\/addons\/detail\/alt-text-display\/ifckdcbddldhagbihbhhijioebgmnlfl?hl=en-GBe \t\t\t\t  2022. Alternative text for Microsoft Edge Explorer. https:\/\/microsoftedge.microsoft.com\/addons\/detail\/alt-text-display\/ifckdcbddldhagbihbhhijioebgmnlfl?hl=en-GBe"},{"key":"e_1_3_2_1_6_1","unstructured":"2022. Alternative text for Microsoft Office 365. https:\/\/support.microsoft.com\/en-us\/office\/ \t\t\t\t  2022. Alternative text for Microsoft Office 365. https:\/\/support.microsoft.com\/en-us\/office\/"},{"key":"e_1_3_2_1_7_1","unstructured":"2022. ArcGIS API for Python. https:\/\/developers.arcgis.com\/python\/guide\/how-image-captioning-works\/ \t\t\t\t  2022. ArcGIS API for Python. https:\/\/developers.arcgis.com\/python\/guide\/how-image-captioning-works\/"},{"key":"e_1_3_2_1_8_1","unstructured":"2023. Image Captioning for the Visually Impaired and Blind People. https:\/\/www.hackster.io\/shahizat\/image-captioning-for-the-visually-impaired-and-blind-people-505c59 \t\t\t\t  2023. Image Captioning for the Visually Impaired and Blind People. https:\/\/www.hackster.io\/shahizat\/image-captioning-for-the-visually-impaired-and-blind-people-505c59"},{"key":"e_1_3_2_1_9_1","unstructured":"2023. TestIC: An Automated Testing Toolkit for Image Captioning. https:\/\/github.com\/RobustNLP\/TestIC \t\t\t\t  2023. TestIC: An Automated Testing Toolkit for Image Captioning. https:\/\/github.com\/RobustNLP\/TestIC"},{"key":"e_1_3_2_1_10_1","volume-title":"Proceedings of the 48th annual meeting of the association for computational linguistics (ACL). 1250\u20131258","author":"Aker Ahmet","year":"2010","unstructured":"Ahmet Aker and Robert Gaizauskas . 2010 . Generating image descriptions using dependency relational patterns . In Proceedings of the 48th annual meeting of the association for computational linguistics (ACL). 1250\u20131258 . Ahmet Aker and Robert Gaizauskas. 2010. Generating image descriptions using dependency relational patterns. In Proceedings of the 48th annual meeting of the association for computational linguistics (ACL). 1250\u20131258."},{"key":"e_1_3_2_1_11_1","volume-title":"International conference on machine learning. 274\u2013283","author":"Athalye Anish","year":"2018","unstructured":"Anish Athalye , Nicholas Carlini , and David Wagner . 2018 . Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples . In International conference on machine learning. 274\u2013283 . Anish Athalye, Nicholas Carlini, and David Wagner. 2018. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning. 274\u2013283."},{"key":"e_1_3_2_1_12_1","volume-title":"Robust object tracking with online multiple instance learning","author":"Babenko Boris","year":"2010","unstructured":"Boris Babenko , Ming-Hsuan Yang , and Serge Belongie . 2010. Robust object tracking with online multiple instance learning . IEEE transactions on pattern analysis and machine intelligence, 33, 8 ( 2010 ), 1619\u20131632. Boris Babenko, Ming-Hsuan Yang, and Serge Belongie. 2010. Robust object tracking with online multiple instance learning. IEEE transactions on pattern analysis and machine intelligence, 33, 8 (2010), 1619\u20131632."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510099"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3490488"},{"volume-title":"25th USENIX security symposium (USENIX security 16). 513\u2013530.","author":"Carlini Nicholas","key":"e_1_3_2_1_15_1","unstructured":"Nicholas Carlini , Pratyush Mishra , Tavish Vaidya , Yuankai Zhang , Micah Sherr , Clay Shields , David Wagner , and Wenchao Zhou . 2016. Hidden voice commands . In 25th USENIX security symposium (USENIX security 16). 513\u2013530. Nicholas Carlini, Pratyush Mishra, Tavish Vaidya, Yuankai Zhang, Micah Sherr, Clay Shields, David Wagner, and Wenchao Zhou. 2016. Hidden voice commands. In 25th USENIX security symposium (USENIX security 16). 513\u2013530."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"crossref","unstructured":"Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). 39\u201357. \t\t\t\t  Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). 39\u201357.","DOI":"10.1109\/SP.2017.49"},{"key":"e_1_3_2_1_17_1","unstructured":"Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543. \t\t\t\t  Tsong Y Chen Shing C Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: a new approach for generating next test cases. arXiv preprint arXiv:2002.12543."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3143561","article-title":"Metamorphic testing: A review of challenges and opportunities","volume":"51","author":"Chen Tsong Yueh","year":"2018","unstructured":"Tsong Yueh Chen , Fei-Ching Kuo , Huai Liu , Pak-Lok Poon , Dave Towey , TH Tse , and Zhi Quan Zhou . 2018 . Metamorphic testing: A review of challenges and opportunities . ACM Computing Surveys (CSUR) , 51 , 1 (2018), 1 \u2013 27 . Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, TH Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. ACM Computing Surveys (CSUR), 51, 1 (2018), 1\u201327.","journal-title":"ACM Computing Surveys (CSUR)"},{"key":"e_1_3_2_1_19_1","unstructured":"Xinlei Chen Hao Fang Tsung-Yi Lin Ramakrishna Vedantam Saurabh Gupta Piotr Doll\u00e1r and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. \t\t\t\t  Xinlei Chen Hao Fang Tsung-Yi Lin Ramakrishna Vedantam Saurabh Gupta Piotr Doll\u00e1r and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58577-8_7"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549093"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Yong Cheng Zhaopeng Tu Fandong Meng Junjie Zhai and Yang Liu. 2018. Towards robust neural machine translation. arXiv preprint arXiv:1805.06130. \t\t\t\t  Yong Cheng Zhaopeng Tu Fandong Meng Junjie Zhai and Yang Liu. 2018. Towards robust neural machine translation. arXiv preprint arXiv:1805.06130.","DOI":"10.18653\/v1\/P18-1163"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2012.2211477"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298878"},{"key":"e_1_3_2_1_25_1","volume-title":"Rachel Lerman.","author":"Jeremy","year":"2022","unstructured":"Jeremy B. Merrill Faiz Siddiqui , Rachel Lerman. 2022 . https:\/\/www.washingtonpost.com\/technology\/2022\/06\/15\/tesla-autopilot-crashes\/ Jeremy B. Merrill Faiz Siddiqui, Rachel Lerman. 2022. https:\/\/www.washingtonpost.com\/technology\/2022\/06\/15\/tesla-autopilot-crashes\/"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-15561-1_2"},{"key":"e_1_3_2_1_27_1","first-page":"6616","article-title":"Large-scale adversarial training for vision-and-language representation learning","volume":"33","author":"Gan Zhe","year":"2020","unstructured":"Zhe Gan , Yen-Chun Chen , Linjie Li , Chen Zhu , Yu Cheng , and Jingjing Liu . 2020 . Large-scale adversarial training for vision-and-language representation learning . Advances in Neural Information Processing Systems , 33 (2020), 6616 \u2013 6628 . Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. 2020. Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems, 33 (2020), 6616\u20136628.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380339"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00047"},{"key":"e_1_3_2_1_30_1","volume-title":"Vivo: Surpassing human performance in novel object captioning with visual vocabulary pre-training. arXiv preprint arXiv:2009.13682.","author":"Hu Xiaowei","year":"2020","unstructured":"Xiaowei Hu , Xi Yin , Kevin Lin , Lijuan Wang , Lei Zhang , Jianfeng Gao , and Zicheng Liu . 2020 . Vivo: Surpassing human performance in novel object captioning with visual vocabulary pre-training. arXiv preprint arXiv:2009.13682. Xiaowei Hu, Xi Yin, Kevin Lin, Lijuan Wang, Lei Zhang, Jianfeng Gao, and Zicheng Liu. 2020. Vivo: Surpassing human performance in novel object captioning with visual vocabulary pre-training. arXiv preprint arXiv:2009.13682."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464825"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534391"},{"key":"e_1_3_2_1_33_1","unstructured":"Sajad Khatiri Christian Birchler Bill Bosshard Alessio Gambi and Sebastiano Panichella. 2021. Machine Learning-based Test Selection for Simulation-based Testing of Self-driving Cars Software. arXiv preprint arXiv:2111.04666. \t\t\t\t  Sajad Khatiri Christian Birchler Bill Bosshard Alessio Gambi and Sebastiano Panichella. 2021. Machine Learning-based Test Selection for Simulation-based Testing of Self-driving Cars Software. arXiv preprint arXiv:2111.04666."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcm.2016.02.012"},{"key":"e_1_3_2_1_35_1","volume-title":"Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems (NIPS), 25","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky , Ilya Sutskever , and Geoffrey E Hinton . 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems (NIPS), 25 ( 2012 ), 1097\u20131105. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems (NIPS), 25 (2012), 1097\u20131105."},{"key":"e_1_3_2_1_36_1","unstructured":"Fred. Lambert. 2016. Understanding the fatal Tesla accident on Autopilot and the NHTSA probe.. https:\/\/electrek.co\/2016\/07\/01\/understanding-fatal-tesla-accident-autopilot-nhtsa-probe\/ \t\t\t\t  Fred. Lambert. 2016. Understanding the fatal Tesla accident on Autopilot and the NHTSA probe.. https:\/\/electrek.co\/2016\/07\/01\/understanding-fatal-tesla-accident-autopilot-nhtsa-probe\/"},{"key":"e_1_3_2_1_37_1","volume-title":"Tesla fatal crash: \u2019autopilot","author":"Levin Sam","year":"2018","unstructured":"Sam Levin . 2018. Tesla fatal crash: \u2019autopilot \u2019 mode sped up car before driver killed, report finds.. https:\/\/www.theguardian.com\/technology\/ 2018 \/jun\/07\/tesla-fatal-crash-silicon-valley-autopilot-mode-report Sam Levin. 2018. Tesla fatal crash: \u2019autopilot\u2019 mode sped up car before driver killed, report finds.. https:\/\/www.theguardian.com\/technology\/2018\/jun\/07\/tesla-fatal-crash-silicon-valley-autopilot-mode-report"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/2018936.2018962"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58577-8_8"},{"key":"e_1_3_2_1_40_1","unstructured":"Yingqi Liu Shiqing Ma Yousra Aafer Wen-Chuan Lee Juan Zhai Weihang Wang and Xiangyu Zhang. 2017. Trojaning attack on neural networks. \t\t\t\t  Yingqi Liu Shiqing Ma Yousra Aafer Wen-Chuan Lee Juan Zhai Weihang Wang and Xiangyu Zhang. 2017. Trojaning attack on neural networks."},{"key":"e_1_3_2_1_41_1","volume-title":"QATest: A Uniform Fuzzing Framework for Question Answering Systems. In 37th IEEE\/ACM International Conference on Automated Software Engineering. 1\u201312","author":"Liu Zixi","year":"2022","unstructured":"Zixi Liu , Yang Feng , Yining Yin , Jingyu Sun , Zhenyu Chen , and Baowen Xu . 2022 . QATest: A Uniform Fuzzing Framework for Question Answering Systems. In 37th IEEE\/ACM International Conference on Automated Software Engineering. 1\u201312 . Zixi Liu, Yang Feng, Yining Yin, Jingyu Sun, Zhenyu Chen, and Baowen Xu. 2022. QATest: A Uniform Fuzzing Framework for Question Answering Systems. In 37th IEEE\/ACM International Conference on Automated Software Engineering. 1\u201312."},{"key":"e_1_3_2_1_42_1","volume-title":"Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32","author":"Lu Jiasen","year":"2019","unstructured":"Jiasen Lu , Dhruv Batra , Devi Parikh , and Stefan Lee . 2019 . Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32 (2019). Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32 (2019)."},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_1_44_1","volume-title":"Interrater reliability: the kappa statistic. Biochemia medica, 22, 3","author":"McHugh Mary L","year":"2012","unstructured":"Mary L McHugh . 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22, 3 ( 2012 ), 276\u2013282. Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22, 3 (2012), 276\u2013282."},{"key":"e_1_3_2_1_45_1","unstructured":"Youssef Mroueh. 2020. Image Captioning as an Assistive Technology. https:\/\/www.ibm.com\/blogs\/research\/2020\/07\/image-captioning-assistive-technology \t\t\t\t  Youssef Mroueh. 2020. Image Captioning as an Assistive Technology. https:\/\/www.ibm.com\/blogs\/research\/2020\/07\/image-captioning-assistive-technology"},{"key":"e_1_3_2_1_46_1","unstructured":"CAROL VAN NATTA. 2020. AI Fails at Photo Captions. https:\/\/author.carolvannatta.com\/ai-fails-at-photo-captions\/ \t\t\t\t  CAROL VAN NATTA. 2020. AI Fails at Photo Captions. https:\/\/author.carolvannatta.com\/ai-fails-at-photo-captions\/"},{"key":"e_1_3_2_1_47_1","unstructured":"BBC News. 2015. Google apologises for Photos app\u2019s racist blunder.. https:\/\/www.bbc.com\/news\/technology-33347866 \t\t\t\t  BBC News. 2015. Google apologises for Photos app\u2019s racist blunder.. https:\/\/www.bbc.com\/news\/technology-33347866"},{"key":"e_1_3_2_1_48_1","unstructured":"Stanford NLP. 2020. Stanza \u2013 A Python NLP Package for Many Human Languages.. https:\/\/stanfordnlp.github.io\/stanza\/ \t\t\t\t  Stanford NLP. 2020. Stanza \u2013 A Python NLP Package for Many Human Languages.. https:\/\/stanfordnlp.github.io\/stanza\/"},{"key":"e_1_3_2_1_49_1","volume-title":"Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems (NeurIPS), 24","author":"Ordonez Vicente","year":"2011","unstructured":"Vicente Ordonez , Girish Kulkarni , and Tamara Berg . 2011. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems (NeurIPS), 24 ( 2011 ). Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems (NeurIPS), 24 (2011)."},{"key":"e_1_3_2_1_50_1","unstructured":"World Health Organization. 2019. World report on vision. ISBN: 9241516577 Publisher: World Health Organization \t\t\t\t  World Health Organization. 2019. World report on vision. ISBN: 9241516577 Publisher: World Health Organization"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICME.2004.1394652"},{"key":"e_1_3_2_1_52_1","volume-title":"Test Case Prioritization for Deep Neural Networks. In 2022 9th International Conference on Dependable Systems and Their Applications (DSA). 624\u2013628","author":"Pan Zhonghao","year":"2022","unstructured":"Zhonghao Pan , Shan Zhou , Jianmin Wang , Jinbo Wang , Jiao Jia , and Yang Feng . 2022 . Test Case Prioritization for Deep Neural Networks. In 2022 9th International Conference on Dependable Systems and Their Applications (DSA). 624\u2013628 . Zhonghao Pan, Shan Zhou, Jianmin Wang, Jinbo Wang, Jiao Jia, and Yang Feng. 2022. Test Case Prioritization for Deep Neural Networks. In 2022 9th International Conference on Dependable Systems and Their Applications (DSA). 624\u2013628."},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-022-10207-5"},{"key":"e_1_3_2_1_54_1","volume-title":"Yves Le Traon, and Mark Harman","author":"Papadakis Mike","year":"2019","unstructured":"Mike Papadakis , Marinos Kintis , Jie Zhang , Yue Jia , Yves Le Traon, and Mark Harman . 2019 . Mutation testing advances: an analysis and survey. In Advances in Computers. 112, Elsevier , 275\u2013378. Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation testing advances: an analysis and survey. In Advances in Computers. 112, Elsevier, 275\u2013378."},{"key":"e_1_3_2_1_55_1","volume-title":"DEVIATE: A Deep Learning Variance Testing Framework. In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 1286\u20131290","author":"Pham Hung Viet","year":"2021","unstructured":"Hung Viet Pham , Mijung Kim , Lin Tan , Yaoliang Yu , and Nachiappan Nagappan . 2021 . DEVIATE: A Deep Learning Variance Testing Framework. In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 1286\u20131290 . Hung Viet Pham, Mijung Kim, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. 2021. DEVIATE: A Deep Learning Variance Testing Framework. In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 1286\u20131290."},{"volume-title":"Prolific: A higher standard of online research.. https:\/\/prolific.co\/","year":"2022","key":"e_1_3_2_1_56_1","unstructured":"Prolific. 2022 . Prolific: A higher standard of online research.. https:\/\/prolific.co\/ Prolific. 2022. Prolific: A higher standard of online research.. https:\/\/prolific.co\/"},{"key":"e_1_3_2_1_57_1","volume-title":"COSMO: Code Coverage Made Easier for Android. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). 417\u2013423","author":"Romdhana Andrea","year":"2021","unstructured":"Andrea Romdhana , Mariano Ceccato , Gabriel Claudiu Georgiu , Alessio Merlo , and Paolo Tonella . 2021 . COSMO: Code Coverage Made Easier for Android. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). 417\u2013423 . Andrea Romdhana, Mariano Ceccato, Gabriel Claudiu Georgiu, Alessio Merlo, and Paolo Tonella. 2021. COSMO: Code Coverage Made Easier for Android. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). 417\u2013423."},{"key":"e_1_3_2_1_58_1","unstructured":"Stan Schroeder. 2016. Microsoft created a bot to auto-caption photos and it\u2019s going hilariously wrong. https:\/\/mashable.com\/article\/microsoft-captionbot Section: Life \t\t\t\t  Stan Schroeder. 2016. Microsoft created a bot to auto-caption photos and it\u2019s going hilariously wrong. https:\/\/mashable.com\/article\/microsoft-captionbot Section: Life"},{"key":"e_1_3_2_1_59_1","unstructured":"Thomas Smith. 2021. AI Is Terrible at Writing Alt Text.. https:\/\/tomsmith585.medium.com\/ai-is-terrible-at-writing-alt-text-e79b0c4ecf51 \t\t\t\t  Thomas Smith. 2021. AI Is Terrible at Writing Alt Text.. https:\/\/tomsmith585.medium.com\/ai-is-terrible-at-writing-alt-text-e79b0c4ecf51"},{"key":"e_1_3_2_1_60_1","unstructured":"Thomas Smith. 2022. Natural Language Toolkit.. https:\/\/www.nltk.org\/ \t\t\t\t  Thomas Smith. 2022. Natural Language Toolkit.. https:\/\/www.nltk.org\/"},{"key":"e_1_3_2_1_61_1","unstructured":"Matteo Stefanini Marcella Cornia Lorenzo Baraldi Silvia Cascianelli Giuseppe Fiameni and Rita Cucchiara. 2021. From show to tell: A survey on image captioning. arXiv preprint arXiv:2107.06912. \t\t\t\t  Matteo Stefanini Marcella Cornia Lorenzo Baraldi Silvia Cascianelli Giuseppe Fiameni and Rita Cucchiara. 2021. From show to tell: A survey on image captioning. arXiv preprint arXiv:2107.06912."},{"key":"e_1_3_2_1_62_1","unstructured":"Youcheng Sun Xiaowei Huang Daniel Kroening James Sharp Matthew Hill and Rob Ashmore. 2018. Testing deep neural networks. arXiv preprint arXiv:1803.04792. \t\t\t\t  Youcheng Sun Xiaowei Huang Daniel Kroening James Sharp Matthew Hill and Rob Ashmore. 2018. Testing deep neural networks. arXiv preprint arXiv:1803.04792."},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3358233"},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"e_1_3_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/WACV51458.2022.00323"},{"key":"e_1_3_2_1_66_1","volume-title":"Attacks meet interpretability: Attribute-steered detection of adversarial samples. arXiv preprint arXiv:1810","author":"Tao Guanhong","year":"2018","unstructured":"Guanhong Tao , Shiqing Ma , Yingqi Liu , and Xiangyu Zhang . 2018 . Attacks meet interpretability: Attribute-steered detection of adversarial samples. arXiv preprint arXiv:1810 .11580. Guanhong Tao, Shiqing Ma, Yingqi Liu, and Xiangyu Zhang. 2018. Attacks meet interpretability: Attribute-steered detection of adversarial samples. arXiv preprint arXiv:1810.11580."},{"key":"e_1_3_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299087"},{"volume-title":"MIT study finds \u2018systematic","key":"e_1_3_2_1_69_1","unstructured":"VentureBeat. 2022. MIT study finds \u2018systematic \u2019 labeling errors in popular AI benchmark datasets.. https:\/\/venturebeat.com\/business\/mit-study-finds-systematic-labeling-errors-in-popular-ai-benchmark-datasets\/ VentureBeat. 2022. MIT study finds \u2018systematic\u2019 labeling errors in popular AI benchmark datasets.. https:\/\/venturebeat.com\/business\/mit-study-finds-systematic-labeling-errors-in-popular-ai-benchmark-datasets\/"},{"key":"e_1_3_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298935"},{"volume-title":"Assessment of source code obfuscation techniques. In 2016 IEEE 16th international working conference on source code analysis and manipulation (SCAM). 11\u201320","author":"Viticchi\u00e9 Alessio","key":"e_1_3_2_1_71_1","unstructured":"Alessio Viticchi\u00e9 , Leonardo Regano , Marco Torchiano , Cataldo Basile , Mariano Ceccato , Paolo Tonella , and Roberto Tiella . 2016. Assessment of source code obfuscation techniques. In 2016 IEEE 16th international working conference on source code analysis and manipulation (SCAM). 11\u201320 . Alessio Viticchi\u00e9, Leonardo Regano, Marco Torchiano, Cataldo Basile, Mariano Ceccato, Paolo Tonella, and Roberto Tiella. 2016. Assessment of source code obfuscation techniques. In 2016 IEEE 16th international working conference on source code analysis and manipulation (SCAM). 11\u201320."},{"key":"e_1_3_2_1_72_1","doi-asserted-by":"crossref","unstructured":"Josiah Wang Pranava Madhyastha and Lucia Specia. 2018. Object counts! bringing explicit detections back into image captioning. arXiv preprint arXiv:1805.00314. \t\t\t\t  Josiah Wang Pranava Madhyastha and Lucia Specia. 2018. Object counts! bringing explicit detections back into image captioning. arXiv preprint arXiv:1805.00314.","DOI":"10.18653\/v1\/N18-1198"},{"key":"e_1_3_2_1_73_1","unstructured":"Peng Wang An Yang Rui Men Junyang Lin Shuai Bai Zhikang Li Jianxin Ma Chang Zhou Jingren Zhou and Hongxia Yang. 2022. Unifying architectures tasks and modalities through a simple sequence-to-sequence learning framework. arXiv preprint arXiv:2202.03052. \t\t\t\t  Peng Wang An Yang Rui Men Junyang Lin Shuai Bai Zhikang Li Jianxin Ma Chang Zhou Jingren Zhou and Hongxia Yang. 2022. Unifying architectures tasks and modalities through a simple sequence-to-sequence learning framework. arXiv preprint arXiv:2202.03052."},{"key":"e_1_3_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416584"},{"key":"e_1_3_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_3_2_1_76_1","volume-title":"International conference on machine learning (ICML). PMLR","author":"Xu Kelvin","year":"2015","unstructured":"Kelvin Xu , Jimmy Ba , Ryan Kiros , Kyunghyun Cho , Aaron Courville , Ruslan Salakhudinov , Rich Zemel , and Yoshua Bengio . 2015 . Show, attend and tell: Neural image caption generation with visual attention . In International conference on machine learning (ICML). PMLR , 2048\u20132057. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning (ICML). PMLR, 2048\u20132057."},{"key":"e_1_3_2_1_77_1","unstructured":"Qiuling Xu Guanhong Tao Siyuan Cheng and Xiangyu Zhang. 2020. Towards feature space adversarial attack. arXiv preprint arXiv:2004.12385. \t\t\t\t  Qiuling Xu Guanhong Tao Siyuan Cheng and Xiangyu Zhang. 2020. Towards feature space adversarial attack. arXiv preprint arXiv:2004.12385."},{"key":"e_1_3_2_1_78_1","volume-title":"Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP). 444\u2013454","author":"Yang Yezhou","year":"2011","unstructured":"Yezhou Yang , Ching Teo , Hal Daum\u00e9 III, and Yiannis Aloimonos . 2011 . Corpus-guided sentence generation of natural images . In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP). 444\u2013454 . Yezhou Yang, Ching Teo, Hal Daum\u00e9 III, and Yiannis Aloimonos. 2011. Corpus-guided sentence generation of natural images. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP). 444\u2013454."},{"key":"e_1_3_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534389"},{"key":"e_1_3_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10076-4"},{"key":"e_1_3_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00051"},{"key":"e_1_3_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_2_1_83_1","doi-asserted-by":"crossref","unstructured":"Pengchuan Zhang Xiujun Li Xiaowei Hu Jianwei Yang Lei Zhang Lijuan Wang Yejin Choi and Jianfeng Gao. 2021. VinVL: Making Visual Representations Matter in Vision-Language Models. arXiv preprint arXiv:2101.00529. \t\t\t\t  Pengchuan Zhang Xiujun Li Xiaowei Hu Jianwei Yang Lei Zhang Lijuan Wang Yejin Choi and Jianfeng Gao. 2021. VinVL: Making Visual Representations Matter in Vision-Language Models. arXiv preprint arXiv:2101.00529.","DOI":"10.1109\/CVPR46437.2021.00553"},{"key":"e_1_3_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2012.6247882"},{"key":"e_1_3_2_1_85_1","unstructured":"Chris. Ziegler. 2016. A Google self-driving car caused a crash for the first time.. https:\/\/www.theverge.com\/2016\/2\/29\/11134344\/google-self-driving-car-crash-report \t\t\t\t  Chris. Ziegler. 2016. A Google self-driving car caused a crash for the first time.. https:\/\/www.theverge.com\/2016\/2\/29\/11134344\/google-self-driving-car-crash-report"}],"event":{"name":"ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","AITO"],"location":"Seattle WA USA","acronym":"ISSTA '23"},"container-title":["Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597926.3598094","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3597926.3598094","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:42Z","timestamp":1750182522000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597926.3598094"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,12]]},"references-count":85,"alternative-id":["10.1145\/3597926.3598094","10.1145\/3597926"],"URL":"https:\/\/doi.org\/10.1145\/3597926.3598094","relation":{},"subject":[],"published":{"date-parts":[[2023,7,12]]},"assertion":[{"value":"2023-07-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}