{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T06:10:03Z","timestamp":1774937403098,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":99,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,7,18]],"date-time":"2022-07-18T00:00:00Z","timestamp":1658102400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,7,18]]},"DOI":"10.1145\/3533767.3534388","type":"proceedings-article","created":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T14:28:50Z","timestamp":1657895330000},"page":"378-390","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":35,"title":["MDPFuzz: testing models solving Markov decision processes"],"prefix":"10.1145","author":[{"given":"Qi","family":"Pang","sequence":"first","affiliation":[{"name":"Hong Kong University of Science and Technology, China"}]},{"given":"Yuanyuan","family":"Yuan","sequence":"additional","affiliation":[{"name":"Hong Kong University of Science and Technology, China"}]},{"given":"Shuai","family":"Wang","sequence":"additional","affiliation":[{"name":"Hong Kong University of Science and Technology, China"}]}],"member":"320","published-online":{"date-parts":[[2022,7,18]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"[n.d.]. Baidu Autonomous Driving Platform.. https:\/\/apollo.auto\/index.html [n.d.]. Baidu Autonomous Driving Platform.. https:\/\/apollo.auto\/index.html"},{"key":"e_1_3_2_1_2_1","unstructured":"[n.d.]. The CARLA Autonomous Driving Challenge.. https:\/\/carlachallenge.org\/ [n.d.]. The CARLA Autonomous Driving Challenge.. https:\/\/carlachallenge.org\/"},{"key":"e_1_3_2_1_3_1","unstructured":"[n.d.]. The CARLA Autonomous Driving Leaderboard.. https:\/\/leaderboard.carla.org\/leaderboard\/ [n.d.]. The CARLA Autonomous Driving Leaderboard.. https:\/\/leaderboard.carla.org\/leaderboard\/"},{"key":"e_1_3_2_1_4_1","unstructured":"[n.d.]. Google self-driving car crash.. https:\/\/www.theverge.com\/2016\/2\/29\/11134344\/google-self-driving-car-crash-report [n.d.]. Google self-driving car crash.. https:\/\/www.theverge.com\/2016\/2\/29\/11134344\/google-self-driving-car-crash-report"},{"key":"e_1_3_2_1_5_1","unstructured":"[n.d.]. Markov Decision Process.. https:\/\/en.wikipedia.org\/wiki\/Markov_decision_process [n.d.]. Markov Decision Process.. https:\/\/en.wikipedia.org\/wiki\/Markov_decision_process"},{"key":"e_1_3_2_1_6_1","unstructured":"[n.d.]. MDPFuzz. https:\/\/sites.google.com\/view\/mdpfuzz [n.d.]. MDPFuzz. https:\/\/sites.google.com\/view\/mdpfuzz"},{"key":"e_1_3_2_1_7_1","unstructured":"[n.d.]. NASA FAA Industry Conduct Initial Sense-and-Avoid Test.. https:\/\/www.nasa.gov\/centers\/armstrong\/Features\/acas_xu_paves_the_way.htmll [n.d.]. NASA FAA Industry Conduct Initial Sense-and-Avoid Test.. https:\/\/www.nasa.gov\/centers\/armstrong\/Features\/acas_xu_paves_the_way.htmll"},{"key":"e_1_3_2_1_8_1","unstructured":"[n.d.]. NAVAIR Plans to Install ACAS Xu on MQ-4C Fleet.. https:\/\/www.flightglobal.com\/news\/articles\/navair-plans-to-install-acas-xu-on-mq-4cfleet-444989\/ [n.d.]. NAVAIR Plans to Install ACAS Xu on MQ-4C Fleet.. https:\/\/www.flightglobal.com\/news\/articles\/navair-plans-to-install-acas-xu-on-mq-4cfleet-444989\/"},{"key":"e_1_3_2_1_9_1","unstructured":"[n.d.]. Tesla Model S crash.. https:\/\/www.teslarati.com\/tesla-model-s-firetruck-crash-details\/ [n.d.]. Tesla Model S crash.. https:\/\/www.teslarati.com\/tesla-model-s-firetruck-crash-details\/"},{"key":"e_1_3_2_1_10_1","unstructured":"[n.d.]. Uber autonomous driving crash.. https:\/\/www.nytimes.com\/2018\/05\/07\/technology\/uber-crash-autonomous-driveai.html [n.d.]. Uber autonomous driving crash.. https:\/\/www.nytimes.com\/2018\/05\/07\/technology\/uber-crash-autonomous-driveai.html"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2020.113816"},{"key":"e_1_3_2_1_12_1","volume-title":"Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03079.","author":"Bansal Mayank","year":"2018","unstructured":"Mayank Bansal , Alex Krizhevsky , and Abhijit Ogale . 2018 . Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03079. Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. 2018. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03079."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-62416-7_19"},{"key":"e_1_3_2_1_14_1","unstructured":"Mohamed Ishmael Belghazi Aristide Baratin Sai Rajeshwar Sherjil Ozair Yoshua Bengio Aaron Courville and Devon Hjelm. 2018. Mutual information neural estimation. In ICML. Mohamed Ishmael Belghazi Aristide Baratin Sai Rajeshwar Sherjil Ozair Yoshua Bengio Aaron Courville and Devon Hjelm. 2018. Mutual information neural estimation. In ICML."},{"key":"e_1_3_2_1_15_1","volume-title":"Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, and Chris Hesse.","author":"Berner Christopher","year":"2019","unstructured":"Christopher Berner , Greg Brockman , Brooke Chan , Vicki Cheung , Przemys\u0142 aw D\u0119biak , Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, and Chris Hesse. 2019 . Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680. Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemys\u0142 aw D\u0119biak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, and Chris Hesse. 2019. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680."},{"key":"e_1_3_2_1_16_1","volume-title":"Coverage-based greybox fuzzing as markov chain","author":"B\u00f6hme Marcel","unstructured":"Marcel B\u00f6hme , Van-Thuan Pham , and Abhik Roychoudhury . 2017. Coverage-based greybox fuzzing as markov chain . IEEE TSE. Marcel B\u00f6hme, Van-Thuan Pham, and Abhik Roychoudhury. 2017. Coverage-based greybox fuzzing as markov chain. IEEE TSE."},{"key":"e_1_3_2_1_17_1","volume-title":"Davide Del Testa","author":"Bojarski Mariusz","year":"2016","unstructured":"Mariusz Bojarski , Davide Del Testa , Daniel Dworakowski, Bernhard Firner , Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, and Jiakai Zhang. 2016 . End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316. Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, and Jiakai Zhang. 2016. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316."},{"key":"e_1_3_2_1_18_1","unstructured":"Mariusz Bojarski Philip Yeres Anna Choromanska Krzysztof Choromanski Bernhard Firner Lawrence Jackel and Urs Muller. 2017. Explaining how a deep neural network trained with end-to-end learning steers a car. arXiv preprint arXiv:1704.07911. Mariusz Bojarski Philip Yeres Anna Choromanska Krzysztof Choromanski Bernhard Firner Lawrence Jackel and Urs Muller. 2017. Explaining how a deep neural network trained with end-to-end learning steers a car. arXiv preprint arXiv:1704.07911."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2007.913919"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9868.2009.00698.x"},{"key":"e_1_3_2_1_21_1","unstructured":"George Casella and Roger L Berger. 2021. Statistical inference. Cengage Learning. George Casella and Roger L Berger. 2021. Statistical inference. Cengage Learning."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Pablo Samuel Castro. 2020. Scalable methods for computing state similarity in deterministic markov decision processes. In AAAI. Pablo Samuel Castro. 2020. Scalable methods for computing state similarity in deterministic markov decision processes. In AAAI.","DOI":"10.1609\/aaai.v34i06.6564"},{"key":"e_1_3_2_1_23_1","volume-title":"Conference on Robot Learning. 66\u201375","author":"Chen Dian","year":"2020","unstructured":"Dian Chen , Brady Zhou , Vladlen Koltun , and Philipp Kr\u00e4henb\u00fchl . 2020 . Learning by cheating . In Conference on Robot Learning. 66\u201375 . Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Kr\u00e4henb\u00fchl. 2020. Learning by cheating. In Conference on Robot Learning. 66\u201375."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Junjie Chen Yanwei Bai Dan Hao Yingfei Xiong Hongyu Zhang and Bing Xie. 2017. Learning to Prioritize Test Programs for Compiler Testing. In ICSE. Junjie Chen Yanwei Bai Dan Hao Yingfei Xiong Hongyu Zhang and Bing Xie. 2017. Learning to Prioritize Test Programs for Compiler Testing. In ICSE.","DOI":"10.1109\/ICSE.2017.70"},{"key":"e_1_3_2_1_25_1","unstructured":"Songqiang Chen Shuo Jin and Xiaoyuan Xie. 2021. Validation on machine reading comprehension software without annotated labels: a property-based method. In FSE. Songqiang Chen Shuo Jin and Xiaoyuan Xie. 2021. Validation on machine reading comprehension software without annotated labels: a property-based method. In FSE."},{"key":"e_1_3_2_1_26_1","volume-title":"Mean shift, mode seeking, and clustering","author":"Cheng Yizong","year":"1995","unstructured":"Yizong Cheng . 1995. Mean shift, mode seeking, and clustering . IEEE transactions on pattern analysis and machine intelligence, 17, 8 ( 1995 ), 790\u2013799. Yizong Cheng. 1995. Mean shift, mode seeking, and clustering. IEEE transactions on pattern analysis and machine intelligence, 17, 8 (1995), 790\u2013799."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1977.tb01600.x"},{"key":"e_1_3_2_1_28_1","volume-title":"Conference on robot learning. 1\u201316","author":"Dosovitskiy Alexey","year":"2017","unstructured":"Alexey Dosovitskiy , German Ros , Felipe Codevilla , Antonio Lopez , and Vladlen Koltun . 2017 . CARLA: An open urban driving simulator . In Conference on robot learning. 1\u201316 . Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. In Conference on robot learning. 1\u201316."},{"key":"e_1_3_2_1_29_1","unstructured":"Xiaoning Du Xiaofei Xie Yi Li Lei Ma Yang Liu and Jianjun Zhao. 2019. DeepStellar: Model-Based Quantitative Analysis of Stateful Deep Learning Systems. In FSE. Xiaoning Du Xiaofei Xie Yi Li Lei Ma Yang Liu and Jianjun Zhao. 2019. DeepStellar: Model-Based Quantitative Analysis of Stateful Deep Learning Systems. In FSE."},{"key":"e_1_3_2_1_30_1","volume-title":"Deepcruiser: Automated guided testing for stateful deep learning systems. arXiv preprint arXiv:1812.05339.","author":"Du Xiaoning","year":"2018","unstructured":"Xiaoning Du , Xiaofei Xie , Yi Li , Lei Ma , Jianjun Zhao , and Yang Liu . 2018 . Deepcruiser: Automated guided testing for stateful deep learning systems. arXiv preprint arXiv:1812.05339. Xiaoning Du, Xiaofei Xie, Yi Li, Lei Ma, Jianjun Zhao, and Yang Liu. 2018. Deepcruiser: Automated guided testing for stateful deep learning systems. arXiv preprint arXiv:1812.05339."},{"key":"e_1_3_2_1_31_1","volume-title":"Neville Dubash, and Sanjay Podder.","author":"Dwarakanath Anurag","year":"2018","unstructured":"Anurag Dwarakanath , Manish Ahuja , Samarth Sikand , Raghotham M. Rao , R. P. Jagadeesh Chandra Bose , Neville Dubash, and Sanjay Podder. 2018 . Identifying Implementation Bugs in Machine Learning Based Image Classifiers Using Metamorphic Testing. In ISSTA. Anurag Dwarakanath, Manish Ahuja, Samarth Sikand, Raghotham M. Rao, R. P. Jagadeesh Chandra Bose, Neville Dubash, and Sanjay Podder. 2018. Identifying Implementation Bugs in Machine Learning Based Image Classifiers Using Metamorphic Testing. In ISSTA."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30281-8_10"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2006.79"},{"key":"e_1_3_2_1_34_1","volume-title":"Genaug: Data augmentation for finetuning text generators. arXiv preprint arXiv:2010.01794.","author":"Feng Steven Y","year":"2020","unstructured":"Steven Y Feng , Varun Gangal , Dongyeop Kang , Teruko Mitamura , and Eduard Hovy . 2020 . Genaug: Data augmentation for finetuning text generators. arXiv preprint arXiv:2010.01794. Steven Y Feng, Varun Gangal, Dongyeop Kang, Teruko Mitamura, and Eduard Hovy. 2020. Genaug: Data augmentation for finetuning text generators. arXiv preprint arXiv:2010.01794."},{"key":"e_1_3_2_1_35_1","volume-title":"Doina Precup, and Prakash Panangaden.","author":"Ferns Norman","year":"2012","unstructured":"Norman Ferns , Pablo Samuel Castro , Doina Precup, and Prakash Panangaden. 2012 . Methods for computing state similarity in Markov de cision processes. arXiv preprint arXiv:1206.6836. Norman Ferns, Pablo Samuel Castro, Doina Precup, and Prakash Panangaden. 2012. Methods for computing state similarity in Markov decision processes. arXiv preprint arXiv:1206.6836."},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1137\/10080484X"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"crossref","unstructured":"Sainyam Galhotra Yuriy Brun and Alexandra Meliou. 2017. Fairness testing: testing software for discrimination. In ACM ESEC\/FSE. 498\u2013510. Sainyam Galhotra Yuriy Brun and Alexandra Meliou. 2017. Fairness testing: testing software for discrimination. In ACM ESEC\/FSE. 498\u2013510.","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_3_2_1_38_1","unstructured":"Arthur Gill. 1962. Introduction to the theory of finite-state machines. Arthur Gill. 1962. Introduction to the theory of finite-state machines."},{"key":"e_1_3_2_1_39_1","unstructured":"Adam Gleave Michael Dennis Cody Wild Neel Kant Sergey Levine and Stuart Russell. 2019. Adversarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv:1905.10615. Adam Gleave Michael Dennis Cody Wild Neel Kant Sergey Levine and Stuart Russell. 2019. Adversarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv:1905.10615."},{"key":"e_1_3_2_1_40_1","volume-title":"Deep learning","author":"Goodfellow Ian","unstructured":"Ian Goodfellow , Yoshua Bengio , and Aaron Courville . 2016. Deep learning . MIT press . Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT press."},{"key":"e_1_3_2_1_41_1","volume-title":"Rnn-test: Adversarial testing framework for recurrent neural network systems. arXiv preprint arXiv:1911.06155.","author":"Guo Jianmin","year":"2019","unstructured":"Jianmin Guo , Yue Zhao , Xueying Han , Yu Jiang , and Jiaguang Sun . 2019 . Rnn-test: Adversarial testing framework for recurrent neural network systems. arXiv preprint arXiv:1911.06155. Jianmin Guo, Yue Zhao, Xueying Han, Yu Jiang, and Jiaguang Sun. 2019. Rnn-test: Adversarial testing framework for recurrent neural network systems. arXiv preprint arXiv:1911.06155."},{"key":"e_1_3_2_1_42_1","unstructured":"Pinjia He Clara Meister and Zhendong Su. 2020. Structure-invariant testing for machine translation. In ICSE. Pinjia He Clara Meister and Zhendong Su. 2020. Structure-invariant testing for machine translation. In ICSE."},{"key":"e_1_3_2_1_43_1","unstructured":"R Devon Hjelm Alex Fedorov Samuel Lavoie-Marchildon Karan Grewal Phil Bachman Adam Trischler and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670. R Devon Hjelm Alex Fedorov Samuel Lavoie-Marchildon Karan Grewal Phil Bachman Adam Trischler and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670."},{"key":"e_1_3_2_1_44_1","volume-title":"Generative adversarial imitation learning. Advances in neural information processing systems, 29","author":"Ho Jonathan","year":"2016","unstructured":"Jonathan Ho and Stefano Ermon . 2016. Generative adversarial imitation learning. Advances in neural information processing systems, 29 ( 2016 ), 4565\u20134573. Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. Advances in neural information processing systems, 29 (2016), 4565\u20134573."},{"key":"e_1_3_2_1_45_1","unstructured":"Wei Huang Youcheng Sun Xiaowei Huang and James Sharp. 2019. testrnn: Coverage-guided testing on recurrent neural networks. arXiv preprint arXiv:1906.08557. Wei Huang Youcheng Sun Xiaowei Huang and James Sharp. 2019. testrnn: Coverage-guided testing on recurrent neural networks. arXiv preprint arXiv:1906.08557."},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.2514\/1.G003724"},{"key":"e_1_3_2_1_47_1","volume-title":"Validation of image-based neural network controllers through adaptive stress testing","author":"Julian Kyle D","unstructured":"Kyle D Julian , Ritchie Lee , and Mykel J Kochenderfer . 2020. Validation of image-based neural network controllers through adaptive stress testing . In IEEE ITSC. Kyle D Julian, Ritchie Lee, and Mykel J Kochenderfer. 2020. Validation of image-based neural network controllers through adaptive stress testing. In IEEE ITSC."},{"key":"e_1_3_2_1_48_1","unstructured":"Jinhan Kim Robert Feldt and Shin Yoo. 2019. Guiding Deep Learning System Testing Using Surprise Adequacy. In ICSE. Jinhan Kim Robert Feldt and Shin Yoo. 2019. Guiding Deep Learning System Testing Using Surprise Adequacy. In ICSE."},{"key":"e_1_3_2_1_49_1","unstructured":"Jinhan Kim Jeongil Ju Robert Feldt and Shin Yoo. 2020. Reducing dnn labelling cost using surprise adequacy: An industrial case study for autonomous driving. In FSE. Jinhan Kim Jeongil Ju Robert Feldt and Shin Yoo. 2020. Reducing dnn labelling cost using surprise adequacy: An industrial case study for autonomous driving. In FSE."},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"crossref","unstructured":"George Klees Andrew Ruef Benji Cooper Shiyi Wei and Michael Hicks. 2018. Evaluating fuzz testing. In CCS. George Klees Andrew Ruef Benji Cooper Shiyi Wei and Michael Hicks. 2018. Evaluating fuzz testing. In CCS.","DOI":"10.1145\/3243734.3243804"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1177\/0278364913495721"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2018.8500400"},{"key":"e_1_3_2_1_53_1","unstructured":"Jernej Kos and Dawn Song. 2017. Delving into adversarial attacks on deep policies. arXiv preprint arXiv:1705.06452. Jernej Kos and Dawn Song. 2017. Delving into adversarial attacks on deep policies. arXiv preprint arXiv:1705.06452."},{"key":"e_1_3_2_1_54_1","unstructured":"Arsenii Kuznetsov Pavel Shvechikov Alexander Grishin and Dmitry Vetrov. 2020. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In ICML. Arsenii Kuznetsov Pavel Shvechikov Alexander Grishin and Dmitry Vetrov. 2020. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In ICML."},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12190"},{"key":"e_1_3_2_1_56_1","unstructured":"Timothy P Lillicrap Jonathan J Hunt Alexander Pritzel Nicolas Heess Tom Erez Yuval Tassa David Silver and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971. Timothy P Lillicrap Jonathan J Hunt Alexander Pritzel Nicolas Heess Tom Erez Yuval Tassa David Silver and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971."},{"key":"e_1_3_2_1_57_1","unstructured":"Ryan Lowe Yi Wu Aviv Tamar Jean Harb Pieter Abbeel and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. arXiv preprint arXiv:1706.02275. Ryan Lowe Yi Wu Aviv Tamar Jean Harb Pieter Abbeel and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. arXiv preprint arXiv:1706.02275."},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_1_59_1","unstructured":"Pingchuan Ma and Shuai Wang. 2022. MT-Teql: Evaluating and Augmenting Neural NLIDB on Real-world Linguistic and Schema Variations. In PVLDB. Pingchuan Ma and Shuai Wang. 2022. MT-Teql: Evaluating and Augmenting Neural NLIDB on Real-world Linguistic and Schema Variations. In PVLDB."},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"crossref","unstructured":"Pingchuan Ma Shuai Wang and Jin Liu. 2020. Metamorphic Testing and Certified Mitigation of Fairness Violations in NLP Models. In IJCAI. 458\u2013465. Pingchuan Ma Shuai Wang and Jin Liu. 2020. Metamorphic Testing and Certified Mitigation of Fairness Violations in NLP Models. In IJCAI. 458\u2013465.","DOI":"10.24963\/ijcai.2020\/64"},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380421"},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1002\/0471721182"},{"key":"e_1_3_2_1_64_1","volume-title":"Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.","author":"Mnih Volodymyr","year":"2016","unstructured":"Volodymyr Mnih , Adria Puigdomenech Badia , Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016 . Asynchronous methods for deep reinforcement learning. In ICML. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In ICML."},{"key":"e_1_3_2_1_65_1","unstructured":"Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602."},{"key":"e_1_3_2_1_66_1","doi-asserted-by":"crossref","unstructured":"Shin Nakajima and Tsong Yueh Chen. 2019. Generating biased dataset for metamorphic testing of machine learning programs. In IFIP-ICTSS. Shin Nakajima and Tsong Yueh Chen. 2019. Generating biased dataset for metamorphic testing of machine learning programs. In IFIP-ICTSS.","DOI":"10.1007\/978-3-030-31280-0_4"},{"key":"e_1_3_2_1_67_1","volume-title":"Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv:1807.10875.","author":"Odena Augustus","year":"2018","unstructured":"Augustus Odena and Ian Goodfellow . 2018 . Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv:1807.10875. Augustus Odena and Ian Goodfellow. 2018. Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv:1807.10875."},{"key":"e_1_3_2_1_68_1","unstructured":"Wesley A Olson. 2015. Airborne collision avoidance system x. MASSACHUSETTS INST OF TECH LEXINGTON LINCOLN LAB. Wesley A Olson. 2015. Airborne collision avoidance system x. MASSACHUSETTS INST OF TECH LEXINGTON LINCOLN LAB."},{"key":"e_1_3_2_1_69_1","volume-title":"A course in game theory","author":"Osborne Martin J","unstructured":"Martin J Osborne and Ariel Rubinstein . 1994. A course in game theory . MIT press . Martin J Osborne and Ariel Rubinstein. 1994. A course in game theory. MIT press."},{"key":"e_1_3_2_1_70_1","unstructured":"Anay Pattanaik Zhenyi Tang Shuijing Liu Gautham Bommannan and Girish Chowdhary. 2017. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632. Anay Pattanaik Zhenyi Tang Shuijing Liu Gautham Bommannan and Girish Chowdhary. 2017. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632."},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"crossref","unstructured":"Kexin Pei Yinzhi Cao Junfeng Yang and Suman Jana. 2017. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. SOSP. Kexin Pei Yinzhi Cao Junfeng Yang and Suman Jana. 2017. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. SOSP.","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_1_72_1","doi-asserted-by":"crossref","unstructured":"Wei Qiu Haipeng Chen and Bo An. 2019. Dynamic Electronic Toll Collection via Multi-Agent Deep Reinforcement Learning with Edge-Based Graph Convolutional Networks. In IJCAI. Wei Qiu Haipeng Chen and Bo An. 2019. Dynamic Electronic Toll Collection via Multi-Agent Deep Reinforcement Learning with Edge-Based Graph Convolutional Networks. In IJCAI.","DOI":"10.24963\/ijcai.2019\/635"},{"key":"e_1_3_2_1_73_1","unstructured":"Antonin Raffin. 2020. RL Baselines3 Zoo. https:\/\/github.com\/DLR-RM\/rl-baselines3-zoo Antonin Raffin. 2020. RL Baselines3 Zoo. https:\/\/github.com\/DLR-RM\/rl-baselines3-zoo"},{"key":"e_1_3_2_1_74_1","volume-title":"Applied factor analysis","author":"Rummel Rudolf J","unstructured":"Rudolf J Rummel . 1988. Applied factor analysis . Northwestern University Press . Rudolf J Rummel. 1988. Applied factor analysis. Northwestern University Press."},{"key":"e_1_3_2_1_75_1","unstructured":"John Schulman Filip Wolski Prafulla Dhariwal Alec Radford and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. John Schulman Filip Wolski Prafulla Dhariwal Alec Radford and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347."},{"key":"e_1_3_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_1_77_1","volume-title":"Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, and Marc Lanctot.","author":"Silver David","year":"2016","unstructured":"David Silver , Aja Huang , Chris J Maddison , Arthur Guez , Laurent Sifre , George Van Den Driessche , Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, and Marc Lanctot. 2016 . Mastering the game of Go with deep neural networks and tree search. nature, 529, 7587 (2016), 484\u2013489. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, and Marc Lanctot. 2016. Mastering the game of Go with deep neural networks and tree search. nature, 529, 7587 (2016), 484\u2013489."},{"key":"e_1_3_2_1_78_1","unstructured":"Youcheng Sun Xiaowei Huang Daniel Kroening James Sharp Matthew Hill and Rob Ashmore. 2018. Testing deep neural networks. arXiv preprint arXiv:1803.04792. Youcheng Sun Xiaowei Huang Daniel Kroening James Sharp Matthew Hill and Rob Ashmore. 2018. Testing deep neural networks. arXiv preprint arXiv:1803.04792."},{"key":"e_1_3_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_2_1_80_1","doi-asserted-by":"crossref","unstructured":"Yuchi Tian Kexin Pei Suman Jana and Baishakhi Ray. 2018. DeepTest: Automated Testing of Deep-neural-network-driven Autonomous Cars. ICSE \u201918. Yuchi Tian Kexin Pei Suman Jana and Baishakhi Ray. 2018. DeepTest: Automated Testing of Deep-neural-network-driven Autonomous Cars. ICSE \u201918.","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_1_81_1","doi-asserted-by":"crossref","unstructured":"Marin Toromanoff Emilie Wirbel and Fabien Moutarde. 2020. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In CVPR. Marin Toromanoff Emilie Wirbel and Fabien Moutarde. 2020. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In CVPR.","DOI":"10.1109\/CVPR42600.2020.00718"},{"key":"e_1_3_2_1_82_1","doi-asserted-by":"crossref","unstructured":"Sakshi Udeshi Pryanshu Arora and Sudipta Chattopadhyay. 2018. Automated Directed Fairness Testing. ASE. Sakshi Udeshi Pryanshu Arora and Sudipta Chattopadhyay. 2018. Automated Directed Fairness Testing. ASE.","DOI":"10.1145\/3238147.3238165"},{"key":"e_1_3_2_1_83_1","unstructured":"Jonathan Uesato Ananya Kumar Csaba Szepesvari Tom Erez Avraham Ruderman Keith Anderson Nicolas Heess and Pushmeet Kohli. 2018. Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. arXiv preprint arXiv:1812.01647. Jonathan Uesato Ananya Kumar Csaba Szepesvari Tom Erez Avraham Ruderman Keith Anderson Nicolas Heess and Pushmeet Kohli. 2018. Rigorous agent evaluation: An adversarial approach to uncover catastrophic failures. arXiv preprint arXiv:1812.01647."},{"key":"e_1_3_2_1_84_1","article-title":"Visualizing data using t-SNE","volume":"9","author":"der Maaten Laurens Van","year":"2008","unstructured":"Laurens Van der Maaten and Geoffrey Hinton . 2008 . Visualizing data using t-SNE .. Journal of machine learning research , 9 , 11 (2008). Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE.. Journal of machine learning research, 9, 11 (2008).","journal-title":"Journal of machine learning research"},{"key":"e_1_3_2_1_85_1","volume-title":"Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575, 7782","author":"Vinyals Oriol","year":"2019","unstructured":"Oriol Vinyals , Igor Babuschkin , Wojciech M Czarnecki , Micha\u00ebl Mathieu , Andrew Dudzik , Junyoung Chung , David H Choi , Richard Powell , Timo Ewalds , and Petko Georgiev . 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575, 7782 ( 2019 ), 350\u2013354. Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Micha\u00ebl Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, and Petko Georgiev. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575, 7782 (2019), 350\u2013354."},{"key":"e_1_3_2_1_86_1","unstructured":"Shiqi Wang Kexin Pei Justin Whitehouse Junfeng Yang and Suman Jana. 2018. Formal security analysis of neural networks using symbolic intervals. In USENIX Security 18. Shiqi Wang Kexin Pei Justin Whitehouse Junfeng Yang and Suman Jana. 2018. Formal security analysis of neural networks using symbolic intervals. In USENIX Security 18."},{"key":"e_1_3_2_1_87_1","doi-asserted-by":"crossref","unstructured":"Shuai Wang and Zhendong Su. 2020. Metamorphic Object Insertion for Testing Object Detection Systems. In ASE. Shuai Wang and Zhendong Su. 2020. Metamorphic Object Insertion for Testing Object Detection Systems. In ASE.","DOI":"10.1145\/3324884.3416584"},{"key":"e_1_3_2_1_88_1","unstructured":"Marco A Wiering. 2000. Multi-agent reinforcement learning for traffic light control. In ICML. Marco A Wiering. 2000. Multi-agent reinforcement learning for traffic light control. In ICML."},{"key":"e_1_3_2_1_89_1","volume-title":"Principal component analysis. Chemometrics and intelligent laboratory systems, 2, 1-3","author":"Wold Svante","year":"1987","unstructured":"Svante Wold , Kim Esbensen , and Paul Geladi . 1987. Principal component analysis. Chemometrics and intelligent laboratory systems, 2, 1-3 ( 1987 ), 37\u201352. Svante Wold, Kim Esbensen, and Paul Geladi. 1987. Principal component analysis. Chemometrics and intelligent laboratory systems, 2, 1-3 (1987), 37\u201352."},{"key":"e_1_3_2_1_90_1","unstructured":"Chaowei Xiao Xinlei Pan Warren He Jian Peng Mingjie Sun Jinfeng Yi Mingyan Liu Bo Li and Dawn Song. 2019. Characterizing attacks on deep reinforcement learning. arXiv preprint arXiv:1907.09470. Chaowei Xiao Xinlei Pan Warren He Jian Peng Mingjie Sun Jinfeng Yi Mingyan Liu Bo Li and Dawn Song. 2019. Characterizing attacks on deep reinforcement learning. arXiv preprint arXiv:1907.09470."},{"key":"e_1_3_2_1_91_1","unstructured":"Xiaofei Xie Lei Ma Felix Juefei-Xu Minhui Xue Hongxu Chen Yang Liu Jianjun Zhao Bo Li Jianxiong Yin and Simon See. 2019. Deephunter: a coverage-guided fuzz testing framework for deep neural networks. In ISSTA. Xiaofei Xie Lei Ma Felix Juefei-Xu Minhui Xue Hongxu Chen Yang Liu Jianjun Zhao Bo Li Jianxiong Yin and Simon See. 2019. Deephunter: a coverage-guided fuzz testing framework for deep neural networks. In ISSTA."},{"key":"e_1_3_2_1_92_1","unstructured":"Xiaofei Xie Lei Ma Haijun Wang Yuekang Li Yang Liu and Xiaohong Li. 2019. DiffChaser: Detecting Disagreements for Deep Neural Networks. In IJCAI. Xiaofei Xie Lei Ma Haijun Wang Yuekang Li Yang Liu and Xiaohong Li. 2019. DiffChaser: Detecting Disagreements for Deep Neural Networks. In IJCAI."},{"key":"e_1_3_2_1_93_1","article-title":"Falsification of cyber-physical systems using deep reinforcement learning","author":"Yamagata Yoriyuki","year":"2020","unstructured":"Yoriyuki Yamagata , Shuang Liu , Takumi Akazaki , Yihai Duan , and Jianye Hao . 2020 . Falsification of cyber-physical systems using deep reinforcement learning . IEEE Transactions on Software Engineering. Yoriyuki Yamagata, Shuang Liu, Takumi Akazaki, Yihai Duan, and Jianye Hao. 2020. Falsification of cyber-physical systems using deep reinforcement learning. IEEE Transactions on Software Engineering.","journal-title":"IEEE Transactions on Software Engineering."},{"key":"e_1_3_2_1_94_1","doi-asserted-by":"crossref","unstructured":"Wei Yang Yuqing Xie Luchen Tan Kun Xiong Ming Li and Jimmy Lin. 2019. Data augmentation for bert fine-tuning in open-domain question answering. arXiv preprint arXiv:1904.06652. Wei Yang Yuqing Xie Luchen Tan Kun Xiong Ming Li and Jimmy Lin. 2019. Data augmentation for bert fine-tuning in open-domain question answering. arXiv preprint arXiv:1904.06652.","DOI":"10.18653\/v1\/N19-4013"},{"key":"e_1_3_2_1_95_1","unstructured":"Yuanyuan Yuan Qi Pang and Shuai Wang. 2021. Enhancing Deep Neural Networks Testing by Traversing Data Manifold. arXiv preprint arXiv:2112.01956. Yuanyuan Yuan Qi Pang and Shuai Wang. 2021. Enhancing Deep Neural Networks Testing by Traversing Data Manifold. arXiv preprint arXiv:2112.01956."},{"key":"e_1_3_2_1_96_1","unstructured":"Yuanyuan Yuan Qi Pang and Shuai Wang. 2021. You Can\u2019t See the Forest for Its Trees: Assessing Deep Neural Network Testing via NeuraL Coverage. arXiv preprint arXiv:2112.01955. Yuanyuan Yuan Qi Pang and Shuai Wang. 2021. You Can\u2019t See the Forest for Its Trees: Assessing Deep Neural Network Testing via NeuraL Coverage. arXiv preprint arXiv:2112.01955."},{"key":"e_1_3_2_1_97_1","volume-title":"Perception Matters: Detecting Perception Failures of VQA Models Using Metamorphic Testing. In CVPR.","author":"Yuan Yuanyuan","year":"2021","unstructured":"Yuanyuan Yuan , Shuai Wang , Mingyue Jiang , and Tsong Yueh Chen . 2021 . Perception Matters: Detecting Perception Failures of VQA Models Using Metamorphic Testing. In CVPR. Yuanyuan Yuan, Shuai Wang, Mingyue Jiang, and Tsong Yueh Chen. 2021. Perception Matters: Detecting Perception Failures of VQA Models Using Metamorphic Testing. In CVPR."},{"key":"e_1_3_2_1_98_1","unstructured":"Micha\u0142 Zalewski. 2021. American Fuzzy Lop. https:\/\/lcamtuf.coredump.cx\/afl\/ Micha\u0142 Zalewski. 2021. American Fuzzy Lop. https:\/\/lcamtuf.coredump.cx\/afl\/"},{"key":"e_1_3_2_1_99_1","volume-title":"Machine learning testing: Survey, landscapes and horizons","author":"Zhang Jie M","unstructured":"Jie M Zhang , Mark Harman , Lei Ma , and Yang Liu . 2020. Machine learning testing: Survey, landscapes and horizons . IEEE TSE. Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE TSE."},{"key":"e_1_3_2_1_100_1","doi-asserted-by":"crossref","unstructured":"Mengshi Zhang Yuqun Zhang Lingming Zhang Cong Liu and Sarfraz Khurshid. 2018. DeepRoad: GAN-based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems. In ASE. Mengshi Zhang Yuqun Zhang Lingming Zhang Cong Liu and Sarfraz Khurshid. 2018. DeepRoad: GAN-based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems. In ASE.","DOI":"10.1145\/3238147.3238187"}],"event":{"name":"ISSTA '22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis","location":"Virtual South Korea","acronym":"ISSTA '22","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533767.3534388","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3533767.3534388","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T18:43:41Z","timestamp":1750272221000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533767.3534388"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,18]]},"references-count":99,"alternative-id":["10.1145\/3533767.3534388","10.1145\/3533767"],"URL":"https:\/\/doi.org\/10.1145\/3533767.3534388","relation":{},"subject":[],"published":{"date-parts":[[2022,7,18]]},"assertion":[{"value":"2022-07-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}