{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T13:26:58Z","timestamp":1792762018953,"version":"4.5.0"},"reference-count":99,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:00:00Z","timestamp":1750291200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,19]]},"abstract":"<jats:p>\n                    The rapid advancement of generative Al and multi-modal foundation models has shown significant potential in advancing robotic manipulation. Vision-language-action (VLA) models, in particular, have emerged as a promising approach for visuomotor control by leveraging large-scale vision-language data and robot demonstrations. However, current VLA models are typically evaluated using a limited set of hand-crafted scenes, leaving their general performance and robustness in diverse scenarios largely unexplored. To address this gap, we present\n                    <jats:sc>VLATest,<\/jats:sc>\n                    a fuzzing framework designed to generate robotic manipulation scenes for testing VLA models. Based on\n                    <jats:sc>VLATest,<\/jats:sc>\n                    we conducted an empirical study to assess the performance of seven representative VLA models. Our study results revealed that current VLA models lack the robustness necessary for practical deployment. Additionally, we investigated the impact of various factors, including the number of confounding objects, lighting conditions, camera poses, unseen objects, and task instruction mutations, on the VLA model\u2019s performance. Our findings highlight the limitations of existing VLA models, emphasizing the need for further research to develop reliable and trustworthy VLA applications.\n                  <\/jats:p>","DOI":"10.1145\/3729343","type":"journal-article","created":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T11:15:34Z","timestamp":1750331734000},"page":"1615-1638","source":"Crossref","is-referenced-by-count":15,"title":["VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4559-5426","authenticated-orcid":false,"given":"Zhijie","family":"Wang","sequence":"first","affiliation":[{"name":"University of Alberta, Edmonton, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9542-4858","authenticated-orcid":false,"given":"Zhehua","family":"Zhou","sequence":"additional","affiliation":[{"name":"University of Alberta, Edmonton, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-7093-9781","authenticated-orcid":false,"given":"Jiayang","family":"Song","sequence":"additional","affiliation":[{"name":"University of Alberta, Edmonton, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3666-4020","authenticated-orcid":false,"given":"Yuheng","family":"Huang","sequence":"additional","affiliation":[{"name":"The University of Tokyo, Tokyo, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5933-254X","authenticated-orcid":false,"given":"Zhan","family":"Shu","sequence":"additional","affiliation":[{"name":"University of Alberta, Edmonton, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8621-2420","authenticated-orcid":false,"given":"Lei","family":"Ma","sequence":"additional","affiliation":[{"name":"The University of Tokyo, Tokyo, Japan"},{"name":"University of Alberta, Edmonton, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,19]]},"reference":[{"key":"e_1_3_1_1_2","doi-asserted-by":"publisher","DOI":"10.1145\/3644388"},{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.3390\/jimaging9020046"},{"key":"e_1_3_1_3_2","doi-asserted-by":"crossref","unstructured":"Jon Ayerdi Valerio Terragni. 2021. Generating metamorphic relations for cyber-physical systems with genetic programming: an industrial case study. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1264\u20131274.","DOI":"10.1145\/3468264.3473920"},{"key":"e_1_3_1_4_2","unstructured":"Yuntao Bai Andy Jones Kamal Ndousse. 2022. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2204.05862"},{"key":"e_1_3_1_5_2","doi-asserted-by":"crossref","unstructured":"Homanga Bharadhwaj Jay Vakil Mohit Sharma Abhinav Gupta Shubham Tulsiani Vikash Kumar. 2024. RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking. In 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE 4788\u20134795.","DOI":"10.1109\/ICRA57147.2024.10611293"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3631970"},{"key":"e_1_3_1_7_2","unstructured":"Anthony Brohan Noah Brown Justice Carbajal. 2022. RT-1: Robotics Transformer for Real-World Control at Scale. arXiv preprint arXiv:2212.06817"},{"key":"e_1_3_1_8_2","unstructured":"Anthony Brohan Noah Brown Justice Carbajal. 2023. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv preprint arXiv:2307.15818(2023)."},{"key":"e_1_3_1_9_2","doi-asserted-by":"crossref","unstructured":"Berk Calli Arjun Singh Aaron Walsman Siddhartha Srinivasa Pieter Abbeel Aaron M Dollar. 2015. The YCB object and Model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR). IEEE 510\u2013517.","DOI":"10.1109\/ICAR.2015.7251504"},{"issue":"7","key":"e_1_3_1_10_2","doi-asserted-by":"crossref","first-page":"3675","DOI":"10.1109\/TSE.2023.3267446","article-title":"MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation","volume":"49","author":"Cassano Federico","year":"2023","unstructured":"Federico Cassano, John Gouwar, Daniel Nguyen. 2023. MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation. IEEE Transactions on Software Engineering 49, 7 (2023), 3675\u20133691.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_1_11_2","unstructured":"Xinyun Chen Maxwell Lin Nathanael Sch\u00e4rli Denny Zhou. 2024. Teaching Large Language Models to Self-Debug. In The Twelfth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=KuPixIqPiq"},{"key":"e_1_3_1_12_2","doi-asserted-by":"crossref","unstructured":"Yuqi Chen Bohan Xuan Christopher M Poskitt Jun Sun Fan Zhang. 2020. Active Fuzzing for Testing and Securing Cyber-Physical Systems. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 14\u201326.","DOI":"10.1145\/3395363.3397376"},{"key":"e_1_3_1_13_2","doi-asserted-by":"crossref","unstructured":"Mingfei Cheng Yuan Zhou Xiaofei Xie. 2023. BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 488\u2013500.","DOI":"10.1145\/3597926.3598072"},{"key":"e_1_3_1_14_2","doi-asserted-by":"crossref","unstructured":"Xueqi Dang Yinghua Li Mike Papadakis Jacques Klein Tegawend\u00e9 F Bissyand\u00e9 Yves Le Traon. 2024. Test Input Prioritization for Machine Learning Classifiers. IEEE Transactions on Software Engineering (2024).","DOI":"10.1109\/TSE.2024.3350019"},{"key":"e_1_3_1_15_2","doi-asserted-by":"crossref","unstructured":"Amandeep Dhaliwal. 2020. The Rise of Automation and Robotics in Warehouse Management. In Transforming Management Using Artificial Intelligence Techniques. CRC Press 63\u201372.","DOI":"10.1201\/9781003032410-5"},{"key":"e_1_3_1_16_2","doi-asserted-by":"crossref","unstructured":"Yan Ding Xiaohan Zhang Chris Paxton Shiqi Zhang. 2023. Task and Motion Planning with Large Language Models for Object Rearrangement. In 2023 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE 2086\u20132092.","DOI":"10.1109\/IROS55552.2023.10342169"},{"key":"e_1_3_1_17_2","unstructured":"Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations."},{"key":"e_1_3_1_18_2","unstructured":"Danny Driess Fei Xia Mehdi S. M. Sajjadi. 2023. PaLM-E: An Embodied Multimodal Language Model. In Proceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research Vol. 202). PMLR 8469\u20138488."},{"key":"e_1_3_1_19_2","unstructured":"Yuqing Du Ksenia Konyushkova Misha Denil Akhil Raju Jessica Landon Felix Hill Nando de Freitas Serkan Cabi. 2023. Vision-Language Models as Success Detectors. In Proceedings of The 2nd Conference on Lifelong Learning Agents (Proceedings of Machine Learning Research Vol. 232). PMLR 120\u2013136."},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.3390\/app12010135"},{"issue":"2022","key":"e_1_3_1_21_2","doi-asserted-by":"crossref","first-page":"1473","DOI":"10.1162\/tacl_a_00529","article-title":"FaithDial: A Faithful Benchmark for Information-Seeking Dialogue","volume":"10","author":"Dziri Nouha","year":"2022","unstructured":"Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M Ponti, Siva Reddy. 2022. FaithDial: A Faithful Benchmark for Information-Seeking Dialogue. Transactions of the Association for Computational Linguistics 10 (2022), 1473\u20131490.","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"e_1_3_1_22_2","doi-asserted-by":"crossref","unstructured":"Yang Feng Qingkai Shi Xinyu Gao Jun Wan Chunrong Fang Zhenyu Chen. 2020. DeepGini: prioritizing massive tests to enhance the robustness of deep neural networks. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 177\u2013188.","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_1_23_2","doi-asserted-by":"crossref","unstructured":"Xinyu Gao Yang Feng Yining Yin Zixi Liu Zhenyu Chen Baowen Xu. 2022. Adaptive test selection for deep neural networks. In Proceedings of the 44th International Conference on Software Engineering. 73\u201385.","DOI":"10.1145\/3510003.3510232"},{"key":"e_1_3_1_24_2","doi-asserted-by":"crossref","unstructured":"Xinyu Gao Zhijie Wang Yang Feng Lei Ma Zhenyu Chen Baowen Xu. 2024. MultiTest: Physical-Aware Object Insertion for Testing Multi-sensor Fusion Perception Systems. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. 1\u201313.","DOI":"10.1145\/3597503.3639191"},{"key":"e_1_3_1_25_2","doi-asserted-by":"crossref","unstructured":"Ruchi Goel Pooja Gupta. 2020. Robotics and Industry 4.0. Springer International Publishing Cham 157\u2013169.","DOI":"10.1007\/978-3-030-14544-6_9"},{"key":"e_1_3_1_26_2","unstructured":"Jiayuan Gu Fanbo Xiang Xuanlin Li. 2023. ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills. arXiv preprint arXiv:2302.04659(2023)."},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3583136"},{"issue":"1","key":"e_1_3_1_28_2","first-page":"188","article-title":"CPS-Based Self-Adaptive Collaborative Control for Smart Production-Logistics Systems","volume":"51","author":"Guo Zhengang","year":"2020","unstructured":"Zhengang Guo, Yingfeng Zhang, Xibin Zhao, Xiaoyu Song. 2020. CPS-Based Self-Adaptive Collaborative Control for Smart Production-Logistics Systems. IEEE transactions on cybernetics 51, 1 (2020), 188\u2013198.","journal-title":"IEEE transactions on cybernetics"},{"key":"e_1_3_1_29_2","doi-asserted-by":"crossref","unstructured":"Aamir Hamid Hemanth Reddy Samidi Tim Finin Primal Pappachan Roberto Yus. 2023. GenAIPABench: A Benchmark for Generative AI-based Privacy Assistants. arXiv preprint arXiv:2309.05138 (2023).","DOI":"10.56553\/popets-2024-0081"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.3390\/s23073762"},{"key":"e_1_3_1_31_2","doi-asserted-by":"crossref","unstructured":"Pinjia He Clara Meister Zhendong Su. 2020. Structure-Invariant Testing for Machine Translation. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering. 961\u2013973.","DOI":"10.1145\/3377811.3380339"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/TASE.2021.3064065"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.3390\/robotics10010047"},{"key":"e_1_3_1_34_2","doi-asserted-by":"crossref","unstructured":"Qiang Hu Yuejun Guo Xiaofei Xie Maxime Cordy Mike Papadakis Lei Ma Yves Le Traon. 2023. Aries: Efficient Testing of Deep Neural Networks via Labeling-Free Accuracy Estimation. In 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). IEEE 1776\u20131787.","DOI":"10.1109\/ICSE48619.2023.00152"},{"key":"e_1_3_1_35_2","unstructured":"Zhisheng Hu Shengjian Guo Zhenyu Zhong Kang Li. 2021. Coverage-based Scene Fuzzing for Virtual Autonomous Driving Testing. arXiv preprint arXiv:2106.00873 (2021)."},{"key":"e_1_3_1_36_2","unstructured":"Siyuan Huang Zhengkai Jiang Hao Dong Yu Qiao Peng Gao Hongsheng Li. 2023. Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model. arXiv preprint arXiv:2305.11176 (2023)."},{"key":"e_1_3_1_37_2","unstructured":"Yuheng Huang Jiayang Song Qiang Hu Felix Juefei-Xu Lei Ma. 2024. Active Testing of Large Language Model via Multi-Stage Sampling. arXiv preprint arXiv:2408.03573 (2024)."},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3519464"},{"key":"e_1_3_1_39_2","unstructured":"Carlos E Jimenez John Yang Alexander Wettig Shunyu Yao Kexin Pei Ofir Press Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In The Twelfth International Conference on Learning Representations."},{"key":"e_1_3_1_40_2","unstructured":"Moo Jin Kim Karl Pertsch Siddharth Karamcheti. 2024. OpenVLA: An Open-Source Vision-Language-Action Model. arXiv preprint arXiv:2406.09246 (2024)."},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.3390\/technologies9010008"},{"key":"e_1_3_1_42_2","unstructured":"Jaeseong Lee Simin Chen Austin Mordahl Cong Liu Wei Yang Shiyi Wei. 2024. Automated Testing Linguistic Capabilities of NLP Models. ACM Transactions on Software Engineering and Methodology (2024)."},{"key":"e_1_3_1_43_2","doi-asserted-by":"crossref","unstructured":"Jaekwon Lee Enrico Vigan\u00f2 Oscar Cornejo Fabrizio Pastore Lionel Briand. 2023. Fuzzing for CPS Mutation Testing. In 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE 1377\u20131389.","DOI":"10.1109\/ASE56229.2023.00079"},{"key":"e_1_3_1_44_2","unstructured":"Xuanlin Li Kyle Hsu Jiayuan Gu. 2024. Evaluating Real-World Robot Manipulation Policies in Simulation. In 8th Annual Conference on Robot Learning."},{"key":"e_1_3_1_45_2","unstructured":"Xinghang Li Minghuan Liu Hanbo Zhang. 2024. Vision-Language Foundation Models as Effective Robot Imitators. In The Twelfth International Conference on Learning Representations."},{"key":"e_1_3_1_46_2","unstructured":"Xiang Li Cristina Mata Jongwoo Park. 2025. LLaRA: Supercharging Robot Learning Data for Vision-Language Policy. In The Thirteenth International Conference on Learning Representations."},{"issue":"5","key":"e_1_3_1_47_2","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1007\/s10664-024-10520-1","article-title":"Prioritizing test cases for deep learning-based video classifiers","volume":"29","author":"Li Yinghua","year":"2024","unstructured":"Yinghua Li, Xueqi Dang, Lei Ma, Jacques Klein, Tegawend\u00e9 F Bissyand\u00e9. 2024. Prioritizing test cases for deep learning-based video classifiers. Empirical Software Engineering 29, 5 (2024), 111.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_1_48_2","doi-asserted-by":"crossref","unstructured":"Jacky Liang Wenlong Huang. 2023. Code as Policies: Language Model Programs for Embodied Control. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE 9493\u20139500.","DOI":"10.1109\/ICRA48891.2023.10160591"},{"key":"e_1_3_1_49_2","unstructured":"Percy Liang Rishi Bommasani Tony Lee. 2023. Holistic Evaluation of Language Models. Transactions on Machine Learning Research (2023)."},{"key":"e_1_3_1_50_2","doi-asserted-by":"crossref","unstructured":"Stephanie Lin Jacob Hilton Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 3214\u20133252.","DOI":"10.18653\/v1\/2022.acl-long.229"},{"key":"e_1_3_1_51_2","unstructured":"Haotian Liu Chunyuan Li Qingyang Wu Yong Jae Lee. 2024. Visual Instruction Tuning. Advances in neural information processing systems 36 (2024)."},{"key":"e_1_3_1_52_2","unstructured":"Jiawei Liu Chunqiu Steven Xia Yuyao Wang Lingming Zhang. 2024. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_3_1_53_2","doi-asserted-by":"crossref","unstructured":"Claudio Menghi Shiva Nejati Lionel Briand Yago Isasi Parache. 2020. Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: An Approach Based on System Identification. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering. 372\u2013384.","DOI":"10.1145\/3377811.3380370"},{"key":"e_1_3_1_54_2","unstructured":"Subhabrata Mukherjee Ahmed Awadallah. 2020. Uncertainty-aware Self-training for Few-shot Text Classification. In Advances in Neural Information Processing Systems 33. Curran Associates Inc. 21199\u201321212."},{"key":"e_1_3_1_55_2","doi-asserted-by":"crossref","unstructured":"Richard M Murray Zexiang Li S Shankar Sastry. 2017. A Mathematical Introduction to Robotic Manipulation. CRC press.","DOI":"10.1201\/9781315136370"},{"key":"e_1_3_1_56_2","unstructured":"Suraj Nair Aravind Rajeswaran Vikash Kumar Chelsea Finn Abhinav Gupta. 2023. R3M: A Universal Visual Representation for Robot Manipulation. In Proceedings of The 6th Conference on Robot Learning (Proceedings of Machine Learning Research Vol. 205). PMLR 892\u2013909."},{"key":"e_1_3_1_57_2","unstructured":"Maxime Oquab Timoth\u00e9e Darcet Th\u00e9o Moutakanni. 2023. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research (2023)."},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.3390\/s23073625"},{"key":"e_1_3_1_59_2","unstructured":"Abhishek Padalkar Acorn Pooley Ajinkya Jain. 2023. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. arXiv preprint arXiv:2310.08864 (2023)."},{"key":"e_1_3_1_60_2","unstructured":"Rangeet Pan Ali Reza Ibrahimzada. 2024. Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering."},{"key":"e_1_3_1_61_2","doi-asserted-by":"crossref","unstructured":"Ethan Perez Florian Strub Harm De Vries Vincent Dumoulin Aaron Courville. 2018. FiLM: Visual Reasoning with a General Conditioning Layer. In Proceedings of the AAAI conference on artificial intelligence 32.","DOI":"10.1609\/aaai.v32i1.11671"},{"key":"e_1_3_1_62_2","unstructured":"Abu Rayhan. 2023. Artificial intelligence in robotics: From automation to autonomous systems."},{"issue":"2015","key":"e_1_3_1_63_2","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet Large Scale Visual Recognition Challenge","volume":"115","author":"Russakovsky Olga","year":"2015","unstructured":"Olga Russakovsky, Jia Deng, Hao Su. 2015. ImageNet Large Scale Visual Recognition Challenge. International journal of computer vision 115 (2015), 211\u2013252.","journal-title":"International journal of computer vision"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.5267\/j.uscm.2021.11.006"},{"key":"e_1_3_1_65_2","doi-asserted-by":"crossref","unstructured":"B Siciliano. 2008. Springer Handbook of Robotics. Springer.","DOI":"10.1007\/978-3-540-30301-5"},{"key":"e_1_3_1_66_2","doi-asserted-by":"crossref","unstructured":"Ishika Singh Valts Blukis Arsalan Mousavian. 2023. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. In 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE 11523\u201311530.","DOI":"10.1109\/ICRA48891.2023.10161317"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3282981"},{"key":"e_1_3_1_68_2","unstructured":"Jiayang Song Zhehua Zhou Jiawei Liu Chunrong Fang Zhan Shu Lei Ma. 2023. Self-Refined Large Language Model as Automated Reward Function Designer for Deep Reinforcement Learning in Robotics. arXiv preprint arXiv:2309.06687 (2023)."},{"issue":"2023","key":"e_1_3_1_69_2","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1016\/j.cogr.2023.04.001","article-title":"Artificial intelligence, machine learning and deep learning in advanced robotics, a review","volume":"3","author":"Soori Mohsen","year":"2023","unstructured":"Mohsen Soori, Behrooz Arezoo, Roza Dastres. 2023. Artificial intelligence, machine learning and deep learning in advanced robotics, a review. Cognitive Robotics 3 (2023), 54\u201370.","journal-title":"Cognitive Robotics"},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1002\/smr.2386"},{"key":"e_1_3_1_71_2","unstructured":"Austin Stone. 2023. Open-World Object Manipulation using Pre-trained Vision-Language Models. In Proceedings of The 7th Conference on Robot Learning (Proceedings of Machine Learning Research Vol. 229). PMLR 3397\u20133417."},{"key":"e_1_3_1_72_2","unstructured":"Lichao Sun Yue Huang. 2024. TrustLLM: Trustworthiness in Large Language Models. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research Vol. 235). PMLR 20166\u201320270."},{"key":"e_1_3_1_73_2","doi-asserted-by":"crossref","unstructured":"Zeyu Sun Jie M Zhang Mark Harman Mike Papadakis Lu Zhang. 2020. Automatic Testing and Improvement of Machine Translation. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering. 974\u2013985.","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_1_74_2","unstructured":"Mingxing Tan Quoc Le. 2019. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research Vol. 97) Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR 6105\u20136114."},{"key":"e_1_3_1_75_2","unstructured":"Octo Model Team Dibya Ghosh Homer Walke. 2024. Octo: An Open-Source Generalist Robot Policy. arXiv preprint arXiv:2405.12213 (2024)."},{"key":"e_1_3_1_76_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288 (2023)."},{"issue":"2023","key":"e_1_3_1_77_2","doi-asserted-by":"crossref","first-page":"100057","DOI":"10.1016\/j.ailsci.2023.100057","article-title":"Application of AI techniques and robotics in agriculture: A review","volume":"3","author":"Wakchaure Manas","year":"2023","unstructured":"Manas Wakchaure, BK Patle, AK Mahindrakar. 2023. Application of AI techniques and robotics in agriculture: A review. Artificial Intelligence in the Life Sciences 3 (2023), 100057.","journal-title":"Artificial Intelligence in the Life Sciences"},{"key":"e_1_3_1_78_2","doi-asserted-by":"crossref","unstructured":"Yuxuan Wan Wenxuan Wang Pinjia He Jiazhen Gu Haonan Bai Michael R Lyu. 2023. BiasAsker: Measuring the Bias in Conversational AI System. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 515\u2013527.","DOI":"10.1145\/3611643.3616310"},{"key":"e_1_3_1_79_2","doi-asserted-by":"crossref","unstructured":"Boxin Wang Weixin Chen Hengzhi Pei. 2023. DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","DOI":"10.52202\/075280-1361"},{"key":"e_1_3_1_80_2","unstructured":"Renzhi Wang Zhehua Zhou Jiayang Song Xuan Xie Xiaofei Xie Lei Ma. 2024. MORTAR: A Model-based Runtime Action Repair Framework for AI-enabled Cyber-Physical Systems. arXiv preprint arXiv:2408.03892 (2024)."},{"key":"e_1_3_1_81_2","doi-asserted-by":"crossref","unstructured":"Shuai Wang Zhendong Su. 2020. Metamorphic Object Insertion for Testing Object Detection Systems. In Proceedings of the 35th IEEE\/ACM International Conference on Automated Software Engineering. 1053\u20131065.","DOI":"10.1145\/3324884.3416584"},{"key":"e_1_3_1_82_2","doi-asserted-by":"crossref","unstructured":"Zhijie Wang Zijie Zhou Da Song Yuheng Huang Shengmai Chen Lei Ma Tianyi Zhang. 2025. Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models. In Proceedings of the IEEE\/ACM 47th International Conference on software Engineering (ICSE\u201925).","DOI":"10.1109\/ICSE55347.2025.00180"},{"key":"e_1_3_1_83_2","doi-asserted-by":"crossref","unstructured":"Jason Wei Xuezhi Wang Dale Schuurmans. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35. Curran Associates Inc. 24824\u201324837.","DOI":"10.52202\/068431-1800"},{"key":"e_1_3_1_84_2","unstructured":"Chunqiu Steven Xia Yinlin Deng Soren Dunn Lingming Zhang. 2024. Agentless: Demystifying LLM-based Software Engineering Agents. arXiv preprint arXiv:2407.01489 (2024)."},{"key":"e_1_3_1_85_2","doi-asserted-by":"crossref","unstructured":"Mingxuan Xiao Yan Xiao. 2023. LEAP: Efficient and Automated Test Method for NLP Software. In Proceedings of the 38th IEEE\/ACM International Conference on Automated Software Engineering. 1136\u20131148.","DOI":"10.1109\/ASE56229.2023.00052"},{"key":"e_1_3_1_86_2","unstructured":"Xuan Xie Jiayang Song Zhehua Zhou Yuheng Huang Da Song Lei Ma. 2024. Online Safety Analysis for LLMs: a Benchmark an Assessment and a Path Forward. arXiv preprint arXiv:2404.08517 (2024)."},{"key":"e_1_3_1_87_2","doi-asserted-by":"crossref","unstructured":"John Yang Carlos E. Jimenez. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems 37. Curran Associates Inc. 50528\u201350652.","DOI":"10.52202\/079017-1601"},{"key":"e_1_3_1_88_2","doi-asserted-by":"crossref","unstructured":"Shunyu Yao Dian Yu Jeffrey Zhao. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems 36. Curran Associates Inc. 11809\u201311822.","DOI":"10.52202\/075280-0517"},{"issue":"2020","key":"e_1_3_1_89_2","doi-asserted-by":"crossref","first-page":"151019","DOI":"10.1109\/ACCESS.2020.3016826","article-title":"Cyber-Physical Power System (CPPS): A Review on Modeling, Simulation, and Analysis With Cyber Security Applications","volume":"8","author":"Yohanandhan Rajaa Vikhram","year":"2020","unstructured":"Rajaa Vikhram Yohanandhan, Rajvikram Madurai Elavarasan. 2020. Cyber-Physical Power System (CPPS): A Review on Modeling, Simulation, and Analysis With Cyber Security Applications. IEEE Access 8 (2020), 151019\u2013151064.","journal-title":"IEEE Access"},{"key":"e_1_3_1_90_2","doi-asserted-by":"crossref","unstructured":"Boxi Yu Zhiqing Zhong Xinran Qin Jiayi Yao Yuancheng Wang Pinjia He. 2022. Automated testing of image captioning systems. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 467\u2013479.","DOI":"10.1145\/3533767.3534389"},{"key":"e_1_3_1_91_2","doi-asserted-by":"crossref","unstructured":"Hao Yu Bo Shen Dezhi Ran Jiaxin Zhang Qi Zhang Yuchi Ma Guangtai Liang Ying Li Qianxiang Wang Tao Xie. 2024. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. In Proceedings of the 46th IEEE\/ACM International Conference on Software Engineering. 1\u201312.","DOI":"10.1145\/3597503.3623316"},{"key":"e_1_3_1_92_2","doi-asserted-by":"crossref","unstructured":"Xiaohua Zhai Basil Mustafa Alexander Kolesnikov Lucas Beyer. 2023. Sigmoid Loss for Language Image Pre-Training. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 11975\u201311986.","DOI":"10.1109\/ICCV51070.2023.01100"},{"key":"e_1_3_1_93_2","doi-asserted-by":"crossref","unstructured":"Kechi Zhang Zhuo Li Jia Li Ge Li Zhi Jin. 2023. Self-Edit: Fault-Aware Code Editor for Code Generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics 769\u2013787.","DOI":"10.18653\/v1\/2023.acl-long.45"},{"key":"e_1_3_1_94_2","doi-asserted-by":"crossref","unstructured":"Yuntong Zhang Haifeng Ruan Zhiyu Fan Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1592\u20131604.","DOI":"10.1145\/3650212.3680384"},{"issue":"4","key":"e_1_3_1_95_2","doi-asserted-by":"crossref","first-page":"1842","DOI":"10.1109\/TSE.2022.3194640","article-title":"FalsifAI: Falsification of AI-Enabled Hybrid Control Systems Guided by Time-Aware Coverage Criteria","volume":"49","author":"Zhang Zhenya","year":"2022","unstructured":"Zhenya Zhang, Deyun Lyu, Paolo Arcaini, Lei Ma, Ichiro Hasuo, Jianjun Zhao. 2022. FalsifAI: Falsification of AI-Enabled Hybrid Control Systems Guided by Time-Aware Coverage Criteria. IEEE Transactions on Software Engineering 49, 4 (2022), 1842\u20131859.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_1_96_2","doi-asserted-by":"crossref","unstructured":"Wenshuai Zhao Jorge Pe\u00f1a Queralta Tomi Westerlund. 2020. Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE 737\u2013744.","DOI":"10.1109\/SSCI47803.2020.9308468"},{"key":"e_1_3_1_97_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3254142"},{"key":"e_1_3_1_98_2","doi-asserted-by":"crossref","unstructured":"Zhehua Zhou Jiayang Song Kunpeng Yao Zhan Shu Lei Ma. 2024. ISR-LLM: Iterative Self-Refined Large Language Model for Long-Horizon Sequential Task Planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE 2081\u20132088.","DOI":"10.1109\/ICRA57147.2024.10610065"},{"key":"e_1_3_1_99_2","doi-asserted-by":"crossref","unstructured":"Kaijie Zhu Jindong Wang Jiaheng Zhou. 2024. PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. In Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis (Salt Lake City UT USA) (LAMPS\u201924). 57\u201368.","DOI":"10.1145\/3689217.3690621"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3729343","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T11:05:16Z","timestamp":1787483116000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3729343"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,19]]},"references-count":99,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2025,6,19]]}},"alternative-id":["10.1145\/3729343"],"URL":"https:\/\/doi.org\/10.1145\/3729343","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,19]]}}}