{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,5]],"date-time":"2026-07-05T09:15:49Z","timestamp":1783242949359,"version":"3.54.6"},"publisher-location":"New York, NY, USA","reference-count":55,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,7,5]],"date-time":"2026-07-05T00:00:00Z","timestamp":1783209600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"InnoGuard Marie Sk\u0142odowska-Curie Doctoral Network","award":["Grant Agreement No. 101169233"],"award-info":[{"award-number":["Grant Agreement No. 101169233"]}]},{"name":"ASPIRE","award":["Grant No. JPMJAP2301, JST"],"award-info":[{"award-number":["Grant No. JPMJAP2301, JST"]}]},{"name":"Formation of Non-Doctoral Research Staff of the Education Department of the Basque Government","award":["Grant n. PRE_2025_2_025"],"award-info":[{"award-number":["Grant n. PRE_2025_2_025"]}]},{"name":"Department of Education, Universities and Research of the Basque Country","award":["IT1519-22"],"award-info":[{"award-number":["IT1519-22"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,7,5]]},"DOI":"10.1145\/3805760.3814910","type":"proceedings-article","created":{"date-parts":[[2026,7,5]],"date-time":"2026-07-05T08:46:13Z","timestamp":1783241173000},"page":"197-207","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["VISOR: A Vision-Language Model-Based Test Oracle for Testing Robots"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-7139-1043","authenticated-orcid":false,"given":"Prasun","family":"Saurabh","sequence":"first","affiliation":[{"name":"Simula Research Laboratory, Department of Engineering Complex Software Systems, Oslo, Norway"},{"name":"Oslo Metropolitan University, Oslo, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0588-316X","authenticated-orcid":false,"given":"Pablo","family":"Valle","sequence":"additional","affiliation":[{"name":"Mondragon University, Mondrag\u00f3n, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7507-5080","authenticated-orcid":false,"given":"Aitor","family":"Arrieta","sequence":"additional","affiliation":[{"name":"Mondragon University, Mondrag\u00f3n, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9979-3519","authenticated-orcid":false,"given":"Shaukat","family":"Ali","sequence":"additional","affiliation":[{"name":"Simula Research Laboratory, Oslo, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6253-4062","authenticated-orcid":false,"given":"Paolo","family":"Arcaini","sequence":"additional","affiliation":[{"name":"National Institute of Informatics, Tokyo, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,5]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","unstructured":"Niket Agarwal Arslan Ali Maciej Bala Yogesh Balaji Erik Barker Tiffany Cai Prithvijit Chattopadhyay Yongxin Chen Yin Cui Yifan Ding Daniel Dworakowski Jiaojiao Fan Michele Fenzi Francesco Ferroni Sanja Fidler Dieter Fox Songwei Ge Yunhao Ge Jinwei Gu Siddharth Gururani Ethan He Jiahui Huang Jacob Samuel Huffman Pooya Jannaty Jingyi Jin Seung Wook Kim Gergely Kl\u00e1r Grace Lam Shiyi Lan Laura Leal-Taix\u00e9 Anqi Li Zhaoshuo Li Chen-Hsuan Lin Tsung-Yi Lin Huan Ling Ming-Yu Liu Xian Liu Alice Luo Qianli Ma Hanzi Mao Kaichun Mo Arsalan Mousavian Seungjun Nah Sriharsha Niverty David Page Despoina Paschalidou Zeeshan Patel Lindsey Pavao Morteza Ramezanali Fitsum Reda Xiaowei Ren Vasanth Rao Naik Sabavat Ed Schmerling Stella Shi Bartosz Stefaniak Shitao Tang Lyne Tchapmi Przemek Tredak Wei-Cheng Tseng Jibin Varghese Hao Wang Haoxiang Wang Heng Wang Ting-Chun Wang Fangyin Wei Xinyue Wei Jay Zhangjie Wu Jiashu Xu Wei Yang Yen-Chen Lin Xiaohui Zeng Yu Zeng Jing Zhang Qinsheng Zhang Yuxuan Zhang Qingqing Zhao and Artur Z\u00f3lkowski. 2025. Cosmos World Foundation Model Platform for Physical AI. CoRR abs\/2501.03575 (2025). arXiv:2501.03575 doi:10.48550\/ARXIV.2501.03575 10.48550\/ARXIV.2501.03575","DOI":"10.48550\/ARXIV.2501.03575"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985795"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.rcim.2024.102862"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai"},{"key":"e_1_3_2_1_5_1","volume-title":"Cyber-physical systems. The impact of control technology 12, 1","author":"Baheti Radhakisan","year":"2011","unstructured":"Radhakisan Baheti and Helen Gill. 2011. Cyber-physical systems. The impact of control technology 12, 1 (2011), 161-166."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_2_1_7_1","volume-title":"Pattern Recognition and Machine Learning","author":"Bishop Christopher M.","unstructured":"Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2503.14734"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2410.24164"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.01370"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA57147.2024.10611163"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2011.2160929"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-018-0921-x"},{"key":"e_1_3_2_1_14_1","volume-title":"Proceedings of The 2nd Conference on Lifelong Learning Agents (Proceedings of Machine Learning Research","volume":"136","author":"Du Yuqing","year":"2023","unstructured":"Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. 2023. Vision-Language Models as Success Detectors. In Proceedings of The 2nd Conference on Lifelong Learning Agents (Proceedings of Machine Learning Research, Vol. 232), Sarath Chandar, Razvan Pascanu, Hanie Sedghi, and Doina Precup (Eds.). PMLR, 120-136. https: \/\/proceedings.mlr.press\/v232\/du23b.html"},{"key":"e_1_3_2_1_15_1","volume-title":"AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation. In The Thirteenth International Conference on Learning Representations, ICLR 2025","author":"Duan Jiafei","year":"2025","unstructured":"Jiafei Duan, Wilbert Pumacay, Nishanth Kumar, Yi Ru Wang, Shulin Tian, Wentao Yuan, Ranjay Krishna, Dieter Fox, Ajay Mandlekar, and Yijie Guo. 2025. AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https:\/\/openreview.net\/forum?id=JVkdSi7Ekg"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397357"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2507.06261"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS58592.2024.10802284"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.3389\/FRAI.2024.1430984"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.2974707"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCPS.2018.00035"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS"},{"key":"e_1_3_2_1_23_1","volume-title":"Proceedings of The 8th Conference on Robot Learning (Proceedings of Machine Learning Research","volume":"2713","author":"Kim Moo Jin","year":"2025","unstructured":"Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. 2025. OpenVLA: An Open-Source Vision- Language-Action Model. In Proceedings of The 8th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 270), Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard (Eds.). PMLR, 2679-2713. https:\/\/proceedings.mlr. press\/v270\/kim25c.html"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2405.17418"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2407.07895"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS60139.2025.11246030"},{"key":"e_1_3_2_1_27_1","volume-title":"Proceedings of The 8th Conference on Robot Learning (Proceedings of Machine Learning Research","volume":"3728","author":"Li Xuanlin","year":"2025","unstructured":"Xuanlin Li, Kyle Hsu, Jiayuan Gu, Oier Mees, Karl Pertsch, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. 2025. Evaluating Real-World Robot Manipulation Policies in Simulation. In Proceedings of The 8th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 270), Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard (Eds.). PMLR, 3705-3728. https:\/\/proceedings.mlr.press\/v270\/li25c.html"},{"key":"e_1_3_2_1_28_1","volume-title":"Proceedings of the 37th International Conference on Neural Information Processing Systems","author":"Liu Bo","year":"2023","unstructured":"Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. 2023. LIBERO: benchmarking knowledge transfer for lifelong robot learning. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 1939, 16 pages."},{"key":"e_1_3_2_1_29_1","volume-title":"Proceedings of the 37th International Conference on Neural Information Processing Systems","author":"Liu Haotian","year":"2023","unstructured":"Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 1516, 25 pages."},{"key":"e_1_3_2_1_30_1","volume-title":"Proceedings of The 7th Conference on Robot Learning (Proceedings of Machine Learning Research","volume":"3484","author":"Liu Zeyi","year":"2023","unstructured":"Zeyi Liu, Arpit Bahety, and Shuran Song. 2023. REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction. In Proceedings of The 7th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 229), Jie Tan, Marc Toussaint, and Kourosh Darvish (Eds.). PMLR, 3468-3484. https: \/\/proceedings.mlr.press\/v229\/liu23g.html"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2505"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2501"},{"key":"e_1_3_2_1_33_1","first-page":"16","article-title":"Summary and analysis of extension program evaluation in R. Rutgers Cooperative Extension: New Brunswick","volume":"125","author":"Mangiafico Salvatore S","year":"2016","unstructured":"Salvatore S Mangiafico. 2016. Summary and analysis of extension program evaluation in R. Rutgers Cooperative Extension: New Brunswick, NJ, USA 125 (2016), 16-22.","journal-title":"NJ, USA"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2504.05299"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cirp.2016.06.005"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2024.XX.050"},{"key":"e_1_3_2_1_37_1","unstructured":"OpenAI. 2023. GPT-4 Technical Report. CoRR abs\/2303.08774 (2023). arXiv:2303.08774 doi:10.48550\/ARXIV.2303.08774 10.48550\/ARXIV.2303.08774"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41526-021-00183-3"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2510.16263"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2501.15830"},{"key":"e_1_3_2_1_41_1","volume-title":"Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"8763","author":"Radford Alec","year":"2021","unstructured":"Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 8748-8763. https:\/\/proceedings.mlr.press\/v139\/radford21a.html"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3099287"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2402.07927"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","unstructured":"Prasun Saurabh Pablo Valle Aitor Arrieta Shaukat Ali and Paolo Arcaini. 2026. Supplementary material for the paper \"VISOR: A Vision-Language Model-based Test Oracle for Testing Robots\". https:\/\/doi.org\/10.5281\/zenodo.19680486 10.5281\/zenodo.19680486","DOI":"10.5281\/zenodo.19680486"},{"key":"e_1_3_2_1_45_1","volume-title":"Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders. In The Thirteenth International Conference on Learning Representations, ICLR 2025","author":"Shi Min","year":"2025","unstructured":"Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. 2025. Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https:\/\/openreview.net\/forum?id= Y2RW9EVwhT"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2024.3511409"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2502.14786"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2507.17049"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2205.14100"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3729343"},{"key":"e_1_3_2_1_51_1","unstructured":"An Yang Anfeng Li Baosong Yang Beichen Zhang Binyuan Hui Bo Zheng Bowen Yu Chang Gao Chengen Huang Chenxu Lv Chujie Zheng Dayiheng Liu Fan Zhou Fei Huang Feng Hu Hao Ge Haoran Wei Huan Lin Jialong Tang Jian Yang Jianhong Tu Jianwei Zhang Jian Yang Jiaxi Yang Jingren Zhou Junyang Lin Kai Dang Keqin Bao Kexin Yang Le Yu Lianghao Deng Mei Li Mingfeng Xue Mingze Li Pei Zhang Peng Wang Qin Zhu Rui Men Ruize Gao Shixuan Liu Shuang Luo Tianhao Li Tianyi Tang Wenbiao Yin Xingzhang Ren Xinyu Wang Xinyu Zhang Xuancheng Ren Yang Fan Yang Su Yichang Zhang Yinger Zhang Yu Wan Yuqiong Liu Zekun Wang Zeyu Cui Zhenru Zhang Zhipeng Zhou and Zihan Qiu. 2025. Qwen3 Technical Report. CoRR abs\/2505.09388 (2025). arXiv:2505.09388 doi:10.48550\/ARXIV.2505.09388 10.48550\/ARXIV.2505.09388"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2024.3369699"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2412.18194"},{"key":"e_1_3_2_1_54_1","volume-title":"Proceedings of the Computer Vision and Pattern Recognition Conference. 6919-6929","author":"Zhou Enshen","year":"2025","unstructured":"Enshen Zhou, Qi Su, Cheng Chi, Zhizheng Zhang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, and He Wang. 2025. Code-as-monitor: Constraint-aware visual programming for reactive and proactive robotic failure detection. In Proceedings of the Computer Vision and Pattern Recognition Conference. 6919-6929."},{"key":"e_1_3_2_1_55_1","volume-title":"Proceedings of The 7th Conference on Robot Learning (Proceedings of Machine Learning Research","volume":"2183","author":"Zitkovich Brianna","year":"2023","unstructured":"Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Reymann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakrishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. 2023. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. In Proceedings of The 7th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 229), Jie Tan, Marc Toussaint, and Kourosh Darvish (Eds.). PMLR, 2165-2183. https:\/\/proceedings.mlr.press\/v229\/zitkovich23a.html"}],"event":{"name":"AIware '26: 3rd ACM International Conference on AI-Powered Software","location":"Montreal QC Canada","acronym":"AIware '26","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 3rd ACM International Conference on AI-Powered Software"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3805760.3814910","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,5]],"date-time":"2026-07-05T08:49:19Z","timestamp":1783241359000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3805760.3814910"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,5]]},"references-count":55,"alternative-id":["10.1145\/3805760.3814910","10.1145\/3805760"],"URL":"https:\/\/doi.org\/10.1145\/3805760.3814910","relation":{},"subject":[],"published":{"date-parts":[[2026,7,5]]},"assertion":[{"value":"2026-07-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}