{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T14:50:59Z","timestamp":1776783059221,"version":"3.51.2"},"reference-count":59,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,12,27]],"date-time":"2024-12-27T00:00:00Z","timestamp":1735257600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Basque Government through their Elkartek","award":["KK-2022\/00119, KK-2022\/00007"],"award-info":[{"award-number":["KK-2022\/00119, KK-2022\/00007"]}]},{"name":"Department of Education, Universities and Research of the Basque Country"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,1,31]]},"abstract":"<jats:p>\n            Autonomous driving systems (ADSs) are complex cyber-physical systems (CPSs) that must ensure safety even in uncertain conditions. Modern ADSs often employ deep neural networks (DNNs), which may not produce correct results in every possible driving scenario. Thus, an approach to estimate the confidence of an ADS at runtime is necessary to prevent potentially dangerous situations. In this article we propose\n            <jats:sc>MarMot<\/jats:sc>\n            , an online monitoring approach for ADSs based on metamorphic relations (MRs), which are properties of a system that hold among multiple inputs and the corresponding outputs. Using domain-specific MRs,\n            <jats:sc>MarMot<\/jats:sc>\n            estimates the uncertainty of the ADS at runtime, allowing the identification of anomalous situations that are likely to cause a faulty behavior of the ADS, such as driving off the road.\n          <\/jats:p>\n          <jats:p>\n            We perform an empirical assessment of\n            <jats:sc>MarMot<\/jats:sc>\n            with five different MRs, using two different subject ADSs, including a small-scale physical ADS and a simulated ADS. Our evaluation encompasses the identification of both external anomalies, e.g., fog, as well as internal anomalies, e.g., faulty DNNs due to mislabeled training data. Our results show that\n            <jats:sc>MarMot<\/jats:sc>\n            can identify up to 65% of the external anomalies and 100% of the internal anomalies in the physical ADS, and up to 54% of the external anomalies and 88% of the internal anomalies in the simulated ADS. With these results,\n            <jats:sc>MarMot<\/jats:sc>\n            outperforms or is comparable to other state-of-the-art approaches, including SelfOracle, Ensemble, and MC Dropout-based ADS monitors.\n          <\/jats:p>","DOI":"10.1145\/3678171","type":"journal-article","created":{"date-parts":[[2024,7,15]],"date-time":"2024-07-15T14:25:39Z","timestamp":1721053539000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["MarMot: Metamorphic Runtime Monitoring of Autonomous Driving Systems"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0491-9711","authenticated-orcid":false,"given":"Jon","family":"Ayerdi","sequence":"first","affiliation":[{"name":"Mondragon University, Mondragon, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1178-7362","authenticated-orcid":false,"given":"Asier","family":"Iriarte","sequence":"additional","affiliation":[{"name":"Mondragon University, Mondragon, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0588-316X","authenticated-orcid":false,"given":"Pablo","family":"Valle","sequence":"additional","affiliation":[{"name":"Mondragon University, Mondragon, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3574-6681","authenticated-orcid":false,"given":"Ibai","family":"Roman","sequence":"additional","affiliation":[{"name":"Mondragon University, Mondragon, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3770-1495","authenticated-orcid":false,"given":"Miren","family":"Illarramendi","sequence":"additional","affiliation":[{"name":"Mondragon University, Mondragon, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7507-5080","authenticated-orcid":false,"given":"Aitor","family":"Arrieta","sequence":"additional","affiliation":[{"name":"Mondragon University, Mondragon, Spain"}]}],"member":"320","published-online":{"date-parts":[[2024,12,27]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"crossref","first-page":"1016","DOI":"10.1145\/3180155.3180160","volume-title":"Proceedings of the 40th International Conference on Software Engineering","author":"Abdessalem Raja Ben","year":"2018","unstructured":"Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2018a. Testing vision-based control systems using learnable evolutionary algorithms. In Proceedings of the 40th International Conference on Software Engineering. 1016\u20131026."},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238192"},{"key":"e_1_3_3_4_2","doi-asserted-by":"crossref","first-page":"1327","DOI":"10.1145\/3512290.3528697","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference","author":"Arrieta Aitor","year":"2022","unstructured":"Aitor Arrieta. 2022. Multi-objective metamorphic follow-up test case selection for deep learning systems. In Proceedings of the Genetic and Evolutionary Computation Conference. 1327\u20131335."},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","unstructured":"Jon Ayerdi Pablo Valle Asier Iriarte Ibai Roman Miren Illarramendi and Aitor Arrieta. 2023. Dataset for \u201cMarMot: Metamorphic runtime monitoring of autonomous driving systems. DOI: 10.5281\/zenodo.10716933","DOI":"10.5281\/zenodo.10716933"},{"issue":"2","key":"e_1_3_3_6_2","article-title":"Metamorphic runtime checking of applications without test oracles","volume":"28","author":"Bell Jonathan","year":"2015","unstructured":"Jonathan Bell, Christian Murphy, and Gail Kaiser. 2015. Metamorphic runtime checking of applications without test oracles. CrossTalk 28, 2 (2015).","journal-title":"CrossTalk"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970311"},{"issue":"4","key":"e_1_3_3_8_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3511701","article-title":"Testing the plasticity of reinforcement learning-based systems","volume":"31","author":"Biagiola Matteo","year":"2022","unstructured":"Matteo Biagiola and Paolo Tonella. 2022. Testing the plasticity of reinforcement learning-based systems. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 4 (2022), 1\u201346.","journal-title":"ACM Transactions on Software Engineering and Methodology (TOSEM)"},{"key":"e_1_3_3_9_2","unstructured":"Mariusz Bojarski Davide Del Testa Daniel Dworakowski Bernhard Firner Beat Flepp Prasoon Goyal Lawrence D. Jackel Mathew Monfort Urs Muller Jiakai Zhang Xin Zhang Jake Zhao and Karol Zieba. 2016. End to end learning for self-driving cars. arXiv:1604.07316. Retrieved from https:\/\/arxiv.org\/abs\/1604.07316"},{"key":"e_1_3_3_10_2","first-page":"375","volume-title":"Proceedings of the IEEE 13th International Conference on Software Testing, Validation and Verification (ICST \u201920)","author":"Calo Alessandro","year":"2020","unstructured":"Alessandro Calo, Paolo Arcaini, Shaukat Ali, Florian Hauer, and Fuyuki Ishikawa. 2020. Generating avoidable collision scenarios for testing autonomous driving systems. In Proceedings of the IEEE 13th International Conference on Software Testing, Validation and Verification (ICST \u201920). IEEE, 375\u2013386."},{"key":"e_1_3_3_11_2","volume-title":"Metamorphic Testing: A New Approach for Generating Next Test Cases","author":"Chen T. Y.","year":"1998","unstructured":"T. Y. Chen, S. C. Cheung, and S. M. Yiu. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report HKUST-CS98-01, Department of Computer Science, The Hong Kong University of Science and Technology."},{"key":"e_1_3_3_12_2","first-page":"32","volume-title":"Proceedings of the IEEE\/ACM 6th International Workshop on Metamorphic Testing (MET \u201921)","author":"Deng Yao","year":"2021","unstructured":"Yao Deng, Guannan Lou, Xi Zheng, Tianyi Zhang, Miryung Kim, Huai Liu, Chen Wang, and Tsong Yueh Chen. 2021. BMT: Behavior driven development-based metamorphic testing for autonomous driving models. In Proceedings of the IEEE\/ACM 6th International Workshop on Metamorphic Testing (MET \u201921). IEEE, 32\u201336."},{"issue":"4","key":"e_1_3_3_13_2","doi-asserted-by":"crossref","first-page":"1964","DOI":"10.1109\/TSE.2022.3206427","article-title":"A declarative metamorphic testing framework for autonomous driving","volume":"49","author":"Deng Yao","year":"2022","unstructured":"Yao Deng, Xi Zheng, Tianyi Zhang, Huai Liu, Guannan Lou, Miryung Kim, and Tsong Yueh Chen. 2022. A declarative metamorphic testing framework for autonomous driving. IEEE Transactions on Software Engineering 49, 4 (2022), 1964\u20131982.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_3_14_2","first-page":"7","volume-title":"Proceedings of the IEEE 26th Pacific Rim International Symposium on Dependable Computing (PRDC \u201921)","author":"Ferreira Raul Sena","year":"2021","unstructured":"Raul Sena Ferreira, Jean Arlat, J\u00e9r\u00e9mie Guiochet, and H\u00e9l\u00e8ne Waeselynck. 2021. Benchmarking safety monitors for image classifiers with machine learning. In Proceedings of the IEEE 26th Pacific Rim International Symposium on Dependable Computing (PRDC \u201921). IEEE, 7\u201316."},{"key":"e_1_3_3_15_2","first-page":"1050","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Gal Yarin","year":"2016","unstructured":"Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the International Conference on Machine Learning. PMLR, 1050\u20131059."},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338942"},{"key":"e_1_3_3_17_2","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1145\/3293882.3330566","volume-title":"Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis","author":"Gambi Alessio","year":"2019","unstructured":"Alessio Gambi, Marc Mueller, and Gordon Fraser. 2019b. Automatically testing self-driving cars with search-based procedural content generation. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 318\u2013328."},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA46639.2022.9811924"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510188"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00155"},{"key":"e_1_3_3_21_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Hendrycks Dan","year":"2019","unstructured":"Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations (2019)."},{"key":"e_1_3_3_22_2","first-page":"27","volume-title":"Proceedings of the IEEE International Conference on Artificial Intelligence Testing (AITest \u201919)","author":"Henriksson Jens","year":"2019","unstructured":"Jens Henriksson, Christian Berger, Markus Borg, Lars Tornberg, Cristofer Englund, Sankar Raman Sathyamoorthy, and Stig Ursing. 2019. Towards structured evaluation of deep neural network supervisors. In Proceedings of the IEEE International Conference on Artificial Intelligence Testing (AITest \u201919). IEEE, 27\u201334."},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.79.8.2554"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380395"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464825"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-021-00310-0"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_3_28_2","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-54927-9","volume-title":"Robot Operating System (ROS)","author":"Koub\u00e2a Anis","year":"2017","unstructured":"Anis Koub\u00e2a et al. 2017. Robot Operating System (ROS). Vol. 1. Springer."},{"key":"e_1_3_3_29_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","volume":"30","author":"Lakshminarayanan Balaji","year":"2017","unstructured":"Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc."},{"key":"e_1_3_3_30_2","unstructured":"LeoRover. 2022. LeoRover Dataset. Retrieved from https:\/\/www.kaggle.com\/datasets\/aleksanderszymaski\/full-track"},{"key":"e_1_3_3_31_2","unstructured":"LeoRover. 2023. LeoRover. Retrieved from https:\/\/github.com\/LeoRover"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.5555\/3103620.3103632"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3150788"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9196844"},{"key":"e_1_3_3_35_2","doi-asserted-by":"crossref","first-page":"197","DOI":"10.1016\/j.jss.2017.10.031","article-title":"Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles","volume":"137","author":"Mullins Galen E.","year":"2018","unstructured":"Galen E. Mullins, Paul G. Stankiewicz, R. Chad Hawthorne, and Satyandra K. Gupta. 2018. Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles. Journal of Systems and Software 137 (2018), 197\u2013215.","journal-title":"Journal of Systems and Software"},{"key":"e_1_3_3_36_2","unstructured":"Christian Murphy and Gail E. Kaiser. 2009. Metamorphic Runtime Checking of Non-Testable Programs. Retrieved from https:\/\/core.ac.uk\/reader\/161435520"},{"key":"e_1_3_3_37_2","first-page":"189","volume-title":"Proceedings of the 18th International Symposium on Software Testing and Analysis","author":"Murphy Christian","year":"2009","unstructured":"Christian Murphy, Kuang Shen, and Gail Kaiser. 2009. Automatic system testing of programs without test oracles. In Proceedings of the 18th International Symposium on Software Testing and Analysis. 189\u2013200."},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_3_39_2","first-page":"1161","volume-title":"Proceedings of the IEEE\/ACM 45th International Conference on Software Engineering (ICSE \u201923)","author":"Riccio Vincenzo","year":"2023","unstructured":"Vincenzo Riccio and Paolo Tonella. 2023. When and why test generators for deep learning produce invalid inputs: An empirical study. In Proceedings of the IEEE\/ACM 45th International Conference on Software Engineering (ICSE \u201923). IEEE, 1161\u20131173."},{"key":"e_1_3_3_40_2","first-page":"1","volume-title":"Proceedings of the Annual Meeting of the Southern Association for Institutional Research","author":"Romano Jeanine","year":"2006","unstructured":"Jeanine Romano, Jeffrey. D. Kromrey, Jesse Coraggio, Jeff Skowronek, and Linda Devine. 2006. Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen\u2019s d indices the most appropriate choices. In Proceedings of the Annual Meeting of the Southern Association for Institutional Research. 1\u201351."},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2021.3052449"},{"key":"e_1_3_3_42_2","first-page":"6088","article-title":"Learning representations by back-propagating errors","volume":"323","author":"Rumelhart David E.","year":"1986","unstructured":"David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1986. Learning representations by back-propagating errors. Nature 323, 6088 (1986), 533\u2013536.","journal-title":"Nature"},{"key":"e_1_3_3_43_2","first-page":"1","volume-title":"Proceedings of the IEEE Intelligent Vehicles Symposium (IV \u201923)","author":"Scheuer Franz","year":"2023","unstructured":"Franz Scheuer, Alessio Gambi, and Paolo Arcaini. 2023. STRETCH: Generating challenging scenarios for testing collision avoidance systems. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV \u201923). IEEE, 1\u20136."},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110574"},{"key":"e_1_3_3_46_2","first-page":"1","volume-title":"Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering","author":"Stocco Andrea","year":"2022","unstructured":"Andrea Stocco, Paulo J Nunes, Marcelo D\u2019Amorim, and Paolo Tonella. 2022a. Thirdeye: Attention maps for safe autonomous driving systems. In Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering. 1\u201312."},{"issue":"4","key":"e_1_3_3_47_2","doi-asserted-by":"crossref","first-page":"1928","DOI":"10.1109\/TSE.2022.3202311","article-title":"Mind the gap! A study on the transferability of virtual vs physical-world testing of autonomous driving systems","volume":"49","author":"Stocco Andrea","year":"2022","unstructured":"Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2022b. Mind the gap! A study on the transferability of virtual vs physical-world testing of autonomous driving systems. IEEE Transactions on Software Engineering 49, 4 (2022), 1928\u20131940.","journal-title":"IEEE Transactions on Software Engineering"},{"issue":"10","key":"e_1_3_3_48_2","first-page":"e2386","article-title":"Confidence-driven weighted retraining for predicting safety-critical failures in autonomous driving systems","volume":"34","author":"Stocco Andrea","year":"2022","unstructured":"Andrea Stocco and Paolo Tonella. 2022. Confidence-driven weighted retraining for predicting safety-critical failures in autonomous driving systems. Journal of Software: Evolution and Process 34, 10 (2022), e2386.","journal-title":"Journal of Software: Evolution and Process"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380353"},{"key":"e_1_3_3_50_2","first-page":"1","volume-title":"Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering","author":"Sun Yang","year":"2022","unstructured":"Yang Sun, Christopher M. Poskitt, Jun Sun, Yuqi Chen, and Zijiang Yang. 2022. LawBreaker: An approach for specifying traffic laws and fuzzing autonomous vehicles. In Proceedings of the 37th IEEE\/ACM International Conference on Automated Software Engineering. 1\u201312."},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_3_52_2","doi-asserted-by":"crossref","first-page":"727","DOI":"10.1145\/3377811.3380379","volume-title":"Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering","author":"Wang Huiyan","year":"2020","unstructured":"Huiyan Wang, Jingwei Xu, Chang Xu, Xiaoxing Ma, and Jian Lu. 2020. Dissector: Input validation for deep learning applications by crossing-layer dissection. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering. 727\u2013738."},{"issue":"6","key":"e_1_3_3_53_2","doi-asserted-by":"crossref","first-page":"e1840","DOI":"10.1002\/stvr.1840","article-title":"Uncertainty quantification for deep neural networks: An empirical comparison and usage guidelines","volume":"33","author":"Weiss Michael","year":"2023","unstructured":"Michael Weiss and Paolo Tonella. 2023. Uncertainty quantification for deep neural networks: An empirical comparison and usage guidelines. Software Testing, Verification and Reliability 33, 6 (2023), e1840.","journal-title":"Software Testing, Verification and Reliability"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00044"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-017-0609-6"},{"key":"e_1_3_3_56_2","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1007\/978-3-319-42061-5_16","volume-title":"Proceedings of the Modelling Foundations and Applications: 12th European Conference, ECMFA 2016, Held as Part of STAF 2016","author":"Zhang Man","year":"2016","unstructured":"Man Zhang, Bran Selic, Shaukat Ali, Tao Yue, Oscar Okariz, and Roland Norgren. 2016. Understanding uncertainty in cyber-physical systems: A conceptual model. In Proceedings of the Modelling Foundations and Applications: 12th European Conference, ECMFA 2016, Held as Part of STAF 2016. Springer, 247\u2013264."},{"key":"e_1_3_3_57_2","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1145\/3238147.3238187","volume-title":"Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering","author":"Zhang Mengshi","year":"2018","unstructured":"Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid. 2018. DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. In Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering. 132\u2013142."},{"issue":"4","key":"e_1_3_3_58_2","doi-asserted-by":"crossref","first-page":"1860","DOI":"10.1109\/TSE.2022.3195640","article-title":"Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles","volume":"49","author":"Zhong Ziyuan","year":"2022","unstructured":"Ziyuan Zhong, Gail Kaiser, and Baishakhi Ray. 2022. Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles. IEEE Transactions on Software Engineering 49, 4 (2022), 1860\u20131875.","journal-title":"IEEE Transactions on Software Engineering"},{"issue":"6","key":"e_1_3_3_59_2","first-page":"3391","article-title":"Specification-based autonomous driving system testing","volume":"49","author":"Zhou Yuan","year":"2023","unstructured":"Yuan Zhou, Yang Sun, Yun Tang, Yuqi Chen, Jun Sun, Christopher M. Poskitt, Yang Liu, and Zijiang Yang. 2023. Specification-based autonomous driving system testing. IEEE Transactions on Software Engineering 49, 6 (2023), 3391\u20133410.","journal-title":"IEEE Transactions on Software Engineering"},{"issue":"3","key":"e_1_3_3_60_2","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1145\/3241979","article-title":"Metamorphic testing of driverless cars","volume":"62","author":"Zhou Zhi Quan","year":"2019","unstructured":"Zhi Quan Zhou and Liqun Sun. 2019. Metamorphic testing of driverless cars. Communications of the ACM 62, 3 (2019), 61\u201367.","journal-title":"Communications of the ACM"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678171","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3678171","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:54:08Z","timestamp":1750287248000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678171"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,27]]},"references-count":59,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1,31]]}},"alternative-id":["10.1145\/3678171"],"URL":"https:\/\/doi.org\/10.1145\/3678171","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,27]]},"assertion":[{"value":"2023-10-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-06","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}