{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:17:35Z","timestamp":1750220255697,"version":"3.41.0"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"5s","license":[{"start":{"date-parts":[[2021,9,22]],"date-time":"2021-09-22T00:00:00Z","timestamp":1632268800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"ONR","award":["N00014-18-1-2501"],"award-info":[{"award-number":["N00014-18-1-2501"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Embed. Comput. Syst."],"published-print":{"date-parts":[[2021,10,31]]},"abstract":"<jats:p>In this paper, we study feedback dynamical systems with memoryless controllers under imperfect information. We develop an algorithm that searches for \u201cadversarial scenarios\u201d, which can be thought of as the strategy for the adversary representing the noise and disturbances, that lead to safety violations. The main challenge is to analyze the closed-loop system's vulnerabilities with a potentially complex or even unknown controller in the loop. As opposed to commonly adopted approaches that treat the system under test as a black-box, we propose a synthesis-guided approach, which leverages the knowledge of a plant model at hand. This hence leads to a way to deal with gray-box systems (i.e., with known plant and unknown controller). Our approach reveals the role of the imperfect information in the violation. Examples show that our approach can find non-trivial scenarios that are difficult to expose by random simulations. This approach is further extended to incorporate model mismatch and to falsify vision-in-the-loop systems against finite-time reach-avoid specifications.<\/jats:p>","DOI":"10.1145\/3477033","type":"journal-article","created":{"date-parts":[[2021,9,22]],"date-time":"2021-09-22T20:48:40Z","timestamp":1632343720000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Synthesis-guided Adversarial Scenario Generation for Gray-box Feedback Control Systems with Sensing Imperfections"],"prefix":"10.1145","volume":"20","author":[{"given":"Liren","family":"Yang","sequence":"first","affiliation":[{"name":"University of Michigan, Ann Arbor, Michigan, USA"}]},{"given":"Necmiye","family":"Ozay","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, Michigan, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,9,22]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.5555\/1987389.1987416"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.23919\/DATE.2019.8715114"},{"key":"e_1_2_1_3_1","first-page":"604","article-title":"Infinite time reachability of state-space regions by using feedback control","volume":"17","author":"Bertsekas Dimitri","year":"1972","unstructured":"Dimitri Bertsekas . 1972 . Infinite time reachability of state-space regions by using feedback control . IEEE TAC 17 , 5 (1972), 604 \u2013 613 . Dimitri Bertsekas. 1972. Infinite time reachability of state-space regions by using feedback control. IEEE TAC 17, 5 (1972), 604\u2013613.","journal-title":"IEEE TAC"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-24743-2_10"},{"key":"e_1_2_1_5_1","volume-title":"Hamilton","author":"Bose Avishek Joey","year":"2020","unstructured":"Avishek Joey Bose , Gauthier Gidel , Hugo Berrard , Andre Cianflone , Pascal Vincent , Simon Lacoste-Julien , and William L . Hamilton . 2020 . Adversarial example games. arxiv:2007.00720 (2020). Avishek Joey Bose, Gauthier Gidel, Hugo Berrard, Andre Cianflone, Pascal Vincent, Simon Lacoste-Julien, and William L. Hamilton. 2020. Adversarial example games. arxiv:2007.00720 (2020)."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39799-8_18"},{"key":"e_1_2_1_7_1","first-page":"2906","article-title":"Using control synthesis to generate corner cases: A case study on autonomous driving","volume":"37","author":"Chou Glen","year":"2018","unstructured":"Glen Chou , Yunus Emre Sahin , Liren Yang , Kwesi J. Rutledge , Petter Nilsson , and Necmiye Ozay . 2018 . Using control synthesis to generate corner cases: A case study on autonomous driving . IEEE TCAD 37 , 11 (2018), 2906 \u2013 2917 . Glen Chou, Yunus Emre Sahin, Liren Yang, Kwesi J. Rutledge, Petter Nilsson, and Necmiye Ozay. 2018. Using control synthesis to generate corner cases: A case study on autonomous driving. IEEE TCAD 37, 11 (2018), 2906\u20132917.","journal-title":"IEEE TCAD"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1014052.1014066"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/11730637_14"},{"key":"e_1_2_1_10_1","volume-title":"Ames","author":"Dean Sarah","year":"2020","unstructured":"Sarah Dean , Andrew J. Taylor , Ryan K. Cosner , Benjamin Recht , and Aaron D . Ames . 2020 . Guaranteeing safety of learned perception modules via measurement-robust control barrier functions. arxiv:2010.16001 (2020). Sarah Dean, Andrew J. Taylor, Ryan K. Cosner, Benjamin Recht, and Aaron D. Ames. 2020. Guaranteeing safety of learned perception modules via measurement-robust control barrier functions. arxiv:2010.16001 (2020)."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126521"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24953-7_35"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-14295-6_17"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5555\/1885174.1885183"},{"key":"e_1_2_1_15_1","volume-title":"CARLA: An open urban driving simulator. In Prod. of the 1st ACRL. 1\u201316.","author":"Dosovitskiy Alexey","year":"2017","unstructured":"Alexey Dosovitskiy , German Ros , Felipe Codevilla , Antonio Lopez , and Vladlen Koltun . 2017 . CARLA: An open urban driving simulator. In Prod. of the 1st ACRL. 1\u201316. Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. In Prod. of the 1st ACRL. 1\u201316."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10817-018-09509-5"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-25540-4_25"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30281-8_10"},{"volume-title":"32nd Intl. Conf. on CAV.","author":"Fremont Daniel J.","key":"e_1_2_1_19_1","unstructured":"Daniel J. Fremont , Johnathan Chiu , Dragos D. Margineantu , Denis Osipychev , and Sanjit A. Seshia . 2020. Formal analysis and redesign of a neural network-based aircraft taxiing system with VerifAI . In 32nd Intl. Conf. on CAV. Daniel J. Fremont, Johnathan Chiu, Dragos D. Margineantu, Denis Osipychev, and Sanjit A. Seshia. 2020. Formal analysis and redesign of a neural network-based aircraft taxiing system with VerifAI. In 32nd Intl. Conf. on CAV."},{"volume-title":"Verifying controllers against adversarial examples with bayesian optimization. In 2018 IEEE ICRA","author":"Ghosh Shromona","key":"e_1_2_1_20_1","unstructured":"Shromona Ghosh , Felix Berkenkamp , Gireeja Ranade , Shaz Qadeer , and Ashish Kapoor . 2018. Verifying controllers against adversarial examples with bayesian optimization. In 2018 IEEE ICRA . IEEE , 7306\u20137313. Shromona Ghosh, Felix Berkenkamp, Gireeja Ranade, Shaz Qadeer, and Ashish Kapoor. 2018. Verifying controllers against adversarial examples with bayesian optimization. In 2018 IEEE ICRA. IEEE, 7306\u20137313."},{"key":"e_1_2_1_21_1","volume-title":"Seshia","author":"Ghosh Shromona","year":"2019","unstructured":"Shromona Ghosh , Hadi Ravanbakhsh , and Sanjit A . Seshia . 2019 . Counterexample-guided synthesis of perception models and control. arxiv:1911.01523 (2019). Shromona Ghosh, Hadi Ravanbakhsh, and Sanjit A. Seshia. 2019. Counterexample-guided synthesis of perception models and control. arxiv:1911.01523 (2019)."},{"key":"e_1_2_1_22_1","volume-title":"Adversarial policies: Attacking deep reinforcement learning. arxiv:1905.10615","author":"Gleave Adam","year":"2019","unstructured":"Adam Gleave , Michael Dennis , Cody Wild , Neel Kant , Sergey Levine , and Stuart Russell . 2019. Adversarial policies: Attacking deep reinforcement learning. arxiv:1905.10615 ( 2019 ). Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. 2019. Adversarial policies: Attacking deep reinforcement learning. arxiv:1905.10615 (2019)."},{"volume-title":"Prod. of the 3rd ICLR.","author":"Goodfellow Ian J.","key":"e_1_2_1_23_1","unstructured":"Ian J. Goodfellow , Jonathon Shlens , and Christian Szegedy . 2015. Explaining and harnessing adversarial examples . In Prod. of the 3rd ICLR. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Prod. of the 3rd ICLR."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.5555\/646880.710454"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.23919\/ECC.2013.6669862"},{"volume-title":"Prod. of 5th ICLR.","author":"Huang Sandy","key":"e_1_2_1_26_1","unstructured":"Sandy Huang , Nicolas Papernot , Ian Goodfellow , Yan Duan , and Pieter Abbeel . 2017. Adversarial attacks on neural network policies . In Prod. of 5th ICLR. Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. 2017. Adversarial attacks on neural network policies. In Prod. of 5th ICLR."},{"key":"e_1_2_1_27_1","unstructured":"Jongwoo Kim Joel M Esposito and Vijay Kumar. 2005. An RRT-based algorithm for testing and validating multi-robot controllers. In Robotics: Science and Systems. 249\u2013256.  Jongwoo Kim Joel M Esposito and Vijay Kumar. 2005. An RRT-based algorithm for testing and validating multi-robot controllers. In Robotics: Science and Systems. 249\u2013256."},{"volume-title":"Prod. of 5th ICLR.","author":"Kos Jernej","key":"e_1_2_1_28_1","unstructured":"Jernej Kos and Dawn Song . 2017. Delving into adversarial attacks on deep policies . In Prod. of 5th ICLR. Jernej Kos and Dawn Song. 2017. Delving into adversarial attacks on deep policies. In Prod. of 5th ICLR."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-014-0838-1"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/2946645.2946684"},{"key":"e_1_2_1_31_1","first-page":"2206","article-title":"Invariance control synthesis for switched nonlinear systems: An interval analysis approach","volume":"63","author":"Li Yinan","year":"2017","unstructured":"Yinan Li and Jun Liu . 2017 . Invariance control synthesis for switched nonlinear systems: An interval analysis approach . IEEE TAC 63 , 7 (2017), 2206 \u2013 2211 . Yinan Li and Jun Liu. 2017. Invariance control synthesis for switched nonlinear systems: An interval analysis approach. IEEE TAC 63, 7 (2017), 2206\u20132211.","journal-title":"IEEE TAC"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3365365.3382219"},{"key":"e_1_2_1_33_1","first-page":"947","article-title":"A time-dependent Hamilton-Jacobi formulation of reachable sets for continuous dynamic games","volume":"50","author":"Mitchell Ian M.","year":"2005","unstructured":"Ian M. Mitchell , Alexandre M. Bayen , and Claire J. Tomlin . 2005 . A time-dependent Hamilton-Jacobi formulation of reachable sets for continuous dynamic games . IEEE TAC 50 , 7 (2005), 947 \u2013 957 . Ian M. Mitchell, Alexandre M. Bayen, and Claire J. Tomlin. 2005. A time-dependent Hamilton-Jacobi formulation of reachable sets for continuous dynamic games. IEEE TAC 50, 7 (2005), 947\u2013957.","journal-title":"IEEE TAC"},{"key":"e_1_2_1_34_1","doi-asserted-by":"crossref","unstructured":"Seyed-Mohsen Moosavi-Dezfooli Alhussein Fawzi and Pascal Frossard. 2016. Deepfool: a simple and accurate method to fool deep neural networks. In Prod. of the IEEE CVPR. 2574\u20132582.  Seyed-Mohsen Moosavi-Dezfooli Alhussein Fawzi and Pascal Frossard. 2016. Deepfool: a simple and accurate method to fool deep neural networks. In Prod. of the IEEE CVPR. 2574\u20132582.","DOI":"10.1109\/CVPR.2016.282"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/1770351.1770414"},{"key":"e_1_2_1_36_1","first-page":"1294","article-title":"Correct-by-construction adaptive cruise control: Two approaches","volume":"24","author":"Nilsson Petter","year":"2015","unstructured":"Petter Nilsson , Omar Hussien , Ayca Balkan , Yuxiao Chen , Aaron D. Ames , Jessy W. Grizzle , Necmiye Ozay , Huei Peng , and Paulo Tabuada . 2015 . Correct-by-construction adaptive cruise control: Two approaches . IEEE TCST 24 , 4 (2015), 1294 \u2013 1307 . Petter Nilsson, Omar Hussien, Ayca Balkan, Yuxiao Chen, Aaron D. Ames, Jessy W. Grizzle, Necmiye Ozay, Huei Peng, and Paulo Tabuada. 2015. Correct-by-construction adaptive cruise control: Two approaches. IEEE TCST 24, 4 (2015), 1294\u20131307.","journal-title":"IEEE TCST"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-00768-2_31"},{"volume-title":"Linear encodings for polytope containment problems. In 2019 CDC","author":"Sadraddini Sadra","key":"e_1_2_1_39_1","unstructured":"Sadra Sadraddini and Russ Tedrake . 2019. Linear encodings for polytope containment problems. In 2019 CDC . IEEE , 4367\u20134372. Sadra Sadraddini and Russ Tedrake. 2019. Linear encodings for polytope containment problems. In 2019 CDC. IEEE, 4367\u20134372."},{"volume-title":"Nonlinear Analysis, Differential Equations and Control","author":"Sontag Eduardo D.","key":"e_1_2_1_40_1","unstructured":"Eduardo D. Sontag . 1999. Stability and stabilization: discontinuities and the effect of disturbances . In Nonlinear Analysis, Differential Equations and Control . Springer , 551\u2013598. Eduardo D. Sontag. 1999. Stability and stabilization: discontinuities and the effect of disturbances. In Nonlinear Analysis, Differential Equations and Control. Springer, 551\u2013598."},{"volume-title":"Prod. of the 2nd ICLR.","author":"Szegedy Christian","key":"e_1_2_1_41_1","unstructured":"Christian Szegedy , Wojciech Zaremba , Ilya Sutskever , Joan Bruna , Dumitru Erhan , Ian Goodfellow , and Rob Fergus . 2014. Intriguing properties of neural networks . In Prod. of the 2nd ICLR. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In Prod. of the 2nd ICLR."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2018.8500421"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3365365.3382193"},{"key":"e_1_2_1_44_1","volume-title":"Falsification-Based Robust Adversarial Reinforcement Learning. arxiv:2007.00691","author":"Wang Xiao","year":"2020","unstructured":"Xiao Wang , Saasha Nair , and Matthias Althoff . 2020. Falsification-Based Robust Adversarial Reinforcement Learning. arxiv:2007.00691 ( 2020 ). Xiao Wang, Saasha Nair, and Matthias Althoff. 2020. Falsification-Based Robust Adversarial Reinforcement Learning. arxiv:2007.00691 (2020)."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3302504.3311814"},{"key":"e_1_2_1_46_1","doi-asserted-by":"crossref","unstructured":"Liren Yang Xiaofan Cui Al-Thaddeus Avestruz and Necmiye Ozay. 2019. Correct-by-construction control synthesis for buck converters with event-triggered state measurement. In 2019 ACC. 1056\u20131063.  Liren Yang Xiaofan Cui Al-Thaddeus Avestruz and Necmiye Ozay. 2019. Correct-by-construction control synthesis for buck converters with event-triggered state measurement. In 2019 ACC. 1056\u20131063.","DOI":"10.23919\/ACC.2019.8814937"},{"volume-title":"Efficient safety control synthesis with imperfect state information. In 2020 CDC","author":"Yang Liren","key":"e_1_2_1_47_1","unstructured":"Liren Yang and Necmiye Ozay . 2020. Efficient safety control synthesis with imperfect state information. In 2020 CDC . IEEE , 874\u2013880. Liren Yang and Necmiye Ozay. 2020. Efficient safety control synthesis with imperfect state information. In 2020 CDC. IEEE, 874\u2013880."}],"container-title":["ACM Transactions on Embedded Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3477033","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3477033","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3477033","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:46Z","timestamp":1750188646000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3477033"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,22]]},"references-count":47,"journal-issue":{"issue":"5s","published-print":{"date-parts":[[2021,10,31]]}},"alternative-id":["10.1145\/3477033"],"URL":"https:\/\/doi.org\/10.1145\/3477033","relation":{},"ISSN":["1539-9087","1558-3465"],"issn-type":[{"type":"print","value":"1539-9087"},{"type":"electronic","value":"1558-3465"}],"subject":[],"published":{"date-parts":[[2021,9,22]]},"assertion":[{"value":"2021-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-09-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}