{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T10:00:44Z","timestamp":1777716044384,"version":"3.51.4"},"reference-count":83,"publisher":"SAGE Publications","issue":"2","license":[{"start":{"date-parts":[[2024,1,24]],"date-time":"2024-01-24T00:00:00Z","timestamp":1706054400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"funder":[{"name":"I\/UCRC Center for Embedded Systems"},{"name":"NSF","award":["2038666"],"award-info":[{"award-number":["2038666"]}]},{"name":"NSF","award":["2039087"],"award-info":[{"award-number":["2039087"]}]},{"name":"NSF","award":["IIP-1361926"],"award-info":[{"award-number":["IIP-1361926"]}]}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of Robotics Research"],"published-print":{"date-parts":[[2024,2]]},"abstract":"<jats:p>Automated vehicles (AV) heavily depend on robust perception systems. Current methods for evaluating vision systems focus mainly on frame-by-frame performance. Such evaluation methods appear to be inadequate in assessing the performance of a perception subsystem when used within an AV. In this paper, we present a logic\u2014referred to as Spatio-Temporal Perception Logic (STPL)\u2014which utilizes both spatial and temporal modalities. STPL enables reasoning over perception data using spatial and temporal operators. One major advantage of STPL is that it facilitates basic sanity checks on the functional performance of the perception system, even without ground truth data in some cases. We identify a fragment of STPL which is efficiently monitorable offline in polynomial time. Finally, we present a range of specifications for AV perception systems to highlight the types of requirements that can be expressed and analyzed through offline monitoring with STPL.<\/jats:p>","DOI":"10.1177\/02783649231223546","type":"journal-article","created":{"date-parts":[[2024,1,24]],"date-time":"2024-01-24T20:15:40Z","timestamp":1706127340000},"page":"203-238","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":5,"title":["Formalizing and evaluating requirements of perception systems for automated vehicles using spatio-temporal perception logic"],"prefix":"10.1177","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0110-0006","authenticated-orcid":false,"given":"Mohammad","family":"Hekmatnejad","sequence":"first","affiliation":[{"name":"School of Computing and Augmented Intelligence (formerly CIDSE), Arizona State University, Tempe, AZ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bardh","family":"Hoxha","sequence":"additional","affiliation":[{"name":"Research & Development, Toyota Motor North America, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jyotirmoy V.","family":"Deshmukh","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Southern California, Los Angeles, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yezhou","family":"Yang","sequence":"additional","affiliation":[{"name":"School of Computing and Augmented Intelligence (formerly CIDSE), Arizona State University, Tempe, AZ, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Georgios","family":"Fainekos","sequence":"additional","affiliation":[{"name":"Research & Development, Toyota Motor North America, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","published-online":{"date-parts":[[2024,1,24]]},"reference":[{"key":"bibr1-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1115\/1.2017-Dec-9"},{"key":"bibr2-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4020-5587-4"},{"key":"bibr3-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1145\/174644.174651"},{"key":"bibr4-02783649231223546","first-page":"309","volume-title":"Handbook of Data Structures and Applications","author":"Aluru S","year":"2018"},{"key":"bibr5-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/RE54965.2022.00037"},{"key":"bibr6-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-44267-4_13"},{"key":"bibr7-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/IROS51168.2021.9636497"},{"key":"bibr8-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-88494-9_18"},{"key":"bibr9-02783649231223546","unstructured":"Baotic M (2009) Polytopic computations in constrained optimal control. 50, 119\u2013134."},{"key":"bibr10-02783649231223546","first-page":"1","volume":"20","author":"Bartocci E","year":"2010","journal-title":"Scientific Annals of Computer Science"},{"key":"bibr11-02783649231223546","first-page":"128","volume":"10457","author":"Bartocci E","year":"2018","journal-title":"Lectures on Runtime Verification - Introductory and Advanced Topics, LNCS"},{"key":"bibr12-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9197053"},{"key":"bibr13-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1145\/2699444"},{"key":"bibr14-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1080\/13875860801926884"},{"key":"bibr15-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-48983-5_8"},{"key":"bibr16-02783649231223546","first-page":"1","author":"Buonamici FB","year":"2019","journal-title":"International Journal on Software Tools for Technology Transfer"},{"key":"bibr17-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01164"},{"key":"bibr18-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2016.7795568"},{"key":"bibr19-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30494-4_18"},{"key":"bibr20-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009712514511"},{"key":"bibr21-02783649231223546","volume-title":"A Survey of Algorithms for Black-Box Safety Validation","author":"Corso A","year":"2020"},{"key":"bibr22-02783649231223546","first-page":"383","volume":"13965","author":"Cosler M","year":"2023","journal-title":"Computer Aided Verification, LNCS"},{"key":"bibr23-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC45102.2020.9294442"},{"key":"bibr24-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/MEMCOD.2016.7797763"},{"key":"bibr25-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-03769-7_23"},{"key":"bibr26-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1145\/3147451"},{"key":"bibr27-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/s10817-018-09509-5"},{"key":"bibr28-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-25540-4_25"},{"key":"bibr29-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1145\/3038927"},{"key":"bibr30-02783649231223546","first-page":"27","author":"Eisner C","year":"2006","journal-title":"A Practical Introduction to PSL"},{"key":"bibr31-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-00151-3_4"},{"key":"bibr32-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/ACC.2012.6315384"},{"key":"bibr33-02783649231223546","volume-title":"Scenic: Language-Based Scene Generation","author":"Fremont DJ","year":"2018"},{"key":"bibr34-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC45102.2020.9294368"},{"key":"bibr35-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i13.27068"},{"key":"bibr36-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1537"},{"key":"bibr37-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1177\/0278364913491297"},{"key":"bibr38-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1177\/0278364913491297"},{"key":"bibr39-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1145\/3501710.3519537"},{"key":"bibr40-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00013"},{"key":"bibr41-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/CDC.2014.7039367"},{"key":"bibr42-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2018.2792152"},{"key":"bibr43-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1145\/2728606.2728633"},{"key":"bibr44-02783649231223546","volume-title":"First-order Temporal Logic Monitoring with Bdds 56","author":"Havelund K","year":"2020"},{"key":"bibr45-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2015.7139022"},{"key":"bibr46-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2018.2889191"},{"key":"bibr47-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1145\/3359986.3361203"},{"key":"bibr48-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511810275"},{"key":"bibr49-02783649231223546","doi-asserted-by":"crossref","unstructured":"Kim E, Shenoy J, Junges S, et al. (2022) Querying labelled data with scenario programs for sim-to-real validation. In: ACM\/IEEE 13th International Conference on Cyber-Physical Systems (ICCPS), Milan, Italy, 4\u20136 May 2022.","DOI":"10.1109\/ICCPS54341.2022.00010"},{"key":"bibr50-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4020-5587-4_9"},{"key":"bibr51-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/BF01995674"},{"key":"bibr52-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2010.5509402"},{"key":"bibr53-02783649231223546","author":"Lee TB","year":"2018","journal-title":"Ars Technica"},{"issue":"26","key":"bibr54-02783649231223546","first-page":"2392","volume":"21","author":"Li T","year":"2021","journal-title":"Runtime verification of spatio-temporal specification language"},{"key":"bibr55-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39718-9_14"},{"key":"bibr56-02783649231223546","volume-title":"A Graph-Based Spatial Temporal Logic for Knowledge Representation and Automated Reasoning in Cognitive Robots","author":"Liu Z","year":"2020"},{"key":"bibr57-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-21437-0_6"},{"key":"bibr58-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/ICCPS48487.2020.00013"},{"key":"bibr59-02783649231223546","first-page":"152","volume":"3253","author":"Maler O","year":"2004","journal-title":"Proceedings of FORMATS-FTRTFT, LNCS"},{"key":"bibr60-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-44267-4_22"},{"key":"bibr61-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-0931-7"},{"key":"bibr62-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1016\/j.tcs.2006.01.019"},{"key":"bibr63-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1016\/j.automatica.2022.110692"},{"key":"bibr64-02783649231223546","unstructured":"Motional (2019) nuScenes dataset. Available at: https:\/\/www.nuscenes.org\/nuscenes (Accessed 14 November 2020)"},{"key":"bibr65-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-23820-3_2"},{"key":"bibr66-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48891.2023.10161125"},{"key":"bibr67-02783649231223546","volume-title":"Programming Languages: Concepts and Implementation","author":"Perugini S","year":"2021"},{"key":"bibr68-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-23778-7_53"},{"key":"bibr69-02783649231223546","first-page":"2232","author":"Richter SR","year":"2017","journal-title":"IEEE International Conference on Computer Vision"},{"key":"bibr70-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66845-1_4"},{"key":"bibr71-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-control-060117-105157"},{"key":"bibr72-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1016\/0146-664X(81)90042-3"},{"key":"bibr73-02783649231223546","volume-title":"Introduction to the Theory of Computation","author":"Sipser M","year":"2006","edition":"2"},{"key":"bibr74-02783649231223546","volume-title":"Spatio-Temporal Perception Logic (STPL) Offline Monitoring Tools","author":"STPL","year":"2023"},{"key":"bibr75-02783649231223546","doi-asserted-by":"crossref","unstructured":"Summers-Stay D, Cassidy T, Voss C (2014) Joint navigation in commander\/robot teams: dialog & task performance when vision is bandwidth-limited. In: Third Workshop on Vision and Language, Dublin, Ireland, 1\u20134 August 2014.","DOI":"10.3115\/v1\/W14-5402"},{"key":"bibr76-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/IROS40897.2019.8967644"},{"key":"bibr77-02783649231223546","unstructured":"Templeton B (2020) Tesla in Taiwan crashes directly into overturned truck, ignores pedestrian, with autopilot on."},{"key":"bibr78-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/TIV.2019.2955903"},{"key":"bibr79-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4020-5587-4_5"},{"key":"bibr80-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2017.60"},{"key":"bibr81-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/BigData47090.2019.9006018"},{"key":"bibr82-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00271"},{"key":"bibr83-02783649231223546","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2983149"}],"container-title":["The International Journal of Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649231223546","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/02783649231223546","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649231223546","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T10:17:12Z","timestamp":1777457832000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/02783649231223546"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,24]]},"references-count":83,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,2]]}},"alternative-id":["10.1177\/02783649231223546"],"URL":"https:\/\/doi.org\/10.1177\/02783649231223546","relation":{},"ISSN":["0278-3649","1741-3176"],"issn-type":[{"value":"0278-3649","type":"print"},{"value":"1741-3176","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,24]]}}}