{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T23:08:19Z","timestamp":1769555299995,"version":"3.49.0"},"reference-count":61,"publisher":"SAGE Publications","issue":"13","license":[{"start":{"date-parts":[[2020,10,1]],"date-time":"2020-10-01T00:00:00Z","timestamp":1601510400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of Robotics Research"],"published-print":{"date-parts":[[2020,11]]},"abstract":"<jats:p> For mobile robots navigating on sidewalks, the ability to safely cross street intersections is essential. Most existing approaches rely on the recognition of the traffic light signal to make an informed crossing decision. Although these approaches have been crucial enablers for urban navigation, the capabilities of robots employing such approaches are still limited to navigating only on streets that contain signalized intersections. In this article, we address this challenge and propose a multimodal convolutional neural network framework to predict the safety of a street intersection for crossing. Our architecture consists of two subnetworks: an interaction-aware trajectory estimation stream ( interaction-aware temporal convolutional neural network (IA-TCNN)), that predicts the future states of all observed traffic participants in the scene; and a traffic light recognition stream AtteNet. Our IA-TCNN utilizes dilated causal convolutions to model the behavior of all the observable dynamic agents in the scene without explicitly assigning priorities to the interactions among them, whereas AtteNet utilizes squeeze-excitation blocks to learn a content-aware mechanism for selecting the relevant features from the data, thereby improving the noise robustness. Learned representations from the traffic light recognition stream are fused with the estimated trajectories from the motion prediction stream to learn the crossing decision. Incorporating the uncertainty information from both modules enables our architecture to learn a likelihood function that is robust to noise and mispredictions from either subnetworks. Simultaneously, by learning to estimate motion trajectories of the surrounding traffic participants and incorporating knowledge of the traffic light signal, our network learns a robust crossing procedure that is invariant to the type of street intersection. Furthermore, we extend our previously introduced Freiburg Street Crossing dataset with sequences captured at multiple intersections of varying types, demonstrating complex interactions among the traffic participants as well as various lighting and weather conditions. We perform comprehensive experimental evaluations on public datasets as well as our Freiburg Street Crossing dataset, which demonstrate that our network achieves state-of-the-art performance for each of the subtasks, as well as for the crossing safety prediction. Moreover, we deploy the proposed architectural framework on a robotic platform and conduct real-world experiments that demonstrate the suitability of the approach for real-time deployment and robustness to various environments. <\/jats:p>","DOI":"10.1177\/0278364920961809","type":"journal-article","created":{"date-parts":[[2020,10,1]],"date-time":"2020-10-01T07:18:48Z","timestamp":1601536728000},"page":"1567-1598","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":31,"title":["Multimodal interaction-aware motion prediction for autonomous street crossing"],"prefix":"10.1177","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1405-845X","authenticated-orcid":false,"given":"Noha","family":"Radwan","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Freiburg, Germany"}]},{"given":"Wolfram","family":"Burgard","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Freiburg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4710-3114","authenticated-orcid":false,"given":"Abhinav","family":"Valada","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Freiburg, Germany"}]}],"member":"179","published-online":{"date-parts":[[2020,10,1]]},"reference":[{"key":"bibr1-0278364920961809","unstructured":"Abadi M, Agarwal A, Barham P, et al. (2015) TensorFlow: Large-scale machine learning on heterogeneous systems. https:\/\/www.tensorflow.org\/."},{"key":"bibr2-0278364920961809","author":"Alahi A","year":"2016","journal-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"bibr3-0278364920961809","author":"Azimi R","year":"2014","journal-title":"International Conference on Cyber-Physical Systems (ICCPS)"},{"key":"bibr4-0278364920961809","author":"Bai S","year":"2018","journal-title":"arXiv preprint arXiv:1803.01271"},{"key":"bibr5-0278364920961809","volume-title":"Technical Report","author":"Baker M","year":"2003"},{"key":"bibr6-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1109\/ICORR.2005.1501081"},{"key":"bibr7-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-009-0011-9"},{"key":"bibr8-0278364920961809","author":"Baumann U","year":"2018","journal-title":"International Conference on Robotics and Automation (ICRA)"},{"key":"bibr9-0278364920961809","author":"Behrendt K","year":"2017","journal-title":"International Conference on Robotics and Automation (ICRA)"},{"key":"bibr10-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-28619-4_3"},{"key":"bibr11-0278364920961809","author":"Campos GDD","year":"2013","journal-title":"International Conference on Intelligent Transportation Systems (ITSC)"},{"key":"bibr12-0278364920961809","author":"Chen YF","year":"2017","journal-title":"International Conference on Intelligent Robots and Systems (IROS)"},{"key":"bibr13-0278364920961809","author":"Diaz M","year":"2017","journal-title":"International Conference on Computer Vision Workshops (ICCV Workshops)"},{"key":"bibr14-0278364920961809","author":"Fang Z","year":"2018","journal-title":"arXiv preprint arXiv:1807.10580"},{"key":"bibr15-0278364920961809","author":"Gupta A","year":"2018","journal-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"bibr16-0278364920961809","author":"Habibi G","year":"2018","journal-title":"arXiv preprint arXiv:1806.09453"},{"key":"bibr17-0278364920961809","author":"He K","year":"2016","journal-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"bibr18-0278364920961809","author":"He K","year":"2016","journal-title":"European Conference on Computer Vision (ECCV)"},{"key":"bibr19-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.51.4282"},{"key":"bibr20-0278364920961809","author":"Hu J","year":"2017","journal-title":"arXiv preprint arXiv:1709.01507"},{"key":"bibr21-0278364920961809","author":"Huang G","year":"2016","journal-title":"arXiv preprint arXiv:1608.06993"},{"key":"bibr22-0278364920961809","author":"Iandola FN","year":"2016","journal-title":"arXiv preprint arXiv:1602.07360"},{"key":"bibr23-0278364920961809","author":"Jaipuria N","year":"2018","journal-title":"arXiv preprint arXiv:1806.09444"},{"key":"bibr24-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2015.2509509"},{"key":"bibr25-0278364920961809","author":"John V","year":"2014","journal-title":"International Conference on Intelligent Transportation Systems (ITSC)"},{"key":"bibr26-0278364920961809","author":"Kim B","year":"2017","journal-title":"arXiv preprint arXiv:1704.07049"},{"key":"bibr27-0278364920961809","author":"Kingma DP","year":"2014","journal-title":"arXiv preprint arXiv:1412.6980"},{"key":"bibr28-0278364920961809","author":"Kretzschmar H","year":"2014","journal-title":"International Conference on Robotics and Automation (ICRA)"},{"key":"bibr29-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1177\/0278364915619772"},{"key":"bibr30-0278364920961809","author":"Kuderer M","year":"2012","journal-title":"Proceedings of Robotics: Science and Systems"},{"key":"bibr31-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1002\/rob.21534"},{"key":"bibr32-0278364920961809","author":"Lea C","year":"2017","journal-title":"Proceedings of CVPR"},{"key":"bibr33-0278364920961809","author":"Lef\u00e8vre S","year":"2011","journal-title":"Intelligent Vehicles Symposium (IV)"},{"key":"bibr34-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1186\/s40648-014-0001-z"},{"key":"bibr35-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8659.2007.01089.x"},{"key":"bibr36-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2017.2749971"},{"key":"bibr37-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2016.2515338"},{"key":"bibr38-0278364920961809","author":"Manh H","year":"2018","journal-title":"arXiv preprint arXiv:1808.04018"},{"key":"bibr39-0278364920961809","author":"Medina AIM","year":"2015","journal-title":"International Conference on Intelligent Transportation Systems (ITSC)"},{"key":"bibr40-0278364920961809","unstructured":"Nexar (2016) Nexar challenge-1: Using deep learning for traffic light recognition. https:\/\/www.getnexar.com\/challenge-1\/."},{"key":"bibr41-0278364920961809","author":"Park S","year":"2018","journal-title":"arXiv preprint arXiv:1802.06338"},{"key":"bibr42-0278364920961809","author":"Pellegrini S","year":"2009","journal-title":"IEEE International Conference on Computer Vision (ICCV)"},{"key":"bibr43-0278364920961809","author":"Pfeiffer M","year":"2018","journal-title":"International Conference on Robotics and Automation (ICRA)"},{"key":"bibr44-0278364920961809","author":"Pfeiffer M","year":"2016","journal-title":"International Conference on Intelligent Robots and Systems (IROS)"},{"key":"bibr45-0278364920961809","author":"Radwan N","year":"2017","journal-title":"International Conference on Intelligent Robots and Systems (IROS)"},{"key":"bibr46-0278364920961809","author":"Redmon J","year":"2016","journal-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"bibr47-0278364920961809","author":"Sadeghian A","year":"2018","journal-title":"arXiv preprint arXiv:1806.01482"},{"key":"bibr48-0278364920961809","author":"Selvaraju RR","year":"2016","journal-title":"CoRR abs\/1610.02391"},{"key":"bibr49-0278364920961809","author":"Sun L","year":"2018","journal-title":"International Conference on Robotics and Automation (ICRA)"},{"key":"bibr50-0278364920961809","author":"Trautman P","year":"2010","journal-title":"International Conference on Intelligent Robots and Systems (IROS)"},{"key":"bibr51-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1177\/0278364914557874"},{"key":"bibr52-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1177\/0278364917727062"},{"key":"bibr53-0278364920961809","first-page":"23","author":"Valada A","year":"2016","journal-title":"IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop, State Estimation and Terrain Perception for All Terrain Mobile Robots"},{"key":"bibr54-0278364920961809","first-page":"3","volume":"1","author":"Valada A","year":"2018","journal-title":"Workshop on Learning and Inference in Robotics: Integrating Structure, Priors and Models at Robotics: Science and Systems (RSS)"},{"key":"bibr55-0278364920961809","author":"Van den Berg J","year":"2008","journal-title":"International Conference on Robotics and Automation (ICRA)"},{"key":"bibr56-0278364920961809","first-page":"2579","volume":"9","author":"Van der Maaten L","year":"2008","journal-title":"Journal of Machine Learning Research"},{"key":"bibr57-0278364920961809","author":"Varshneya D","year":"2017","journal-title":"arXiv preprint arXiv:1705.09436"},{"key":"bibr58-0278364920961809","author":"Vemula A","year":"2018","journal-title":"International Conference on Robotics and Automation (ICRA)"},{"key":"bibr59-0278364920961809","doi-asserted-by":"publisher","DOI":"10.1109\/WACV.2018.00135"},{"key":"bibr60-0278364920961809","author":"Yamaguchi K","year":"2011","journal-title":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)"},{"key":"bibr61-0278364920961809","author":"Yan Z","year":"2017","journal-title":"International Conference on Intelligent Robots and Systems (IROS)"}],"container-title":["The International Journal of Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0278364920961809","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/0278364920961809","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0278364920961809","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,3]],"date-time":"2025-03-03T01:55:20Z","timestamp":1740966920000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/0278364920961809"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,1]]},"references-count":61,"journal-issue":{"issue":"13","published-print":{"date-parts":[[2020,11]]}},"alternative-id":["10.1177\/0278364920961809"],"URL":"https:\/\/doi.org\/10.1177\/0278364920961809","relation":{},"ISSN":["0278-3649","1741-3176"],"issn-type":[{"value":"0278-3649","type":"print"},{"value":"1741-3176","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,1]]}}}