{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,29]],"date-time":"2025-10-29T06:24:58Z","timestamp":1761719098306,"version":"build-2065373602"},"reference-count":34,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2021,11,29]],"date-time":"2021-11-29T00:00:00Z","timestamp":1638144000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Driver situation awareness is critical for safety. In this paper, we propose a fast, accurate method for obtaining real-time situation awareness using a single type of sensor: monocular cameras. The system tracks the host vehicle\u2019s trajectory using sparse optical flow and tracks vehicles in the surrounding environment using convolutional neural networks. Optical flow is used to measure the linear and angular velocity of the host vehicle. The convolutional neural networks are used to measure target vehicles\u2019 positions relative to the host vehicle using image-based detections. Finally, the system fuses host and target vehicle trajectories in the world coordinate system using the velocity of the host vehicle and the target vehicles\u2019 relative positions with the aid of an Extended Kalman Filter (EKF). We implement and test our model quantitatively in simulation and qualitatively on real-world test video. The results show that the algorithm is superior to state-of-the-art sequential state estimation methods such as visual SLAM in performing accurate global localization and trajectory estimation for host and target vehicles.<\/jats:p>","DOI":"10.3390\/s21237969","type":"journal-article","created":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T01:45:02Z","timestamp":1638323102000},"page":"7969","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Vehicle Trajectory Estimation Based on Fusion of Visual Motion Features and Deep Learning"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4794-7624","authenticated-orcid":false,"given":"Lianen","family":"Qu","sequence":"first","affiliation":[{"name":"Department of Information and Communication Technologies, Asian Institute of Technology, Klong Luang, Pathum Thani 12120, Thailand"},{"name":"International College, Qingdao University of Science and Technology, Qingdao 266061, China"}]},{"given":"Matthew N.","family":"Dailey","sequence":"additional","affiliation":[{"name":"Department of Information and Communication Technologies, Asian Institute of Technology, Klong Luang, Pathum Thani 12120, Thailand"}]}],"member":"1968","published-online":{"date-parts":[[2021,11,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1627","DOI":"10.1109\/TPAMI.2009.167","article-title":"Object detection with discriminatively trained part-based models","volume":"32","author":"Felzenszwalb","year":"2009","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","unstructured":"Viola, P., and Jones, M. (2001, January 8\u201314). Rapid object detection using a boosted cascade of simple features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, Kauai, HI, USA."},{"key":"ref_3","unstructured":"Dalal, N., and Triggs, B. (2005, January 20\u201325). Histograms of Oriented Gradients for Human Detection. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201905), San Diego, CA, USA."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23\u201328). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.81"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Girshick, R. (2015, January 7\u201313). Fast R-CNN. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_6","unstructured":"Cortes, C., Lawrence, N., Lee, D., Sugiyamam, M., and Garnett, R. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems 28 (NIPS 2015), Neural Information Processing Systems Foundation Inc."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_8","unstructured":"Schubert, R., Richter, E., and Wanielik, G. (July, January 30). Comparison and evaluation of advanced motion models for vehicle tracking. Proceedings of the 2008 11th International Conference on Information Fusion, Cologne, Germany."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Berthelot, A., Tamke, A., Dang, T., and Breuel, G. (2011, January 5\u20139). Handling uncertainties in criticality assessment. Proceedings of the 2011 IEEE Intelligent Vehicles Symposium (IV), Baden-Baden, Germany.","DOI":"10.1109\/IVS.2011.5940483"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Tamke, A., Dang, T., and Breuel, G. (2011, January 5\u20139). A flexible method for criticality assessment in driver assistance systems. Proceedings of the 2011 IEEE Intelligent Vehicles Symposium (IV), Baden-Baden, Germany.","DOI":"10.1109\/IVS.2011.5940482"},{"key":"ref_11","unstructured":"Huang, J., and Tan, H.-S. (2006, January 14\u201316). Vehicle future trajectory prediction with a DGPS\/INS-based positioning system. Proceedings of the 2006 American Control Conference, Minneapolis, MN, USA."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Liu, P., Kurt, A., and \u00d6zg\u00fcner, \u00dc. (2014, January 8\u201314). Trajectory prediction of a lane changing vehicle based on driver behavior estimation and classification. Proceedings of the 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), Qingdao, China.","DOI":"10.1109\/ITSC.2014.6957810"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1209","DOI":"10.1109\/TITS.2011.2160342","article-title":"A new vehicle motion model for improved predictions and situation assessment","volume":"12","author":"Sorstedt","year":"2011","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1515\/auto-2016-0129","article-title":"Bayesian environment representation, prediction, and criticality assessment for driver assistance systems","volume":"65","author":"Schreier","year":"2017","journal-title":"at-Automatisierungstechnik"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Ammoun, S., and Nashashibi, F. (2009, January 27\u201329). Real time trajectory prediction for collision risk estimation between vehicles. Proceedings of the 2009 IEEE 5th International Conference on Intelligent Computer Communication and Processing, Cluj-Napoca, Romania.","DOI":"10.1109\/ICCP.2009.5284727"},{"key":"ref_16","unstructured":"Morris, D., Haley, P., Zachar, W., and McLean, S. (2017). Ladar-based vehicle tracking and trajectory estimation for urban driving. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Dickmann, J., Klappstein, J., Hahn, M., Appenrodt, N., Bloecher, H.-L., Werber, K., and Sailer, A. (2016, January 2\u20136). Automotive radar the key technology for autonomous driving: From detection and ranging to environmental understanding. Proceedings of the 2016 IEEE Radar Conference (RadarConf), Philadelphia, PA, USA.","DOI":"10.1109\/RADAR.2016.7485214"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Clarke, D., Andre, D., and Zhang, F. (2021, January 19\u201321). Synthetic aperture radar for lane boundary detection in driver assistance systems. Proceedings of the 2016 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), Baden-Baden, Germany.","DOI":"10.1109\/MFI.2016.7849495"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1007\/s10846-014-0081-2","article-title":"Joint localization of pursuit quadcopters and target using monocular cues","volume":"78","author":"Basit","year":"2015","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Wojke, N., Bewley, A., and Paulus, D. (2017, January 17\u201320). Simple online and realtime tracking with a deep association metric. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.","DOI":"10.1109\/ICIP.2017.8296962"},{"key":"ref_21","unstructured":"Yao, Z., Jiang, H., Cheng, Y., Jiang, Y., and Ran, B. (2020). Integrated Schedule and Trajectory Optimization for Connected Automated Vehicles in a Conflict Zone. IEEE Trans. Intell. Transp. Syst., 1\u201311."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"102759","DOI":"10.1016\/j.trc.2020.102759","article-title":"An analytical optimization approach to the joint trajectory and signal optimization problem for connected automated vehicles","volume":"120","author":"Soleimaniamiri","year":"2020","journal-title":"Transp. Res. Part C Emerg. Technol."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Zambrano-Martinez, J.L., Calafate, C.T., Soler, D., Cano, J.-C., and Manzoni, P. (2018). Modeling and Characterization of Traffic Flows in Urban Environments. Sensors, 18.","DOI":"10.3390\/s18072020"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1016\/j.trc.2010.10.002","article-title":"Real-time road traffic prediction with spatio-temporal correlations","volume":"19","author":"Min","year":"2011","journal-title":"Transp. Res. Part C Emerg. Technol."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Pandey, T., Pena, D., Byrne, J., and Moloney, D. (2021). Leveraging Deep Learning for Visual Odometry Using Optical Flow. Sensors, 21.","DOI":"10.3390\/s21041313"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"107187","DOI":"10.1016\/j.patcog.2019.107187","article-title":"PoseConvGRU: A Monocular Approach for Visual Ego-motion Estimation by Learning","volume":"102","author":"Zhai","year":"2020","journal-title":"Pattern Recognit."},{"key":"ref_27","unstructured":"Bian, J.W., Li, Z., Wang, N., Zhan, H., Shen, C., Cheng, M.M., and Reid, I. (2019, January 8\u201314). Unsupervised scale-consistent depth and ego-motion learning from monocular video. Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, BC, Canada."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Wang, S., Clark, R., Wen, H., and Trigoni, N. (June, January 29). Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.","DOI":"10.1109\/ICRA.2017.7989236"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Zhou, T., Brown, M., Snavely, N., and Lowe, D.G. (2017, January 21\u201326). Unsupervised Learning of Depth and Ego-Motion from Video. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.700"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Li, R., Wang, S., Long, Z., and Gu, D. (2018, January 21\u201325). Undeepvo: Monocular visual odometry through unsupervised deep learning. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia.","DOI":"10.1109\/ICRA.2018.8461251"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Munaro, M., Basso, F., and Menegatti, E. (2012, January 7\u201312). Tracking people within groups with RGB-D data. Proceedings of the 2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Vilamoura-Algarve, Portugal.","DOI":"10.1109\/IROS.2012.6385772"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1145\/358669.358692","article-title":"Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography","volume":"24","author":"Fischler","year":"1981","journal-title":"Commun. ACM"},{"key":"ref_33","unstructured":"Redmon, J., and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1147","DOI":"10.1109\/TRO.2015.2463671","article-title":"ORB-SLAM: A versatile and accurate monocular slam system","volume":"31","author":"Montiel","year":"2015","journal-title":"IEEE Trans. Robot."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/23\/7969\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:37:24Z","timestamp":1760168244000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/23\/7969"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,29]]},"references-count":34,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["s21237969"],"URL":"https:\/\/doi.org\/10.3390\/s21237969","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,11,29]]}}}