{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:07:04Z","timestamp":1760242024804,"version":"build-2065373602"},"reference-count":53,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2018,11,19]],"date-time":"2018-11-19T00:00:00Z","timestamp":1542585600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>State estimation is crucial for robot autonomy, visual odometry (VO) has received significant attention in the robotics field because it can provide accurate state estimation. However, the accuracy and robustness of most existing VO methods are degraded in complex conditions, due to the limited field of view (FOV) of the utilized camera. In this paper, we present a novel tightly-coupled multi-keyframe visual-inertial odometry (called VINS-MKF), which can provide an accurate and robust state estimation for robots in an indoor environment. We first modify the monocular ORBSLAM (Oriented FAST and Rotated BRIEF Simultaneous Localization and Mapping) to multiple fisheye cameras alongside an inertial measurement unit (IMU) to provide large FOV visual-inertial information. Then, a novel VO framework is proposed to ensure the efficiency of state estimation, by adopting a GPU (Graphics Processing Unit) based feature extraction method and parallelizing the feature extraction thread that is separated from the tracking thread with the mapping thread. Finally, a nonlinear optimization method is formulated for accurate state estimation, which is characterized as being multi-keyframe, tightly-coupled and visual-inertial. In addition, accurate initialization and a novel MultiCol-IMU camera model are coupled to further improve the performance of VINS-MKF. To the best of our knowledge, it\u2019s the first tightly-coupled multi-keyframe visual-inertial odometry that joins measurements from multiple fisheye cameras and IMU. The performance of the VINS-MKF was validated by extensive experiments using home-made datasets, and it showed improved accuracy and robustness over the state-of-art VINS-Mono.<\/jats:p>","DOI":"10.3390\/s18114036","type":"journal-article","created":{"date-parts":[[2018,11,22]],"date-time":"2018-11-22T09:18:25Z","timestamp":1542878305000},"page":"4036","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["VINS-MKF: A Tightly-Coupled Multi-Keyframe Visual-Inertial Odometry for Accurate and Robust State Estimation"],"prefix":"10.3390","volume":"18","author":[{"given":"Chaofan","family":"Zhang","sequence":"first","affiliation":[{"name":"Institute of Applied Technology, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China"},{"name":"Science Island Branch of Graduate School, University of Science and Technology of China, Hefei 230026, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yong","family":"Liu","sequence":"additional","affiliation":[{"name":"Institute of Applied Technology, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fan","family":"Wang","sequence":"additional","affiliation":[{"name":"Institute of Applied Technology, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China"},{"name":"Science Island Branch of Graduate School, University of Science and Technology of China, Hefei 230026, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yingwei","family":"Xia","sequence":"additional","affiliation":[{"name":"Institute of Applied Technology, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wen","family":"Zhang","sequence":"additional","affiliation":[{"name":"Institute of Applied Technology, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2018,11,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1109\/MRA.2011.943233","article-title":"Visual Odometry Part I: The First 30 Years and Fundamentals","volume":"18","author":"Scaramuzza","year":"2011","journal-title":"IEEE Robot. Autom Mag."},{"key":"ref_2","unstructured":"Nist\u00e9r, D., Naroditsky, O., and Bergen, J. (July, January 27). Visual odometry. Proceedings of the 2004 IEEE International Conference on Computer Vision and Pattern Recognition(CVPR), Washington, DC, USA."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Forster, C., Pizzoli, M., and Scaramuzza, D. (June, January 31). SVO: Fast Semi-Direct Monocular Visual Odometry. Proceedings of the 2014 IEEE International Conference on Robotics and Automation(ICRA), Hong Kong, China.","DOI":"10.1109\/ICRA.2014.6906584"},{"key":"ref_4","unstructured":"Singh, A. (2017, March 30). Monocular Visual Odometry. Available online: http:\/\/avisingh599.github.io\/assets\/ugp2-report.pdf."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Song, S., Chandraker, M., and Guest, C. (2013, January 6\u201310). Parallel, real-time monocular visual odometry. Proceedings of the 2013 IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany.","DOI":"10.1109\/ICRA.2013.6631246"},{"key":"ref_6","unstructured":"Olson, C.F., Matthies, L.H., Schoppers, M., and Maimone, M.W. (2000, January 15). Robust stereo ego-motion for long distance navigation. Proceedings of the 2000 IEEE International Conference on Computer Vision and Pattern Recognition(CVPR), Hilton Head Island, SC, USA."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Witt, J., and Weltin, U. (2013, January 3\u20138). Robust stereo visual odometry using iterative closest multiple lines. Proceedings of the 2013 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan.","DOI":"10.1109\/IROS.2013.6696953"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Wang, R., Schw\u00f6rer, M., and Cremers, D. (2017, January 22\u201329). Stereo dso: Large-scale direct sparse visual odometry with stereo cameras. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.421"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Valiente, D., Gil, A., Reinoso, O., Julia, M., and Holloway, M. (2017). Improved Omnidirectional Odometry for a View-Based Mapping Approach. Sensors, 17.","DOI":"10.3390\/s17020325"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Ouerghi, S., Boutteau, R., Savatier, X., and Thai, F. (2018). Visual Odometry and Place Recognition Fusion for Vehicle Position Tracking in Urban Environments. Sensors, 18.","DOI":"10.3390\/s18040939"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1177\/0278364914554813","article-title":"Keyframe-based visual-inertial odometry using nonlinear optimization","volume":"34","author":"Leutenegger","year":"2015","journal-title":"Int. J. Robot. Res."},{"key":"ref_12","unstructured":"Frahm, J.M., Koser, K., and Koch, R. (2006, January 15\u201319). Pose estimation for multi-camera systems. Proceedings of the 2004 Annual Pattern Recognition of the German-Association-for-Pattern-Recognition, Orlando, FL, USA."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Zhao, C., Fan, B., Hu, J., Tian, L., Zhang, Z., Li, S., and Pan, Q. (2017, January 27\u201329). Pose estimation for multi-camera systems. Proceedings of the 2017 IEEE International Conference on Unmanned Systems (ICUS), Beijing, China.","DOI":"10.1109\/ICUS.2017.8278403"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1007\/s10514-015-9466-8","article-title":"Self-calibration and visual SLAM with a multi-camera system on a micro aerial vehicle","volume":"39","author":"Heng","year":"2015","journal-title":"Auton. Robots"},{"key":"ref_15","unstructured":"Pless, R. (2003, January 18\u201320). Using many cameras as one. Proceedings of the 2003 IEEE International Conference on Computer Vision and Pattern Recognition(CVPR), Madison, WI, USA."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"291","DOI":"10.1007\/s10846-014-0085-y","article-title":"Multi-Camera Tracking and Mapping for Unmanned Aerial Vehicles in Unstructured Environments","volume":"78","author":"Harmat","year":"2015","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1016\/j.robot.2017.03.018","article-title":"Multi-camera visual SLAM for autonomous navigation of micro aerial vehicles","volume":"93","author":"Yang","year":"2017","journal-title":"Robot. Autom. Syst."},{"key":"ref_18","unstructured":"Urban, S., and Hinz, S. (arXiv, 2016). MultiCol-SLAM\u2014A Modular Real-Time Multi-Camera SLAM System, arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1177\/0278364907079279","article-title":"An introduction to inertial and visual sensing","volume":"26","author":"Corke","year":"2007","journal-title":"Int. J. Robot. Res."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"690","DOI":"10.1177\/0278364913481251","article-title":"High-precision, consistent EKF-based visual-inertial odometry","volume":"32","author":"Li","year":"2013","journal-title":"Int. J. Robot. Res."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Weiss, S., and Siegwart, R.Y. (2011, January 9\u201313). Real-Time Metric State Estimation for Modular Vision-Inertial Systems. Proceedings of the 2011 IEEE International Conference on Robotics and Automation(ICRA), Shanghai, China.","DOI":"10.1109\/ICRA.2011.5979982"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Shen, S., Michael, N., and Kumar, V. (2015, January 26\u201330). Tightly-Coupled Monocular Visual-Inertial Fusion for Autonomous Flight of Rotorcraft MAVs. Proceedings of the 2015 IEEE International Conference on Robotics and Automation(ICRA), Seattle, WA, USA.","DOI":"10.1109\/ICRA.2015.7139939"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TRO.2016.2597321","article-title":"On-Manifold Preintegration for Real-Time Visual--Inertial Odometry","volume":"33","author":"Forster","year":"2017","journal-title":"IEEE Trans. Robot."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"796","DOI":"10.1109\/LRA.2017.2653359","article-title":"Visual-Inertial Monocular SLAM With Map Reuse","volume":"2","author":"Tardos","year":"2017","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1004","DOI":"10.1109\/TRO.2018.2853729","article-title":"VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator","volume":"34","author":"Qin","year":"2018","journal-title":"IEEE Trans. Robot."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1147","DOI":"10.1109\/TRO.2015.2463671","article-title":"ORB-SLAM: A Versatile and Accurate Monocular SLAM System","volume":"31","author":"Montiel","year":"2015","journal-title":"IEEE Trans. Robot."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1186\/s41074-017-0027-2","article-title":"Visual SLAM algorithms: A survey from 2010 to 2016","volume":"9","author":"Taketomi","year":"2017","journal-title":"IPSJ Trans. Comput. Vis. Appl."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1052","DOI":"10.1109\/TPAMI.2007.1049","article-title":"MonoSLAM: Real-time single camera SLAM","volume":"29","author":"Davison","year":"2007","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1007\/978-3-319-23437-3_13","article-title":"Incorporating Static Environment Elements into the EKF-Based Visual SLAM","volume":"Volume 391","author":"Schmidt","year":"2016","journal-title":"Man\u2013Machine Interactions 4"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"609","DOI":"10.1002\/rob.20345","article-title":"1-Point RANSAC for Extended Kalman Filtering: Application to Real-Time Structure from Motion and Visual Odometry","volume":"27","author":"Civera","year":"2010","journal-title":"J. Field Robot."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Davison, A.J. (2003, January 13\u201316). Real-time simultaneous localisation and mapping with a single camera. Proceedings of the 2003 IEEE International Conference on Computer Vision, Nice, France.","DOI":"10.1109\/ICCV.2003.1238654"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Klein, G., and Murray, D. (2007, January 13\u201316). Parallel tracking and mapping for small AR workspaces. Proceedings of the 2007 IEEE and ACM International Symposium on Mixed and Augmented Reality(ISMAR), Nara, Japan.","DOI":"10.1109\/ISMAR.2007.4538852"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Engel, J., Schoeps, T., and Cremers, D. (2014, January 6\u201312). LSD-SLAM: Large-Scale Direct Monocular SLAM. Proceedings of the 2014 European Conference on Computer Vision (ECCV), Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10605-2_54"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1109\/TPAMI.2017.2658577","article-title":"Direct Sparse Odometry","volume":"40","author":"Engel","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Strasdat, H., Davison, A.J., Montiel, J.M.M., and Konolige, K. (2011, January 6\u201313). Double Window Optimisation for Constant Time Visual SLAM. Proceedings of the 2011 IEEE International Conference on Computer Vision, Barcelona, Spain.","DOI":"10.1109\/ICCV.2011.6126517"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Valiente, D., Paya, L., Jimenez, L.M., Sebastian, J.M., and Reinoso, O. (2018). Visual Information Fusion through Bayesian Inference for Adaptive Probability-Oriented Feature Matching. Sensors, 18.","DOI":"10.3390\/s18072041"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Strasdat, H., Montiel, J.M.M., and Davison, A.J. (2010, January 3\u20138). Real-time Monocular SLAM: Why Filter?. Proceedings of the 2010 IEEE International Conference on Robotics and Automation, Anchorage, AK, USA.","DOI":"10.1109\/ROBOT.2010.5509636"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"837","DOI":"10.1177\/0278364914557969","article-title":"Minimal solutions for the multi-camera pose estimation problem","volume":"34","author":"Lee","year":"2015","journal-title":"Int. J. Robot. Res."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Harmat, A., Sharf, I., and Trentini, M. (2012, January 3\u20135). Parallel tracking and mapping with multiple cameras on an unmanned aerial vehicle. Proceedings of the 2012 IEEE International Conference on Intelligent Robotics and Applications, Montreal, QC, Canada.","DOI":"10.1007\/978-3-642-33509-9_42"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1109\/TPAMI.2012.104","article-title":"CoSLAM: Collaborative Visual SLAM in Dynamic Environments","volume":"35","author":"Zou","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1007\/s00138-013-0541-x","article-title":"Intrinsic and extrinsic active self-calibration of multi-camera systems","volume":"25","author":"Brueckner","year":"2014","journal-title":"Mach. Vision Appl."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Lynen, S., Achtelik, M.W., Weiss, S., Chli, M., and Siegwart, R. (2013, January 3\u20138). A Robust and Modular Multi-Sensor Fusion Approach Applied to MAV Navigation. Proceedings of the 2013 IEEE International Conference on Intelligent Robots and Systems, Tokyo, Japan.","DOI":"10.1109\/IROS.2013.6696917"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Qin, T., and Shen, S. (2017, January 24\u201328). Robust Initialization of Monocular Visual-Inertial Estimation on Aerial Robots. Proceedings of the 2017 IEEE International Conference on Intelligent Robots and Systems, Vancouver, BC, Canada.","DOI":"10.1109\/IROS.2017.8206284"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Houben, S., Quenzel, J., Krombach, N., and Behnke, S. (2016, January 9\u201314). Efficient multi-camera visual-inertial SLAM for micro aerial vehicles. Proceedings of the 2016 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea.","DOI":"10.1109\/IROS.2016.7759261"},{"key":"ref_45","unstructured":"Thrun, S., Burgard, W., and Fox, D. (2005). Probabilistic Robotics, MIT press."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1109\/TRO.2011.2170332","article-title":"Visual-Inertial-Aided Navigation for High-Dynamic Motion in Built Environments Without Initial Conditions","volume":"28","author":"Lupton","year":"2012","journal-title":"IEEE Trans. Robot."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Forster, C., Carlone, L., Dellaert, F., and Scaramuzza, D. (2015, January 13\u201317). IMU preintegration on manifold for efficient visual-inertial maximum-a-posteriori estimation. Proceedings of the 2015 Robotics: Science and Systems, Rome, Italy.","DOI":"10.15607\/RSS.2015.XI.006"},{"key":"ref_48","first-page":"105","article-title":"The Levenberg-Marquardt algorithm: Implementation and theory","volume":"Volume 630","year":"1978","journal-title":"Numerical Analysis"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Nikolic, J., Rehder, J., Burri, M., Gohl, P., Leutenegger, S., Furgale, P.T., and Siegwart, R. (June, January 31). A synchronized visual-inertial sensor system with FPGA pre-processing for accurate real-time SLAM. Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China.","DOI":"10.1109\/ICRA.2014.6906892"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Furgale, P., Rehder, J., and Siegwart, R. (2013, January 3\u20138). Unified Temporal and Spatial Calibration for Multi-Sensor Systems. Proceedings of the 2013 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan.","DOI":"10.1109\/IROS.2013.6696514"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1109\/TASE.2016.2550621","article-title":"Monocular Visual-Inertial State Estimation With Online Initialization and Camera-IMU Extrinsic Calibration","volume":"14","author":"Yang","year":"2017","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Kneip, L., Furgale, P., and Siegwart, R. (2013, January 6\u201310). Using Multi-Camera Systems in Robotics: Efficient Solutions to the NPnP Problem. Proceedings of the 2013 IEEE International Conference on Robotics and Automation, Karlsruhe, Germany.","DOI":"10.1109\/ICRA.2013.6631107"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Sturm, J., Engelhard, N., Endres, F., Burgard, W., and Cremers, D. (, January 7\u201312). Benchmark for the Evaluation of RGB-D SLAM Systems. Proceedings of the 2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Algarve, Portugal.","DOI":"10.1109\/IROS.2012.6385773"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/11\/4036\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:30:42Z","timestamp":1760196642000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/11\/4036"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,11,19]]},"references-count":53,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2018,11]]}},"alternative-id":["s18114036"],"URL":"https:\/\/doi.org\/10.3390\/s18114036","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2018,11,19]]}}}