{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,28]],"date-time":"2025-10-28T10:54:54Z","timestamp":1761648894342,"version":"build-2065373602"},"reference-count":46,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2021,6,7]],"date-time":"2021-06-07T00:00:00Z","timestamp":1623024000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["688403"],"award-info":[{"award-number":["688403"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>With the emergence of low-cost robotic systems, such as unmanned aerial vehicle, the importance of embedded high-performance image processing has increased. For a long time, FPGAs were the only processing hardware that were capable of high-performance computing, while at the same time preserving a low power consumption, essential for embedded systems. However, the recently increasing availability of embedded GPU-based systems, such as the NVIDIA Jetson series, comprised of an ARM CPU and a NVIDIA Tegra GPU, allows for massively parallel embedded computing on graphics hardware. With this in mind, we propose an approach for real-time embedded stereo processing on ARM and CUDA-enabled devices, which is based on the popular and widely used Semi-Global Matching algorithm. In this, we propose an optimization of the algorithm for embedded CUDA GPUs, by using massively parallel computing, as well as using the NEON intrinsics to optimize the algorithm for vectorized SIMD processing on embedded ARM CPUs. We have evaluated our approach with different configurations on two public stereo benchmark datasets to demonstrate that they can reach an error rate as low as 3.3%. Furthermore, our experiments show that the fastest configuration of our approach reaches up to 46 FPS on VGA image resolution. Finally, in a use-case specific qualitative evaluation, we have evaluated the power consumption of our approach and deployed it on the DJI Manifold 2-G attached to a DJI Matrix 210v2 RTK unmanned aerial vehicle (UAV), demonstrating its suitability for real-time stereo processing onboard a UAV.<\/jats:p>","DOI":"10.3390\/s21113938","type":"journal-article","created":{"date-parts":[[2021,6,7]],"date-time":"2021-06-07T22:23:00Z","timestamp":1623104580000},"page":"3938","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["ReS2tAC\u2014UAV-Borne Real-Time SGM Stereo Optimized for Embedded ARM and CUDA Devices"],"prefix":"10.3390","volume":"21","author":[{"given":"Boitumelo","family":"Ruf","sequence":"first","affiliation":[{"name":"Fraunhofer Center for Machine Learning, Fraunhofer Institute of Optronics, System Technologies and Image Exploitation (IOSB), 76131 Karlsruhe, Germany"},{"name":"Institute of Photogrammetry and Remote Sensing, Karlsruhe Institute of Technology (KIT), 76131 Karlsruhe, Germany"}]},{"given":"Jonas","family":"Mohrs","sequence":"additional","affiliation":[{"name":"Fraunhofer Center for Machine Learning, Fraunhofer Institute of Optronics, System Technologies and Image Exploitation (IOSB), 76131 Karlsruhe, Germany"}]},{"given":"Martin","family":"Weinmann","sequence":"additional","affiliation":[{"name":"Institute of Photogrammetry and Remote Sensing, Karlsruhe Institute of Technology (KIT), 76131 Karlsruhe, Germany"}]},{"given":"Stefan","family":"Hinz","sequence":"additional","affiliation":[{"name":"Institute of Photogrammetry and Remote Sensing, Karlsruhe Institute of Technology (KIT), 76131 Karlsruhe, Germany"}]},{"given":"J\u00fcrgen","family":"Beyerer","sequence":"additional","affiliation":[{"name":"Fraunhofer Center for Machine Learning, Fraunhofer Institute of Optronics, System Technologies and Image Exploitation (IOSB), 76131 Karlsruhe, Germany"},{"name":"Vision and Fusion Laboratory, Karlsruhe Institute of Technology (KIT), 76131 Karlsruhe, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2021,6,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s12518-013-0120-x","article-title":"UAV for 3D mapping applications: A review","volume":"6","author":"Nex","year":"2014","journal-title":"Appl. Geomat."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"316","DOI":"10.4236\/wjet.2015.33C047","article-title":"Drone applications for supporting disaster management","volume":"03","author":"Restas","year":"2015","journal-title":"World J. Eng. Technol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1108\/AEAT-01-2018-0056","article-title":"UAV application for precision agriculture","volume":"91","author":"Perz","year":"2019","journal-title":"Aircr. Eng. Aerosp. Technol."},{"key":"ref_4","unstructured":"Sebbane, Y.B. (2018). Intelligent Autonomy of UAVs: Advanced Missions and Future Use, CRC Press."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"48572","DOI":"10.1109\/ACCESS.2019.2909530","article-title":"Unmanned Aerial Vehicles (UAVs): A survey on civil applications and key research challenges","volume":"7","author":"Shakhatreh","year":"2019","journal-title":"IEEE Access"},{"key":"ref_6","first-page":"107","article-title":"LiDAR or photogrammetry? Integration is the answer","volume":"43","author":"Nex","year":"2011","journal-title":"Eur. J. Remote Sens."},{"key":"ref_7","unstructured":"Hirschmueller, H. (2005, January 20\u201326). Accurate and efficient stereo processing by semi-global matching and mutual information. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1109\/TPAMI.2007.1166","article-title":"Stereo processing by semi-global matching and mutual information","volume":"30","author":"Hirschmueller","year":"2008","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Gehrig, S.K., Eberli, F., and Meyer, T. (2009, January 13\u201315). A real-time low-power stereo vision engine using semi-global matching. Proceedings of the International Conference on Computer Vision Systems, Li\u00e8ge, Belgium.","DOI":"10.1007\/978-3-642-04667-4_14"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Banz, C., Hesselbarth, S., Flatt, H., Blume, H., and Pirsch, P. (2010, January 19\u201322). Real-time stereo vision system using semi-global matching disparity estimation: Architecture and FPGA-implementation. Proceedings of the International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation, Samos, Greece.","DOI":"10.1109\/ICSAMOS.2010.5642077"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1696","DOI":"10.1109\/TCSVT.2015.2397196","article-title":"Real-Time High-Quality Stereo Vision System in FPGA","volume":"25","author":"Wang","year":"2015","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Schmid, K., Tomic, T., Ruess, F., Hirschmueller, H., and Suppa, M. (2013, January 3\u20137). Stereo vision based indoor\/outdoor navigation for flying robots. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan.","DOI":"10.1109\/IROS.2013.6696922"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Gehrig, S.K., and Rabe, C. (2010, January 13\u201318). Real-time semi-global matching on the CPU. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, San Francisco, CA, USA.","DOI":"10.1109\/CVPRW.2010.5543779"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Honegger, D., Oleynikova, H., and Pollefeys, M. (2014, January 14\u201318). Real-time and low latency embedded computer vision hardware based on a combination of FPGA and mobile CPU. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems, Chicago, IL, USA.","DOI":"10.1109\/IROS.2014.6943263"},{"key":"ref_15","unstructured":"Barry, A.J., Oleynikova, H., Honegger, D., Pollefeys, M., and Tedrake, R. (October, January 28). FPGA vs. pushbroom stereo vision for MAVs. Proceedings of the IROS Workshop on Vision-based Control and Navigation of Small Lightweight UAVs, Hamburg, Germany."},{"key":"ref_16","unstructured":"Hofmann, J., Korinth, J., and Koch, A. (July, January 26). A scalable high-performance hardware architecture for real-time stereo vision by semi-global matching. In Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Las Vegas, NV, USA."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Rahnama, O., Cavalleri, T., Golodetz, S., Walker, S., and Torr, P. (2018, January 10\u201314). R3SGM: Real-time raster-respecting semi-global matching for power-constrained systems. Proceedings of the IEEE International Conference on Field-Programmable Technology, Naha, Japan.","DOI":"10.1109\/FPT.2018.00025"},{"key":"ref_18","unstructured":"Zhao, J., Liang, T., Feng, L., Ding, W., Sinha, S., Zhang, W., and Shen, S. (September, January 31). FP-Stereo: Hardware-efficient stereo vision for embedded applications. Proceedings of the IEEE International Conference on Field-Programmable Logic and Applications, Gothenburg, Sweden."},{"key":"ref_19","first-page":"363","article-title":"Real-time on-board obstacle avoidance for UAVs based on embedded stereo vision","volume":"XLII-1","author":"Ruf","year":"2018","journal-title":"ISPRS Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kalb, T., Kalms, L., G\u00f6hringer, D., Pons, C., Muddukrishna, A., Jahre, M., Ruf, B., Schuchert, T., Tchouchenkov, I., and Ehrenstr\u00e5hle, C. (2019). Developing low-power image processing applications with the TULIPP reference platform instance. Hardware Accelerators in Data Centers, Springer.","DOI":"10.1007\/978-3-319-92792-3_10"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Rosenberg, I.D., Davidson, P.L., Muller, C.M., and Han, J.Y. (2006). Real-time stereo vision using semi-global matching on programmable graphics hardware. ACM SIGGRAPH 2006 Sketches, Association for Computing Machinery.","DOI":"10.1145\/1179849.1179960"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Ernst, I., and Hirschm\u00fcller, H. (2008, January 1\u20133). Mutual information based semi-global stereo matching on the GPU. Proceedings of the International Symposium on Advances in Visual Computing, Las Vegas, NV, USA.","DOI":"10.1007\/978-3-540-89639-5_22"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Banz, C., Blume, H., and Pirsch, P. (2011, January 6\u201313). Real-time semi-global matching disparity estimation on the GPU. Proceedings of the IEEE International Conference on Computer Vision Workshops, Barcelona, Spain.","DOI":"10.1109\/ICCVW.2011.6130286"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Michael, M., Salmen, J., Stallkamp, J., and Schlipsing, M. (2013, January 23\u201326). Real-time stereo vision: Optimizing semi-global matching. Proceedings of the IEEE Intelligent Vehicles Symposium, Gold Coast City, Australia.","DOI":"10.1109\/IVS.2013.6629629"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1016\/j.procs.2016.05.305","article-title":"Embedded real-time stereo estimation via semi-global matching on the GPU","volume":"80","author":"Espinosa","year":"2016","journal-title":"Procedia Comput. Sci."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Chang, Q., Zha, A., Wang, W., Liu, X., Onishi, M., and Maruyama, T. (2020, January 18\u201321). Z2-ZNCC: ZigZag scanning based zero-means normalized cross correlation for fast and accurate stereo matching on embedded GPU. Proceedings of the IEEE International Conference on Computer Design, Hartford, CT, USA.","DOI":"10.1109\/ICCD50377.2020.00104"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Spangenberg, R., Langner, T., Adfeldt, S., and Rojas, R. (2014, January 8\u201311). Large scale semi-global matching on the CPU. Proceedings of the IEEE Intelligent Vehicles Symposium, Dearborn, MI, USA.","DOI":"10.1109\/IVS.2014.6856419"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Arndt, O.J., Becker, D., Banz, C., and Blume, H. (2013, January 15\u201318). Parallel implementation of real-time semi-global matching on embedded multi-core architectures. Proceedings of the International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation, Samos, Greece.","DOI":"10.1109\/SAMOS.2013.6621106"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"2008","DOI":"10.1109\/LRA.2018.2800786","article-title":"Real-Time dense stereo matching with ELAS on FPGA-accelerated embedded devices","volume":"3","author":"Rahnama","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_30","unstructured":"Geiger, A., Roser, M., and Urtasun, R. Efficient large-scale stereo matching. Proceedings of the Asian Conference on Computer Vision."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Saidi, T.E., Khouas, A., and Amira, A. (2020, January 10\u201321). Accelerating stereo matching on mutlicore ARM platform. Proceedings of the IEEE International Symposium on Circuits and Systems, Seville, Spain.","DOI":"10.1109\/ISCAS45731.2020.9180982"},{"key":"ref_32","unstructured":"ARM (2013). NEON Programmers Guide, ARM."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1023\/A:1014573219977","article-title":"A taxonomy and evaluation of dense two-frame stereo correspondence algorithms","volume":"47","author":"Scharstein","year":"2002","journal-title":"Int. J. Comput. Vis."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1023\/B:VISI.0000029664.99615.94","article-title":"Distinctive image features from scale-invariant keypoints","volume":"60","author":"Lowe","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Bay, H., Tuytelaars, T., and Van Gool, L. (2006, January 7\u201313). SURF: Speeded up robust features. Proceedings of the European Conference on Computer Vision, Graz, Austria.","DOI":"10.1007\/11744023_32"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Rublee, E., Rabaud, V., Konolige, K., and Bradski, G. (2011, January 6\u201313). ORB: An efficient alternative to SIFT or SURF. Proceedings of the International Conference on Computer Vision, Barcelona, Spain.","DOI":"10.1109\/ICCV.2011.6126544"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Zabih, R., and Woodfill, J. (1994, January 2\u20136). Non-parametric local transforms for computing visual correspondence. Proceedings of the European Conference on Computer Vision, Stockholm, Sweden.","DOI":"10.1007\/BFb0028345"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1330","DOI":"10.1109\/34.888718","article-title":"A Flexible New Technique for Camera Calibration","volume":"22","author":"Zhang","year":"2000","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Menze, M., and Geiger, A. (2015, January 7\u201312). Object scene flow for autonomous vehicles. Proceedings of the Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298925"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Scharstein, D., Hirschm\u00fcller, H., Kitajima, Y., Krathwohl, G., Ne\u0161i\u0107, N., Wang, X., and Westling, P. (2014, January 2\u20135). High-resolution stereo datasets with subpixel-accurate ground truth. Proceedings of the German Conference on Pattern Recognition, M\u00fcnster, Germany.","DOI":"10.1007\/978-3-319-11752-2_3"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Cui, H., and Dahnoun, N. (2019, January 10\u201314). Real-time stereo vision implementation on Nvidia Jetson TX2. Proceedings of the Mediterranean Conference on Embedded Computing, Budva, Montenegro.","DOI":"10.1109\/MECO.2019.8760027"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Sch\u00f6nberger, J.L., Sinha, S.N., and Pollefeys, M. (2018, January 8\u201314). Learning to fuse proposals from multiple scanline optimizations in semi-global matching. Proceedings of the European Conference on Computer Vision, Munich, Germany.","DOI":"10.1007\/978-3-030-01261-8_45"},{"key":"ref_43","unstructured":"(2021, March 19). DJI Onboard SDK Advanced Sensing-Stereo Camera. Available online: https:\/\/developer.dji.com\/onboard-sdk\/documentation\/guides\/component-guide-advanced-sensing-stereo-camera.html."},{"key":"ref_44","unstructured":"DJI (2020). Matrice 200 V2-Serie-User Manual, DJI."},{"key":"ref_45","unstructured":"ARM (2017). ARM Architecture Reference Manual for ARMv8 Architecture Profile, ARM."},{"key":"ref_46","first-page":"513","article-title":"Sorting and searching","volume":"3","author":"Knuth","year":"1998","journal-title":"Art Comput. Program."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/11\/3938\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:11:50Z","timestamp":1760163110000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/11\/3938"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,7]]},"references-count":46,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2021,6]]}},"alternative-id":["s21113938"],"URL":"https:\/\/doi.org\/10.3390\/s21113938","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,6,7]]}}}