{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T16:58:01Z","timestamp":1781369881455,"version":"3.54.1"},"reference-count":42,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2024,3,13]],"date-time":"2024-03-13T00:00:00Z","timestamp":1710288000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Chinese Scholarship Council","award":["201907820021"],"award-info":[{"award-number":["201907820021"]}]},{"name":"Chinese Scholarship Council","award":["Tetra project AI2Source"],"award-info":[{"award-number":["Tetra project AI2Source"]}]},{"name":"Flemish Innovation agency VLAIO","award":["201907820021"],"award-info":[{"award-number":["201907820021"]}]},{"name":"Flemish Innovation agency VLAIO","award":["Tetra project AI2Source"],"award-info":[{"award-number":["Tetra project AI2Source"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>While Siamese object tracking has witnessed significant advancements, its hard real-time behaviour on embedded devices remains inadequately addressed. In many application cases, an embedded implementation should not only have a minimal execution latency, but this latency should ideally also have zero variance, i.e., be predictable. This study aims to address this issue by meticulously analysing real-time predictability across different components of a deep-learning-based video object tracking system. Our detailed experiments not only indicate the superiority of Field-Programmable Gate Array (FPGA) implementations in terms of hard real-time behaviour but also unveil important time predictability bottlenecks. We introduce dedicated hardware accelerators for key processes, focusing on depth-wise cross-correlation and padding operations, utilizing high-level synthesis (HLS). Implemented on a KV260 board, our enhanced tracker exhibits not only a speed up, with a factor of 6.6, in mean execution time but also significant improvements in hard real-time predictability by yielding 11 times less latency variation as compared to our baseline. A subsequent analysis of power consumption reveals our approach\u2019s contribution to enhanced power efficiency. These advancements underscore the crucial role of hardware acceleration in realizing time-predictable object tracking on embedded systems, setting new standards for future hardware\u2013software co-design endeavours in this domain.<\/jats:p>","DOI":"10.3390\/jimaging10030070","type":"journal-article","created":{"date-parts":[[2024,3,13]],"date-time":"2024-03-13T03:46:31Z","timestamp":1710301591000},"page":"70","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Enhancing Embedded Object Tracking: A Hardware Acceleration Approach for Real-Time Predictability"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8530-6257","authenticated-orcid":false,"given":"Mingyang","family":"Zhang","sequence":"first","affiliation":[{"name":"PSI-EAVISE Research Group, Department of Electrical Engineering, KU Leuven, 2860 Sint-Katelijne-Waver, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3667-7406","authenticated-orcid":false,"given":"Kristof","family":"Van Beeck","sequence":"additional","affiliation":[{"name":"PSI-EAVISE Research Group, Department of Electrical Engineering, KU Leuven, 2860 Sint-Katelijne-Waver, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7477-8961","authenticated-orcid":false,"given":"Toon","family":"Goedem\u00e9","sequence":"additional","affiliation":[{"name":"PSI-EAVISE Research Group, Department of Electrical Engineering, KU Leuven, 2860 Sint-Katelijne-Waver, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,3,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"110149","DOI":"10.1109\/ACCESS.2021.3101988","article-title":"Siamese visual object tracking: A survey","volume":"9","year":"2021","journal-title":"IEEE Access"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1109","DOI":"10.1007\/s00521-018-3761-1","article-title":"A survey of FPGA-based accelerators for convolutional neural networks","volume":"32","author":"Mittal","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Li, B., Wu, W., Wang, Q., Zhang, F., Xing, J., and Yan, J. (2019, January 16\u201320). SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00441"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1109\/34.1000236","article-title":"Mean shift: A robust approach toward feature space analysis","volume":"24","author":"Comaniciu","year":"2002","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1016\/j.cviu.2008.08.006","article-title":"Object tracking using SIFT features and mean shift","volume":"113","author":"Zhou","year":"2009","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"2165","DOI":"10.1016\/j.patrec.2008.08.007","article-title":"A spatial-color mean-shift object tracking algorithm with scale and orientation estimation","volume":"29","author":"Hu","year":"2008","journal-title":"Pattern Recognit. Lett."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1115\/1.3662552","article-title":"A new approach to linear filtering and prediction problems","volume":"82","author":"Kalman","year":"1960","journal-title":"Trans. ASME-J. Basic Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1190","DOI":"10.1016\/j.jvcir.2006.03.004","article-title":"Video object tracking using adaptive Kalman filter","volume":"17","author":"Weng","year":"2006","journal-title":"J. Vis. Commun. Image Represent."},{"key":"ref_9","first-page":"326","article-title":"Moving object tracking using kalman filter","volume":"2","author":"Patel","year":"2013","journal-title":"Int. J. Comput. Sci. Mob. Comput."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1409","DOI":"10.1109\/TPAMI.2011.239","article-title":"Tracking-learning-detection","volume":"34","author":"Kalal","year":"2011","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Hare, S., Saffari, A., and Torr, P.H.S. (2011, January 6\u201313). Struck: Structured output tracking with kernels. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Barcelona, Spain.","DOI":"10.1109\/ICCV.2011.6126251"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Nam, H., and Han, B. (2016, January 1\u201326). Learning Multi-Domain Convolutional Neural Networks for Visual Tracking. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.465"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Held, D., Thrun, S., and Savarese, S. (2016, January 8\u201316). Learning to Track at 100 FPS with Deep Regression Networks. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46448-0_45"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Ning, G., Zhang, Z., Huang, C., He, Z., Ren, X., and Wang, H. (2016, January 12\u201314). Spatially Supervised Recurrent Convolutional Neural Networks for Visual Object Tracking. Proceedings of the International Symposium on Visual Computing (ISVC), Las Vegas, NV, USA.","DOI":"10.1109\/ISCAS.2017.8050867"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Tao, R., Gavves, E., and Smeulders, A. (2016, January 1\u201326). Siamese Instance Search for Tracking. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.158"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Bertinetto, L., Valmadre, J., Henriques, J.F., Vedaldi, A., and Torr, P.H.S. (2016, January 8\u201316). Fully-Convolutional Siamese Networks for Object Tracking. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-48881-3_56"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"He, A., Luo, C., Tian, X., and Zeng, W. (2018, January 18\u201322). A Twofold Siamese Network for Real-Time Object Tracking. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00508"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Li, B., Yan, J., Wu, W., Zhu, Z., and Hu, X. (2018, January 18\u201322). High Performance Visual Tracking With Siamese Region Proposal Network. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00935"},{"key":"ref_19","unstructured":"Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7\u201312). Faster r-cnn: Towards real-time object detection with region proposal networks. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Montreal, QC, Canada."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 1\u201326). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_21","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20136). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Lake Tahoe, NV, USA."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Yan, B., Peng, H., Fu, J., Wang, D., and Lu, H. (2021, January 10\u201317). Learning Spatio-Temporal Transformer for Visual Tracking. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.01028"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Wang, N., Zhou, W., Wang, J., and Li, H. (2021, January 20\u201325). Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00162"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Chen, X., Yan, B., Zhu, J., Wang, D., Yang, X., and Lu, H. (2021, January 20\u201325). Transformer tracking. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00803"},{"key":"ref_25","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23\u201327). End-to-end object detection with transformers. Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel.","DOI":"10.1007\/978-3-030-58452-8_13"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 10\u201317). Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00986"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Zhang, B., Li, X., Han, J., and Zeng, X. (2018, January 19\u201321). MiniTracker: A Lightweight CNN-based System for Visual Object Tracking on Embedded Device. Proceedings of the IEEE 23rd International Conference on Digital Signal Processing (DSP), Shanghai, China.","DOI":"10.1109\/ICDSP.2018.8631813"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Przewlocka, D., Wasala, M., Szolc, H., Blachut, K., and Kryjak, T. (2020, January 14\u201316). Optimisation of a Siamese Neural Network for Real-Time Energy Efficient Object Tracking. Proceedings of the Computer Vision and Graphics, Warsaw, Poland.","DOI":"10.36227\/techrxiv.12594437"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Cao, Y., Ji, H., Zhang, W., and Shirani, S. (2019, January 2\u20135). Extremely Tiny Siamese Networks with Multi-level Fusions for Visual Object Tracking. Proceedings of the 22th International Conference on Information Fusion (FUSION), Ottawa, ON, Canada.","DOI":"10.23919\/FUSION43075.2019.9011338"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Cui, Z., and An, J. (2020, January 20\u201323). Heterogeneous Siamese Tracking System Based on PYNQ Framework. Proceedings of the 6th International Conference on Control, Automation and Robotics (ICCAR), Singapore.","DOI":"10.1109\/ICCAR49639.2020.9108096"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Przewlocka-Rus, D., and Kryjak, T. (2022, January 20\u201322). Towards Real-Time and Energy Efficient Siamese Tracking \u2013 A Hardware-Software Approach. Proceedings of the Design and Architecture for Signal and Image Processing, Budapest, Hungary.","DOI":"10.36227\/techrxiv.19808521.v1"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Umuroglu, Y., Fraser, N.J., Gambardella, G., Blott, M., Leong, P., Jahre, M., and Vissers, K. (2017, January 22\u201324). FINN: A Framework for Fast, Scalable Binarized Neural Network Inference. Proceedings of the ACM\/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA), Monterey, CA, USA.","DOI":"10.1145\/3020078.3021744"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Kathail, V. (2020, January 23\u201325). Xilinx vitis unified software platform. Proceedings of the ACM\/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA), Seaside, CA, USA.","DOI":"10.1145\/3373087.3375887"},{"key":"ref_35","unstructured":"Pappalardo, A. (Xilinx\/Brevitas, 2023). Xilinx\/Brevitas, version 0.10.2."},{"key":"ref_36","unstructured":"Xilinx (DPUCZDX8G for Zynq UltraScale + MPSoCs Product Guide (PG338), 2022). DPUCZDX8G for Zynq UltraScale + MPSoCs Product Guide (PG338), version 4.1."},{"key":"ref_37","unstructured":"Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Zajc, L.\u010c., Voj\u00ed\u0159, T., Bhat, G., Luke\u017ei\u010d, A., and Eldesokey, A. (2018, January 8\u201314). The Sixth Visual Object Tracking VOT2018 Challenge Results. Proceedings of the European Conference on Computer Vision (ECCV) Workshops, Munich, Germany."},{"key":"ref_38","unstructured":"Hubara, I., Nahshan, Y., Hanani, Y., Banner, R., and Soudry, D. (2020). Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv."},{"key":"ref_39","unstructured":"Xilinx (Vitis AI User Guide (UG1414), 2023). Vitis AI User Guide (UG1414), version 3.5."},{"key":"ref_40","unstructured":"Xilinx (PYNQ\u2014Python Productivity for Zynq, 2022). PYNQ\u2014Python Productivity for Zynq, version 3.0."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1038\/s41592-019-0686-2","article-title":"SciPy 1.0: Fundamental algorithms for scientific computing in Python","volume":"17","author":"Virtanen","year":"2020","journal-title":"Nat. Methods"},{"key":"ref_42","unstructured":"Xilinx (Vitis HLS, 2022). Vitis HLS, version 2022.2."}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/10\/3\/70\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:12:45Z","timestamp":1760105565000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/10\/3\/70"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,13]]},"references-count":42,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,3]]}},"alternative-id":["jimaging10030070"],"URL":"https:\/\/doi.org\/10.3390\/jimaging10030070","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,13]]}}}