{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T18:09:52Z","timestamp":1774030192574,"version":"3.50.1"},"reference-count":48,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2022,11,4]],"date-time":"2022-11-04T00:00:00Z","timestamp":1667520000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"the National Key Research and Development Program of China","award":["2021YFC3340500"],"award-info":[{"award-number":["2021YFC3340500"]}]},{"name":"the National Key Research and Development Program of China","award":["2022CDJKYJH046"],"award-info":[{"award-number":["2022CDJKYJH046"]}]},{"name":"The Fundamental Research Funds for the Central Universities","award":["2021YFC3340500"],"award-info":[{"award-number":["2021YFC3340500"]}]},{"name":"The Fundamental Research Funds for the Central Universities","award":["2022CDJKYJH046"],"award-info":[{"award-number":["2022CDJKYJH046"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Taking advantage of the functional complementarity between infrared and visible light sensors imaging, pixel-level real-time image fusion based on infrared and visible light images of different resolutions is a promising strategy for visual enhancement, which has demonstrated tremendous potential for autonomous driving, military reconnaissance, video surveillance, etc. Great progress has been made in this field in recent years, but the fusion speed and quality of visual enhancement are still not satisfactory. Herein, we propose a multi-scale FPGA-based image fusion technology with substantially enhanced visual enhancement capability and fusion speed. Specifically, the source images are first decomposed into three distinct layers using guided filter and saliency detection, which are the detail layer, saliency layer and background layer. Fusion weight map of the saliency layer is subsequently constructed using attention mechanism. Afterwards weight fusion strategy is used for saliency layer fusion and detail layer fusion, while weight average fusion strategy is used for the background layer fusion, followed by the incorporation of image enhancement technology to improve the fused image contrast. Finally, high-level synthesis tool is used to design the hardware circuit. The method in the present study is thoroughly tested on XCZU15EG board, which could not only effectively improve the image enhancement capability in glare and smoke environments, but also achieve fast real-time image fusion with 55FPS for infrared and visible images with a resolution of 640 \u00d7 470.<\/jats:p>","DOI":"10.3390\/s22218487","type":"journal-article","created":{"date-parts":[[2022,11,4]],"date-time":"2022-11-04T09:26:00Z","timestamp":1667553960000},"page":"8487","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["A Real-Time FPGA Implementation of Infrared and Visible Image Fusion Using Guided Filter and Saliency Detection"],"prefix":"10.3390","volume":"22","author":[{"given":"Ling","family":"Zhang","sequence":"first","affiliation":[{"name":"The School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2321-7662","authenticated-orcid":false,"given":"Xuefei","family":"Yang","sequence":"additional","affiliation":[{"name":"The School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China"}]},{"given":"Zhenlong","family":"Wan","sequence":"additional","affiliation":[{"name":"National Information Center of GACC, Beijing 100005, China"}]},{"given":"Dingxin","family":"Cao","sequence":"additional","affiliation":[{"name":"The School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7478-3103","authenticated-orcid":false,"given":"Yingcheng","family":"Lin","sequence":"additional","affiliation":[{"name":"The School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1080\/19479830903561035","article-title":"Multi-source remote sensing data fusion: Status and trends","volume":"1","author":"Zhang","year":"2010","journal-title":"Int. J. Image Data Fusion"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1016\/j.inffus.2018.02.004","article-title":"Infrared and visible image fusion methods and applications: A survey","volume":"45","author":"Ma","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_3","unstructured":"Mertens, T., Kautz, J., and Reeth, F.V. (November, January 29). Exposure fusion. Proceedings of the 15th Pacific Conference on Computer Graphics and Applications (PG\u201907), Maui, HI, USA."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.biosystemseng.2009.02.009","article-title":"Image fusion of visible and thermal images for fruit detection","volume":"103","author":"Bulanon","year":"2009","journal-title":"Biosyst. Eng."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"151","DOI":"10.1016\/j.infrared.2015.01.022","article-title":"Fusion of visible and infrared images using multiobjective evolutionary algorithm based on decomposition","volume":"71","author":"Jin","year":"2015","journal-title":"Infrared Phys. Technol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/j.inffus.2005.09.006","article-title":"Pixel-and region-based image fusion with complex wavelets","volume":"8","author":"Lewis","year":"2007","journal-title":"Inf. Fusion"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"275138","DOI":"10.1155\/2012\/275138","article-title":"Airborne infrared and visible image fusion for target perception based on target region segmentation and discrete wavelet transform","volume":"2012","author":"Niu","year":"2012","journal-title":"Math. Probl. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1041","DOI":"10.1016\/j.asoc.2011.11.020","article-title":"Infrared and visible image fusion using fuzzy logic and population-based optimization","volume":"12","author":"Saeedi","year":"2012","journal-title":"Appl. Soft Comput."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1016\/j.inffus.2006.02.001","article-title":"Remote sensing image fusion using the curvelet transform","volume":"8","author":"Nencini","year":"2007","journal-title":"Inf. Fusion"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1016\/j.inffus.2009.05.001","article-title":"Image fusion based on a new contourlet packet","volume":"11","author":"Yang","year":"2010","journal-title":"Inf. Fusion"},{"key":"ref_11","first-page":"299","article-title":"An adaptive bilateral filter based framework for image denoising","volume":"40","author":"Zhang","year":"2014","journal-title":"Neurocomputing"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"617","DOI":"10.1109\/LSP.2007.894966","article-title":"On the convergence of bilateral filter for edge-preserving image smoothing","volume":"14","author":"Dong","year":"2007","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1397","DOI":"10.1109\/TPAMI.2012.213","article-title":"Guided image filtering","volume":"35","author":"He","year":"2012","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Shen, X., Xu, L., and Jia, J. (2014, January 6\u201312). Rolling guidance filter. Proceedings of the Computer Vision-ECCV 2014, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10578-9_53"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2864","DOI":"10.1109\/TIP.2013.2244222","article-title":"Image fusion with guided filtering","volume":"22","author":"Li","year":"2013","journal-title":"IEEE Trans. Image Process."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"5576","DOI":"10.1007\/s00034-019-01131-z","article-title":"Multi-scale guided image and video fusion: A fast and efficient approach","volume":"38","author":"Bavirisetti","year":"2019","journal-title":"Circuits Syst. Signal Process."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1007\/s11045-015-0343-6","article-title":"Image fusion based on complex-shearlet domain with guided filtering","volume":"28","author":"Liu","year":"2017","journal-title":"Multidim Syst Sign Process"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1193","DOI":"10.1007\/s11760-013-0556-9","article-title":"Image fusion based on pixel significance using cross bilateral filter","volume":"9","year":"2015","journal-title":"Signal Image Video Process."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1016\/j.future.2018.01.039","article-title":"Multi-scale image fusion through rolling guidance filter","volume":"83","author":"Jian","year":"2018","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/j.inffus.2015.01.001","article-title":"Robust multi-modal medical image fusion via anisotropic heat diffusion guided low-rank structural analysis","volume":"26","author":"Wang","year":"2015","journal-title":"Inf. Fusion"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1016\/j.inffus.2008.08.006","article-title":"Multifocus image fusion using the log-Gabor transform and a multisize windows technique","volume":"10","author":"Redondo","year":"2009","journal-title":"Inf. Fusion"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.infrared.2016.01.009","article-title":"Two-scale image fusion of visible and infrared images using saliency detection","volume":"76","author":"Bavirisetti","year":"2016","journal-title":"Infrared Phys. Technol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"165775","DOI":"10.1016\/j.ijleo.2020.165775","article-title":"Infrared and visible image fusion using multi-scale edge-preserving decomposition and multiple saliency features","volume":"228","author":"Duan","year":"2021","journal-title":"Optik"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"169218","DOI":"10.1016\/j.ijleo.2022.169218","article-title":"Adaptive infrared and visible image fusion method by using rolling guidance filter and saliency detection","volume":"262","author":"Lin","year":"2022","journal-title":"Optik"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"108942","DOI":"10.1109\/ACCESS.2021.3101639","article-title":"An Infrared and Visible Image Fusion Method Guided by Saliency and Gradient Information","volume":"9","author":"Li","year":"2021","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.neucom.2015.10.080","article-title":"Image fusion with saliency map and interest points","volume":"177","author":"Meng","year":"2016","journal-title":"Neurocomputing"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Li, H., Wu, X.J., and Kittler, J. (2018, January 20\u201324). Infrared and Visible Image Fusion using a Deep Learning Framework. Proceedings of the 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China.","DOI":"10.1109\/ICPR.2018.8546006"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1850018","DOI":"10.1142\/S0219691318500182","article-title":"Infrared and visible image fusion with convolutional neural networks","volume":"16","author":"Liu","year":"2018","journal-title":"Int. J. Wavelets Multiresolut. Inf. Process."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"103039","DOI":"10.1016\/j.infrared.2019.103039","article-title":"Infrared and visible image fusion with ResNet and zero-phase component analysis","volume":"102","author":"Li","year":"2019","journal-title":"Infrared Phys. Technol."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"2614","DOI":"10.1109\/TIP.2018.2887342","article-title":"DenseFuse: A Fusion Approach to Infrared and Visible Images","volume":"28","author":"Li","year":"2019","journal-title":"IEEE Trans. Image Process."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"116533","DOI":"10.1016\/j.image.2021.116533","article-title":"A multi-focus color image fusion algorithm based on low vision image reconstruction and focused feature extraction","volume":"100","author":"Liu","year":"2022","journal-title":"Signal Process. Image Commun."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wang, K., Qi, G., Zhu, Z., and Chai, Y. (2017). A novel geometric dictionary construction approach for sparse representation based image fusion. Entropy, 19.","DOI":"10.3390\/e19070306"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TIM.2020.2986875","article-title":"Infrared and visible image fusion using visual saliency sparse representation and detail injection model","volume":"70","author":"Yang","year":"2020","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1016\/j.infrared.2017.02.005","article-title":"Infrared and visible image fusion based on visual saliency map and weighted least square optimization","volume":"82","author":"Ma","year":"2017","journal-title":"Infrared Phys. Technol."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"106354","DOI":"10.1016\/j.optlaseng.2020.106354","article-title":"A multi-modal image fusion framework based on guided filter and sparse representation","volume":"137","author":"Zhang","year":"2021","journal-title":"Opt. Lasers Eng."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1016\/j.infrared.2019.05.017","article-title":"Infrared & visible images fusion based on redundant directional lifting-based wavelet and saliency detection","volume":"101","author":"Song","year":"2019","journal-title":"Infrared Phys. Technol."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"103662","DOI":"10.1016\/j.infrared.2021.103662","article-title":"Infrared and visible image fusion based on weighted variance guided filter and image contrast enhancement","volume":"114","author":"Ren","year":"2021","journal-title":"Infrared Phys. Technol."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Sims, O., and Irvine, J. (2006, January 28\u201330). An FPGA implementation of pattern-selective pyramidal image fusion. Proceedings of the 2006 International Conference on Field Programmable Logic and Applications, Madrid, Spain.","DOI":"10.1109\/FPL.2006.311296"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Song, Y., Gao, K., Ni, G., and Lu, R. (2007, January 11\u201315). Implementation of real-time Laplacian pyramid image fusion processing based on FPGA. Proceedings of the Electronic Imaging and Multimedia Technology V, Beijing, China.","DOI":"10.1117\/12.756574"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.jesit.2014.03.006","article-title":"Discrete wavelet transform based image fusion and de-noising in FPGA","volume":"1","author":"Suraj","year":"2014","journal-title":"J. Electr. Syst. Inf. Technol."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Pemmaraju, M., Mashetty, S.C., Aruva, S., Saduvelly, M., and Edara, B.B. (2017, January 11\u201312). Implementation of image fusion based on wavelet domain using FPGA. Proceedings of the 2017 International Conference on Trends in Electronics and Informatics (ICEI), Tirunelveli, India.","DOI":"10.1109\/ICOEI.2017.8300978"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Aydin, F., Ugurdag, H.F., Levent, V.E., Guzel, A.E., Annafianto, N.F., Ozkan, M.A., Akgun, T., and Erbas, C. (November, January 28). Rapid design of real-time image fusion on FPGA using HLS and other techniques. Proceedings of the 2018 IEEE\/ACS 15th International Conference on Computer Systems and Applications (AICCSA), Aqaba, Jordan.","DOI":"10.1109\/AICCSA.2018.8612836"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"4639","DOI":"10.1109\/JSEN.2017.2712777","article-title":"Modified Frei-Chen operator-based infrared and visible sensor image fusion for real-time applications","volume":"17","author":"Mishra","year":"2017","journal-title":"IEEE Sens. J."},{"key":"ref_44","unstructured":"Alexander, T. (2022, June 23). TNO Image Fusion Dataset. Available online: https:\/\/figshare.com\/articles\/dataset\/TNO_Image_Fusion_Dataset\/1008029."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Gonz\u00e1lez, A., Fang, Z., Socarras, Y., Serrat, J., V\u00e1zquez, D., Xu, J., and L\u00f3pez, A.M. (2016). Pedestrian detection at day\/night time with visible and FIR cameras: A comparison. Sensors, 16.","DOI":"10.3390\/s16060820"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"4733","DOI":"10.1109\/TIP.2020.2975984","article-title":"MDLatLRR: A novel decomposition method for infrared and visible image fusion","volume":"29","author":"Li","year":"2020","journal-title":"IEEE Trans. Image Process."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"106182","DOI":"10.1016\/j.knosys.2020.106182","article-title":"Fast infrared and visible image fusion with structural decomposition","volume":"204","author":"Li","year":"2020","journal-title":"Knowl.-Based Syst."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Zhang, X., Ye, P., and Xiao, G. (2020, January 14\u201319). VIFB: A visible and infrared image fusion benchmark. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA.","DOI":"10.1109\/CVPRW50498.2020.00060"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/21\/8487\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:10:33Z","timestamp":1760145033000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/21\/8487"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,4]]},"references-count":48,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2022,11]]}},"alternative-id":["s22218487"],"URL":"https:\/\/doi.org\/10.3390\/s22218487","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,4]]}}}