{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T16:31:36Z","timestamp":1776443496150,"version":"3.51.2"},"reference-count":58,"publisher":"MDPI AG","issue":"22","license":[{"start":{"date-parts":[[2020,11,20]],"date-time":"2020-11-20T00:00:00Z","timestamp":1605830400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61901330"],"award-info":[{"award-number":["61901330"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Convolutional neural networks have recently been used for multi-focus image fusion. However, some existing methods have resorted to adding Gaussian blur to focused images, to simulate defocus, thereby generating data (with ground-truth) for supervised learning. Moreover, they classify pixels as \u2018focused\u2019 or \u2018defocused\u2019, and use the classified results to construct the fusion weight maps. This then necessitates a series of post-processing steps. In this paper, we present an end-to-end learning approach for directly predicting the fully focused output image from multi-focus input image pairs. The suggested approach uses a CNN architecture trained to perform fusion, without the need for ground truth fused images. The CNN exploits the image structural similarity (SSIM) to calculate the loss, a metric that is widely accepted for fused image quality evaluation. What is more, we also use the standard deviation of a local window of the image to automatically estimate the importance of the source images in the final fused image when designing the loss function. Our network can accept images of variable sizes and hence, we are able to utilize real benchmark datasets, instead of simulated ones, to train our network. The model is a feed-forward, fully convolutional neural network that can process images of variable sizes during test time. Extensive evaluation on benchmark datasets show that our method outperforms, or is comparable with, existing state-of-the-art techniques on both objective and subjective benchmarks.<\/jats:p>","DOI":"10.3390\/s20226647","type":"journal-article","created":{"date-parts":[[2020,11,20]],"date-time":"2020-11-20T09:46:18Z","timestamp":1605865578000},"page":"6647","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["Structural Similarity Loss for Learning to Fuse Multi-Focus Images"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4184-1300","authenticated-orcid":false,"given":"Xiang","family":"Yan","sequence":"first","affiliation":[{"name":"School of Physics and Optoelectronic Engineering, Xidian University, Xi\u2019an 710071, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7448-2327","authenticated-orcid":false,"given":"Syed Zulqarnain","family":"Gilani","sequence":"additional","affiliation":[{"name":"School of Science, Edith Cowan University, Joondalup, WA 6027, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hanlin","family":"Qin","sequence":"additional","affiliation":[{"name":"School of Physics and Optoelectronic Engineering, Xidian University, Xi\u2019an 710071, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ajmal","family":"Mian","sequence":"additional","affiliation":[{"name":"Computer Science and Software Engineering, The University of Western Australia, Crawley, WA 6009, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,11,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1295","DOI":"10.1016\/j.patrec.2008.02.002","article-title":"Multifocus image fusion by combining curvelet and wavelet transform","volume":"29","author":"Li","year":"2008","journal-title":"Pattern Recognit. Lett."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1121","DOI":"10.1016\/j.dsp.2013.03.001","article-title":"Mutual spectral residual approach for multifocus image fusion","volume":"23","author":"Saha","year":"2013","journal-title":"Digit. Signal Prog."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1016\/j.inffus.2014.07.003","article-title":"Steerable local frequency based multispectral multifocus image fusion","volume":"23","author":"Gangapure","year":"2015","journal-title":"Inf. Fusion"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.sigpro.2013.09.001","article-title":"Discrete cosine transform based fusion of multi-focus images for visual sensor networks","volume":"95","author":"Phamila","year":"2014","journal-title":"Signal Process."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1016\/j.infrared.2014.07.019","article-title":"Adaptive fusion method of visible light and infrared images based on non-subsampled shearlet transform and fast non-negative matrix factorization","volume":"67","author":"Kong","year":"2014","journal-title":"Infrared Phys. Technol."},{"key":"ref_6","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Noh, H., Hong, S., and Han, B. (2015, January 7\u201313). Learning deconvolution network for semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.178"},{"key":"ref_10","unstructured":"Simonyan, K., and Zisserman, A. (2014, January 8\u201313). Two-stream convolutional networks for action recognition in videos. Proceedings of the 28th Annual Conference on Neural Information Processing Systems (NIPS), Montreal, QC, Canada."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., and Van Gool, L. (2016, January 11\u201314). Temporal segment networks: Towards good practices for deep action recognition. Proceedings of the European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46484-8_2"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., van der Smagt, P., Cremers, D., and Brox, T. (2015, January 7\u201313). Flownet: Learning optical flow with convolutional networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.316"},{"key":"ref_13","unstructured":"Lai, W., Huang, J., and Yang, M. (2017, January 4\u20139). Semi-supervised learning for optical flow with generative adversarial networks. Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1109\/TPAMI.2015.2439281","article-title":"Image super-resolution using deep convolutional networks","volume":"38","author":"Dong","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21\u201326). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.19"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Lai, W., Huang, J., Ahuja, N., and Yang, M. (2017, January 21\u201326). Deep laplacian pyramid networks for fast and accurate super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.618"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Prabhakar, K.R., Srikar, V.S., and Babu, R.V. (2017, January 22\u201329). Deepfuse: A deep unsupervised approach for exposure fusion with extreme exposure image pairs. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.505"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1016\/j.inffus.2016.12.001","article-title":"Multi-focus image fusion with a deep convolutional neural network","volume":"36","author":"Liu","year":"2017","journal-title":"Inf. Fusion"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1016\/j.ins.2017.12.043","article-title":"Pixel convolutional neural network for multi-focus image fusion","volume":"433\u2013434","author":"Tang","year":"2018","journal-title":"Inf. Sci."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1109\/TCI.2018.2889959","article-title":"Multi-level features convolutional neural network for multi-focus image fusion","volume":"5","author":"Yang","year":"2019","journal-title":"IEEE Trans. Comput. Imaging"},{"key":"ref_21","unstructured":"Gilani, S.Z., and Mian, A. (2018, January 18\u201322). Learning from millions of 3D scans for large-scale 3D face recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Taigman, Y., Yang, M., Ranzato, M., and Wolf, L. (2014, January 23\u201328). Deepface: Closing the gap to human-level performance in face verification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.220"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Schroff, F., Kalenichenko, D., and Philbin, J. (2015, January 7\u201312). Facenet: A unified embedding for face recognition and clustering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1016\/j.inffus.2016.05.004","article-title":"Pixel-level image fusion: A survey of the state of the art","volume":"33","author":"Li","year":"2017","journal-title":"Inf. Fusion"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1016\/j.inffus.2005.09.001","article-title":"Pixel-based and region-based image fusion schemes using ica bases","volume":"8","author":"Mitianoudis","year":"2007","journal-title":"Inf. Fusion"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1109\/TIP.2004.823821","article-title":"Gradient-based multiresolution image fusion","volume":"13","author":"Petrovic","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Hill, P.R., Canagarajah, C.N., and Bull, D.R. (2002, January 2\u20135). Image fusion using complex wavelets. Proceedings of the 13th British Machine Vision Conference (BMVC), University of Cardiff.","DOI":"10.5244\/C.16.47"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/j.inffus.2005.09.006","article-title":"Pixel-and region-based image fusion with complex wavelets","volume":"8","author":"Lewis","year":"2007","journal-title":"Inf. Fusion"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1334","DOI":"10.1016\/j.sigpro.2009.01.012","article-title":"Multifocus image fusion using the nonsubsampled contourlet transform","volume":"89","author":"Zhang","year":"2009","journal-title":"Signal Process."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1855","DOI":"10.1016\/j.patcog.2004.03.010","article-title":"A wavelet-based image fusion tutorial","volume":"37","author":"Pajares","year":"2004","journal-title":"Pattern Recognit."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1540","DOI":"10.1016\/j.optcom.2010.11.048","article-title":"A novel algorithm of image fusion using shearlets","volume":"284","author":"Miao","year":"2011","journal-title":"Opt. Commun."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"18846","DOI":"10.1364\/OE.20.018846","article-title":"Multifocus color image fusion based on quaternion curvelet transform","volume":"20","author":"Guo","year":"2012","journal-title":"Opt. Express"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1016\/j.inffus.2014.05.004","article-title":"Multi-focus image fusion with dense sift","volume":"23","author":"Liu","year":"2015","journal-title":"Inf. Fusion"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/j.inffus.2011.07.001","article-title":"Image matting for fusion of multi-focus images in dynamic scenes","volume":"14","author":"Li","year":"2013","journal-title":"Inf. Fusion"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"2864","DOI":"10.1109\/TIP.2013.2244222","article-title":"Image fusion with guided filtering","volume":"22","author":"Li","year":"2013","journal-title":"IEEE Trans. Image Process."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1526","DOI":"10.1109\/TIP.2017.2779274","article-title":"Robust multi-focus image fusion using edge model and multi-matting","volume":"27","author":"Chen","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.inffus.2018.01.009","article-title":"Multi-focus image fusion using content adaptive blurring","volume":"45","author":"Farid","year":"2019","journal-title":"Inf. Fusion"},{"key":"ref_38","unstructured":"Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto. Technical Report TR-2009."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Isola, P., Zhu, J., Zhou, T., and Efros, A. (2017, January 21\u201326). Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.632"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1016\/j.neucom.2018.05.103","article-title":"Pixel-wise regression using u-net and its application on pansharpening","volume":"312","author":"Yao","year":"2018","journal-title":"Neurocomputing"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"9755","DOI":"10.1109\/JSEN.2019.2928818","article-title":"Multi-focus image fusion using u-shaped networks with a hybrid objective","volume":"19","author":"Li","year":"2019","journal-title":"IEEE Sens. J."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Xu, H., Ma, J., Le, Z., Jiang, J., and Guo, X. (2020, January 7\u201312). FusionDN: A Unified Densely Connected Network for Image Fusion. Proceedings of the Thirty-Fourth AAAI Conference on Afitifical Intelligence (AAAI), New York, NY, USA.","DOI":"10.1609\/aaai.v34i07.6936"},{"key":"ref_43","unstructured":"Xu, H., Ma, J., Jiang, J., Guo, X., and Ling, J. (2020). U2fusion: A unified unsupervised image fusion network. IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1016\/j.inffus.2020.08.022","article-title":"MFF-GAN: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion","volume":"66","author":"Zhang","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"2599","DOI":"10.1109\/TPAMI.2018.2865304","article-title":"Fast and accurate image super-resolution with deep laplacian pyramid networks","volume":"41","author":"Lai","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_47","unstructured":"Maas, A.L., Hannun, A.Y., and Ng, A.Y. (2013, January 16\u201321). Rectifier nonlinearities improve neural network acoustic models. Proceedings of the 30th International Conference on Machine Learning (ICML), Atlanta, GA, USA."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.inffus.2014.10.004","article-title":"Multi-focus image fusion using dictionary-based sparse representation","volume":"25","author":"Nejati","year":"2015","journal-title":"Inf. Fusion"},{"key":"ref_49","unstructured":"(2018, January 01). Available online: http:\/\/www.imagefusion.org."},{"key":"ref_50","unstructured":"Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., and Isard, M. (2016, January 2\u20134). Tensorflow: A system for large-scale machine learning. Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI), Savannah, GA, USA."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1109\/TPAMI.2011.109","article-title":"Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study","volume":"34","author":"Liu","year":"2012","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1016\/j.inffus.2011.08.002","article-title":"A new image fusion performance metric based on visual information fidelity","volume":"14","author":"Han","year":"2013","journal-title":"Inf. Fusion"},{"key":"ref_53","first-page":"178","article-title":"A similarity metric for assessment of image fusio algorithms","volume":"2","author":"Cvejic","year":"2005","journal-title":"Int. J. Signal Process."},{"key":"ref_54","unstructured":"Wang, P., and Liu, B. (2008, January 26\u201329). A novel image fusion metric based on multi-scale analysis. Proceedings of the 9th International Conference on Signal Processing (ICSP), Beijing, China."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"626","DOI":"10.1049\/el:20060693","article-title":"Image fusion metric based on mutual information and Tsallis entropy","volume":"42","author":"Cvejic","year":"2006","journal-title":"Electron. Lett."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1016\/j.inffus.2005.10.001","article-title":"A human perception inspired quality metric for image fusion based on regional information","volume":"8","author":"Chen","year":"2007","journal-title":"Inf. Fusion"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1016\/j.inffus.2016.09.006","article-title":"Boundary finding based multi-focus image fusion through multi-scale morphological focus-measure","volume":"35","author":"Zhang","year":"2017","journal-title":"Inf. Fusion"},{"key":"ref_58","unstructured":"(2020, November 17). Available online: https:\/\/github.com\/yuliu316316\/MFIF."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/22\/6647\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:34:46Z","timestamp":1760178886000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/22\/6647"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,20]]},"references-count":58,"journal-issue":{"issue":"22","published-online":{"date-parts":[[2020,11]]}},"alternative-id":["s20226647"],"URL":"https:\/\/doi.org\/10.3390\/s20226647","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,20]]}}}