{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:33:50Z","timestamp":1760142830765,"version":"build-2065373602"},"reference-count":53,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2024,1,4]],"date-time":"2024-01-04T00:00:00Z","timestamp":1704326400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Convolutional neural networks (CNNs) have become instrumental in advancing multi-frame image super-resolution (SR), a technique that merges multiple low-resolution images of the same scene into a high-resolution image. In this paper, a novel deep learning multi-frame SR algorithm is introduced. The proposed CNN model, named Exponential Fusion of Interpolated Frames Network (EFIF-Net), seamlessly integrates fusion and restoration within an end-to-end network. Key features of the new EFIF-Net include a custom exponentially weighted fusion (EWF) layer for image fusion and a modification of the Residual Channel Attention Network for restoration to deblur the fused image. Input frames are registered with subpixel accuracy using an affine motion model to capture the camera platform motion. The frames are externally upsampled using single-image interpolation. The interpolated frames are then fused with the custom EWF layer, employing subpixel registration information to give more weight to pixels with less interpolation error. Realistic image acquisition conditions are simulated to generate training and testing datasets with corresponding ground truths. The observation model captures optical degradation from diffraction and detector integration from the sensor. The experimental results demonstrate the efficacy of EFIF-Net using both simulated and real camera data. The real camera results use authentic, unaltered camera data without artificial downsampling or degradation.<\/jats:p>","DOI":"10.3390\/s24010296","type":"journal-article","created":{"date-parts":[[2024,1,4]],"date-time":"2024-01-04T04:05:38Z","timestamp":1704341138000},"page":"296","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Exponential Fusion of Interpolated Frames Network (EFIF-Net): Advancing Multi-Frame Image Super-Resolution with Convolutional Neural Networks"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-0658-3549","authenticated-orcid":false,"given":"Hamed","family":"Elwarfalli","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Dayton, 300 College Park, Dayton, OH 45469, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5601-459X","authenticated-orcid":false,"given":"Dylan","family":"Flaute","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Dayton, 300 College Park, Dayton, OH 45469, USA"},{"name":"Applied Sensing Division, University of Dayton Research Institute, 300 College Park, Dayton, OH 45469, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1216-3865","authenticated-orcid":false,"given":"Russell C.","family":"Hardie","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Dayton, 300 College Park, Dayton, OH 45469, USA"}]}],"member":"1968","published-online":{"date-parts":[[2024,1,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1109\/MSP.2003.1203207","article-title":"Super-resolution image reconstruction: A technical overview","volume":"20","author":"Park","year":"2003","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e621","DOI":"10.7717\/peerj-cs.621","article-title":"A comprehensive review of deep learning-based single image super-resolution","volume":"7","author":"Bashir","year":"2021","journal-title":"PeerJ Comput. Sci."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Tan, R., Yuan, Y., Huang, R., and Luo, J. (2022, January 18\u201322). Video super-resolution with spatial-temporal transformer encoder. Proceedings of the 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, Taiwan.","DOI":"10.1109\/ICME52920.2022.9859774"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Li, H., and Zhang, P. (2021, January 18\u201322). Spatio-temporal fusion network for video super-resolution. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.","DOI":"10.1109\/IJCNN52387.2021.9533610"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Thawakar, O., Patil, P.W., Dudhane, A., Murala, S., and Kulkarni, U. (2019, January 18\u201321). Image and video super resolution using recurrent generative adversarial network. Proceedings of the 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Taipei, Taiwan.","DOI":"10.1109\/AVSS.2019.8909900"},{"key":"ref_6","first-page":"6479","article-title":"Multi-view silhouette and depth decomposition for high resolution 3d object representation","volume":"32","author":"Smith","year":"2018","journal-title":"Conf. Neural Inf. Process. Syst."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Li, B., Li, X., Lu, Y., Liu, S., Feng, R., and Chen, Z. (2022, January 23\u201327). Hst: Hierarchical swin transformer for compressed image super-resolution. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.","DOI":"10.1007\/978-3-031-25063-7_41"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1016\/j.neucom.2022.01.029","article-title":"Srdiff: Single image super-resolution with diffusion probabilistic models","volume":"479","author":"Li","year":"2022","journal-title":"Neurocomputing"},{"key":"ref_9","unstructured":"Ma, Z., Liao, R., Tao, X., Xu, L., Jia, J., and Wu, E. (2015, January 7\u201312). Handling motion blur in multi-frame super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Johnson, J., Alahi, A., and Fei-Fei, L. (2016, January 11\u201314). Perceptual losses for real-time style transfer and super-resolution. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46475-6_43"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ledig, C., Theis, L., Husz\u00e1r, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21\u201326). Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.19"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Lai, W.S., Huang, J.B., Ahuja, N., and Yang, M.H. (2017, January 21\u201326). Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.618"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Lim, B., Son, S., Kim, H., Nah, S., and Lee, K.M. (2017, January 21\u201326). Enhanced Deep Residual Networks for Single Image Super-Resolution. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA.","DOI":"10.1109\/CVPRW.2017.151"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Sajjadi, M.S.M., Sch\u00f6lkopf, B., and Hirsch, M. (2017, January 22\u201329). EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.481"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Bulat, A., and Tzimiropoulos, G. (2018, January 18\u201323). Super-FAN: Integrated Facial Landmark Localization and Super-Resolution of Real-World Low Resolution Faces in Arbitrary Poses with GANs. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00019"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Chu, X., Zhang, B., Ma, H., Xu, R., and Li, Q. (2021, January 10\u201315). Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.","DOI":"10.1109\/ICPR48806.2021.9413080"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Ahn, N., Kang, B., and Sohn, K.A. (2018, January 18\u201322). Image Super-Resolution via Progressive Cascading Residual Network. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPRW.2018.00123"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1109\/TPAMI.2015.2439281","article-title":"Image super-resolution using deep convolutional networks","volume":"38","author":"Dong","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Dong, C., Loy, C.C., and Tang, X. (2016, January 11\u201314). Accelerating the super-resolution convolutional neural network. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46475-6_25"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Kim, J., Lee, J.K., and Lee, K.M. (2016, January 27\u201330). Accurate Image Super-Resolution Using Very Deep Convolutional Networks. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.182"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Kim, J., Lee, J.K., and Lee, K.M. (2016, January 27\u201330). Deeply-recursive convolutional network for image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.181"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Tai, Y., Yang, J., and Liu, X. (2017, January 21\u201326). Image super-resolution via deep recursive residual network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.298"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Tai, Y., Yang, J., Liu, X., and Xu, C. (2017, January 22\u201329). Memnet: A persistent memory network for image restoration. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.486"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_25","unstructured":"Yu, J., Fan, Y., Yang, J., Xu, N., Wang, Z., Wang, X., and Huang, T. (2018). Wide activation for efficient and accurate image super-resolution. arXiv."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., and Fu, Y. (2018, January 8\u201314). Image super-resolution using very deep residual channel attention networks. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_18"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"31","DOI":"10.3389\/fphy.2015.00031","article-title":"Impact of detector-element active-area shape and fill factor on super-resolution","volume":"3","author":"Hardie","year":"2015","journal-title":"Front. Phys."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Milanfar, P. (2017). Super-Resolution Imaging, CRC Press.","DOI":"10.1201\/9781439819319"},{"key":"ref_29","unstructured":"Deudon, M., Kalaitzis, A., Goytom, I., Arefin, M.R., Lin, Z., Sankaran, K., Michalski, V., Kahou, S.E., Cornebise, J., and Bengio, Y. (2020). Highres-net: Recursive fusion for multi-frame super-resolution of satellite imagery. arXiv."},{"key":"ref_30","unstructured":"Arefin, M.R., Michalski, V., St-Charles, P.L., Kalaitzis, A., Kim, S., Kahou, S.E., and Bengio, Y. (2020, January 14\u201319). Multi-image super-resolution for remote sensing using deep recurrent networks. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"3644","DOI":"10.1109\/TGRS.2019.2959248","article-title":"Deepsum: Deep neural network for super-resolution of unregistered multitemporal images","volume":"58","author":"Molini","year":"2019","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_32","unstructured":"Bajo, M. (2020). Multi-Frame Super Resolution of Unregistered Temporal Images Using WDSR Nets, Available online: https:\/\/zenodo.org\/records\/3733116."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Dorr, F. (2020). Satellite image multi-frame super resolution using 3D wide-activation neural networks. Remote Sens., 12.","DOI":"10.20944\/preprints202009.0678.v1"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Salvetti, F., Mazzia, V., Khaliq, A., and Chiaberge, M. (2020). Multi-image super resolution of remotely sensed images using residual attention deep neural networks. Remote Sens., 12.","DOI":"10.3390\/rs12142207"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Bhat, G., Danelljan, M., Van Gool, L., and Timofte, R. (2021, January 20\u201325). Deep burst super-resolution. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00909"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Tian, Y., Zhang, Y., Fu, Y., and Xu, C. (2020, January 13\u201319). Tdan: Temporally-deformable alignment network for video super-resolution. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00342"},{"key":"ref_37","unstructured":"Ustinova, E., and Lempitsky, V. (2017). Deep multi-frame face super-resolution. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Wang, X., Chan, K.C., Yu, K., Dong, C., and Loy, C.C. (2019, January 16\u201317). EDVR: Video Restoration With Enhanced Deformable Convolutional Networks. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00247"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zhu, X., Hu, H., Lin, S., and Dai, J. (2019, January 15\u201320). Deformable convnets v2: More deformable, better results. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00953"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Cao, F., and Su, M. (2022, January 6\u20138). Research on Face Recognition Algorithm Based on CNN and Image Super-resolution Reconstruction. Proceedings of the 2022 IEEE 8th Intl Conference on Big Data Security on Cloud (BigDataSecurity), IEEE Intl Conference on High Performance and Smart Computing,(HPSC) and IEEE Intl Conference on Intelligent Data and Security (IDS), Jinan, China.","DOI":"10.1109\/BigDataSecurityHPSCIDS54978.2022.00038"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1373","DOI":"10.1109\/JSTARS.2022.3143532","article-title":"TR-MISR: Multiimage super-resolution based on feature fusion with transformers","volume":"15","author":"An","year":"2022","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Gonbadani, M.M.A., and Abbasfar, A. (2020, January 4\u20136). Combined Single and Multi-frame Image Super-resolution. Proceedings of the 2020 28th Iranian Conference on Electrical Engineering (ICEE), Tabriz, Iran.","DOI":"10.1109\/ICEE50131.2020.9260802"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"103097","DOI":"10.1016\/j.cviu.2020.103097","article-title":"Fifnet: A convolutional neural network for motion-based multiframe super-resolution using fusion of interpolated frames","volume":"202","author":"Elwarfalli","year":"2021","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2953","DOI":"10.1109\/TIP.2007.909416","article-title":"A fast image super-resolution algorithm using an adaptive Wiener filter","volume":"16","author":"Hardie","year":"2007","journal-title":"IEEE Trans. Image Process."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"26208","DOI":"10.1364\/OE.19.026208","article-title":"Fast super-resolution with affine motion using an adaptive Wiener filter and its application to airborne imaging","volume":"19","author":"Hardie","year":"2011","journal-title":"Opt. Express"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"083103","DOI":"10.1117\/1.OE.58.8.083103","article-title":"Fusion of interpolated frames superresolution in the presence of atmospheric optical turbulence","volume":"58","author":"Hardie","year":"2019","journal-title":"Opt. Eng."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Zhang, Y. (2023, November 01). RCAN: PyTorch Code for Our ECCV 2018 Paper \u201cImage Super-Resolution Using Very Deep Residual Channel Attention Networks\u201d. Available online: https:\/\/github.com\/yulunzhang\/RCAN\/tree\/master.","DOI":"10.1007\/978-3-030-01234-2_18"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Agustsson, E., and Timofte, R. (2017, January 21\u201326). NTIRE 2017 challenge on single image super-resolution: Dataset and study. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.","DOI":"10.1109\/CVPRW.2017.150"},{"key":"ref_49","unstructured":"Kingma, D.P., and Ba, J. (2017). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"898","DOI":"10.1109\/TPAMI.2010.161","article-title":"Contour detection and hierarchical image segmentation","volume":"33","author":"Arbelaez","year":"2010","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"21053","DOI":"10.1364\/OE.20.021053","article-title":"Fast super-resolution using an adaptive Wiener filter with robustness to local motion","volume":"20","author":"Hardie","year":"2012","journal-title":"Opt. Express"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"28","DOI":"10.3389\/fphy.2015.00028","article-title":"Robust super-resolution by fusion of interpolated frames for color and grayscale images","volume":"3","author":"Karch","year":"2015","journal-title":"Front. Phys."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/1\/296\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T13:39:33Z","timestamp":1760103573000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/1\/296"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,4]]},"references-count":53,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,1]]}},"alternative-id":["s24010296"],"URL":"https:\/\/doi.org\/10.3390\/s24010296","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2024,1,4]]}}}