{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T14:25:40Z","timestamp":1760711140092,"version":"build-2065373602"},"reference-count":54,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2023,3,7]],"date-time":"2023-03-07T00:00:00Z","timestamp":1678147200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002631","name":"Gachon University","doi-asserted-by":"publisher","award":["GCU-202206860001"],"award-info":[{"award-number":["GCU-202206860001"]}],"id":[{"id":"10.13039\/501100002631","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Video deblurring aims at removing the motion blur caused by the movement of objects or camera shake. Traditional video deblurring methods have mainly focused on frame-based deblurring, which takes only blurry frames as the input to produce sharp frames. However, frame-based deblurring has shown poor picture quality in challenging cases of video restoration where severely blurred frames are provided as the input. To overcome this issue, recent studies have begun to explore the event-based approach, which uses the event sequence captured by an event camera for motion deblurring. Event cameras have several advantages compared to conventional frame cameras. Among these advantages, event cameras have a low latency in imaging data acquisition (0.001 ms for event cameras vs. 10 ms for frame cameras). Hence, event data can be acquired at a high acquisition rate (up to one microsecond). This means that the event sequence contains more accurate motion information than video frames. Additionally, event data can be acquired with less motion blur. Due to these advantages, the use of event data is highly beneficial for achieving improvements in the quality of deblurred frames. Accordingly, the results of event-based video deblurring are superior to those of frame-based deblurring methods, even for severely blurred video frames. However, the direct use of event data can often generate visual artifacts in the final output frame (e.g., image noise and incorrect textures), because event data intrinsically contain insufficient textures and event noise. To tackle this issue in event-based deblurring, we propose a two-stage coarse-refinement network by adding a frame-based refinement stage that utilizes all the available frames with more abundant textures to further improve the picture quality of the first-stage coarse output. Specifically, a coarse intermediate frame is estimated by performing event-based video deblurring in the first-stage network. A residual hint attention (RHA) module is also proposed to extract useful attention information from the coarse output and all the available frames. This module connects the first and second stages and effectively guides the frame-based refinement of the coarse output. The final deblurred frame is then obtained by refining the coarse output using the residual hint attention and all the available frame information in the second-stage network. We validated the deblurring performance of the proposed network on the GoPro synthetic dataset (33 videos and 4702 frames) and the HQF real dataset (11 videos and 2212 frames). Compared to the state-of-the-art method (D2Net), we achieved a performance improvement of 1 dB in PSNR and 0.05 in SSIM on the GoPro dataset, and an improvement of 1.7 dB in PSNR and 0.03 in SSIM on the HQF dataset.<\/jats:p>","DOI":"10.3390\/s23062880","type":"journal-article","created":{"date-parts":[[2023,3,8]],"date-time":"2023-03-08T02:08:14Z","timestamp":1678241294000},"page":"2880","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Multi-Stage Network for Event-Based Video Deblurring with Residual Hint Attention"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0660-7498","authenticated-orcid":false,"given":"Jeongmin","family":"Kim","sequence":"first","affiliation":[{"name":"School of Computing, Gachon University, Seongnam 13120, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6173-0857","authenticated-orcid":false,"given":"Yong Ju","family":"Jung","sequence":"additional","affiliation":[{"name":"School of Computing, Gachon University, Seongnam 13120, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,3,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Su, S., Delbracio, M., Wang, J., Sapiro, G., Heidrich, W., and Wang, O. (2017, January 21\u201326). Deep video deblurring for hand-held cameras. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.33"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Kim, T.H., Lee, K.M., Scholkopf, B., and Hirsch, M. (2017, January 22\u201329). Online video deblurring via dynamic temporal blending network. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.435"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kim, T.H., Sajjadi, M.S., Hirsch, M., and Scholkopf, B. (2018, January 8\u201314). Spatio-temporal transformer network for video restoration. Proceedings of the European Conference on Computer Vision, Munich, Germany.","DOI":"10.1007\/978-3-030-01219-9_7"},{"key":"ref_4","unstructured":"Shen, Z., Wang, W., Lu, X., Shen, J., Ling, H., Xu, T., and Shao, L. (November, January 27). Human-aware motion deblurring. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zhou, S., Zhang, J., Pan, J., Xie, H., Zuo, W., and Ren, J. (2018, January 8\u201314). Spatio-temporal filter adaptive network for video deblurring. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Munich, Germany.","DOI":"10.1109\/ICCV.2019.00257"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Nah, S., Son, S., and Lee, K.M. (2019, January 15\u201320). Recurrent neural networks with intra-frame iterations for video deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00829"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"3025","DOI":"10.1109\/TCSVT.2020.3035722","article-title":"Recursive neural network for video deblurring","volume":"31","author":"Zhang","year":"2020","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Pan, J., Bai, H., and Tang, J. (2020, January 13\u201319). Cascaded deep video deblurring using temporal sharpness prior. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00311"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wu, J., Yu, X., Liu, D., Chandraker, M., and Wang, Z. (2020, January 1\u20135). DAVID: Dual-attentional video deblurring. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Snowmass Village, CO, USA.","DOI":"10.1109\/WACV45572.2020.9093529"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zhong, Z., Gao, Y., Zheng, Y., and Zheng, B. (2020, January 23\u201328). Efficient spatio-temporal recurrent neural network for video deblurring. Proceedings of the European Conference on Computer Vision, Glasgow, UK.","DOI":"10.1007\/978-3-030-58539-6_12"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ji, B., and Yao, A. (2022, January 19\u201320). Multi-scale memory-based video deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.","DOI":"10.1109\/CVPR52688.2022.00196"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Lin, S., Zhang, J., Pan, J., Jiang, Z., Zou, D., Wang, Y., Chen, J., and Ren, J. (2020, January 23\u201328). Learning event-driven video deblurring and interpolation. Proceedings of the European Conference on Computer Vision, Glasgow, UK.","DOI":"10.1007\/978-3-030-58598-3_41"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Shang, W., Ren, D., Zou, D., Ren, J.S., Luo, P., and Zuo, W. (2021, January 10\u201317). Bringing events into video deblurring with non-consecutively blurry frames. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.00449"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1109\/JSSC.2007.914337","article-title":"A 128 \u00d7 128 120 dB 15 \u03bcs latency asynchronous temporal contrast vision sensor","volume":"43","author":"Lichtsteiner","year":"2008","journal-title":"IEEE J.-Solid-State Circuits"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2333","DOI":"10.1109\/JSSC.2014.2342715","article-title":"A 240 \u00d7 180 130 db 3 \u03bcs latency global shutter spatiotemporal vision sensor","volume":"49","author":"Brandli","year":"2014","journal-title":"IEEE J.-Solid-State Circuits"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Wang, L., Kim, T.K., and Yoon, K.J. (2020, January 13\u201319). Eventsr: From asynchronous events to image reconstruction, restoration, and super-resolution via end-to-end adversarial learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00834"},{"key":"ref_17","unstructured":"Ahmed, S.H., Jang, H.W., Uddin, S.N., and Jung, Y.J. (2022, January 24\u201328). Deep event stereo leveraged by event-to-image translation. Proceedings of the AAAI Conference on Artificial Intelligence, Pomona, CA, USA."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Tulyakov, S., Gehrig, D., Georgoulis, S., Erbach, J., Gehrig, M., Li, Y., and Scaramuzza, D. (2021, January 20\u201325). Time lens: Event-based video frame interpolation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01589"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"7489","DOI":"10.1109\/TCSVT.2022.3189480","article-title":"Unsupervised deep event stereo for depth estimation","volume":"32","author":"Uddin","year":"2022","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Pan, L., Scheerlinck, C., Yu, X., Hartley, R., Liu, M., and Dai, Y. (2019, January 15\u201320). Bringing a blurry frame alive at high frame-rate with an event camera. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00698"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1964","DOI":"10.1109\/TPAMI.2019.2963386","article-title":"High speed and high dynamic range video with an event camera","volume":"43","author":"Rebecq","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Nah, S., Kim, T.H., and Lee, K.M. (2017, January 21\u201326). Deep multi-scale convolutional neural network for dynamic scene deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.35"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Stoffregen, T., Scheerlinck, C., Scaramuzza, D., Drummond, T., Barnes, N., Kleeman, L., and Mahony, R.E. (2020, January 23\u201328). Reducing the Sim-to-Real Gap for Event Cameras. Proceedings of the 2020 European Conference on Computer Vision (ECCV), Glasgow, UK.","DOI":"10.1007\/978-3-030-58583-9_32"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Tao, X., Gao, H., Shen, X., Wang, J., and Jia, J. (2018, January 18\u201323). Scale-recurrent network for deep image deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00853"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Li, L., Pan, J., Lai, W.S., Gao, C., Sang, N., and Yang, M.H. (2018, January 18\u201322). Learning a discriminative prior for blind image deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00692"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Aittala, M., and Durand, F. (2018, January 8\u201314). Burst image deblurring using permutation invariant convolutional neural networks. Proceedings of the European Conference on Computer Vision, Munich, Germany.","DOI":"10.1007\/978-3-030-01237-3_45"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Zhang, H., Dai, Y., Li, H., and Koniusz, P. (2019, January 15\u201320). Deep stacked hierarchical multi-patch network for image deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00613"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., and Matas, J. (2018, January 18\u201323). Deblurgan: Blind motion deblurring using conditional adversarial networks. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00854"},{"key":"ref_29","unstructured":"Kupyn, O., Martyniuk, T., Wu, J., and Wang, Z. (November, January 27). Deblurgan-v2: Deblurring 52(orders-of-magnitude) faster and better. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zhang, K., Luo, W., Zhong, Y., Ma, L., Stenger, B., Liu, W., and Li, H. (2020, January 13\u201319). Deblurring by realistic blurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00281"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Truong, N.Q., Lee, Y.W., Owais, M., Nguyen, D.T., Batchuluun, G., Pham, T.D., and Park, K.R. (2020). SlimDeblurGAN-based motion deblurring and marker detection for autonomous drone landing. Sensors, 20.","DOI":"10.3390\/s20143918"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Chen, L., Zhang, J., Pan, J., Lin, S., Fang, F., and Ren, J.S. (2021, January 20\u201325). Learning a non-blind deblurring network for night blurry images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01040"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Dong, J., Roth, S., and Schiele, B. (2021, January 20\u201325). Learning spatially-variant MAP models for non-blind image deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00485"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Chen, L., Zhang, J., Lin, S., Fang, F., and Ren, J.S. (2021, January 20\u201325). Blind deblurring for saturated images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00624"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Tran, P., Tran, A.T., Phung, Q., and Hoai, M. (2021, January 20\u201325). Explore image deblurring via encoded blur kernel space. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01178"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Suin, M., and Rajagopalan, A. (2021, January 20\u201325). Gated spatio-temporal attention-guided video deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00771"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Li, D., Xu, C., Zhang, K., Yu, X., Zhong, Y., Ren, W., Suominen, H., and Li, H. (2021, January 20\u201325). Arvo: Learning all-range volumetric correspondence for video deblurring. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00763"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Wang, J., Wang, Z., and Yang, A. (2022). Iterative dual CNNs for image deblurring. Mathematics, 10.","DOI":"10.3390\/math10203891"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Xu, F., Yu, L., Wang, B., Yang, W., Xia, G.S., Jia, X., Qiao, Z., and Liu, J. (2021, January 11\u201317). Motion deblurring with real events. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.00258"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Sun, L., Sakaridis, C., Liang, J., Jiang, Q., Yang, K., Sun, P., Ye, Y., Wang, K., and Gool, L.V. (2022, January 23\u201327). Event-based fusion for motion deblurring with cross-modal attention. Proceedings of the Computer Vision\u2013ECCV 2022: 17th European Conference, Tel Aviv, Israel.","DOI":"10.1007\/978-3-031-19797-0_24"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Li, J., Gong, W., and Li, W. (2018). Combining motion compensation with spatiotemporal constraint for video deblurring. Sensors, 18.","DOI":"10.3390\/s18061774"},{"key":"ref_42","first-page":"667","article-title":"Dynamic filter networks","volume":"29","author":"Jia","year":"2016","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Niklaus, S., Mai, L., and Liu, F. (2017, January 21\u201326). Video frame interpolation via adaptive convolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.244"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Mildenhall, B., Barron, J.T., Chen, J., Sharlet, D., Ng, R., and Carroll, R. (2018, January 18\u201322). Burst denoising with kernel prediction networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00265"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Wang, X., Yu, K., Dong, C., and Loy, C.C. (2018, January 18\u201322). Recovering realistic texture in image super-resolution by deep spatial feature transform. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00070"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Wang, L., Ho, Y.S., and Yoon, K.J. (2019, January 15\u201320). Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.01032"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Hu, J., Shen, L., and Sun, G. (2018, January 18\u201322). Squeeze-and-excitation networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00745"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 8\u201314). Cbam: Convolutional block attention module. Proceedings of the European Conference on Computer Vision, Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_1"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Uddin, S.N., and Jung, Y.J. (2020). Global and local attention-based free-form image inpainting. Sensors, 20.","DOI":"10.3390\/s20113204"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Yoon, H., Uddin, S.N., and Jung, Y.J. (2022). Multi-scale attention-guided non-local network for HDR image reconstruction. Sensors, 22.","DOI":"10.3390\/s22187044"},{"key":"ref_51","unstructured":"Rebecq, H., Gehrig, D., and Scaramuzza, D. (2018, January 29\u201331). ESIM: An open event camera simulator. Proceedings of the Conference on Robot Learning. PMLR, Z\u00fcrich, Switzerland."},{"key":"ref_52","first-page":"8024","article-title":"Pytorch: An imperative style, high-performance deep learning library","volume":"32","author":"Paszke","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_53","first-page":"802","article-title":"Convolutional LSTM network: A machine learning approach for precipitation nowcasting","volume":"28","author":"Shi","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Mehri, A., Ardakani, P.B., and Sappa, A.D. (2021, January 5\u20139). MPRNet: Multi-path residual network for lightweight image super resolution. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Virtual.","DOI":"10.1109\/WACV48630.2021.00275"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/6\/2880\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:49:53Z","timestamp":1760122193000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/6\/2880"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,7]]},"references-count":54,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2023,3]]}},"alternative-id":["s23062880"],"URL":"https:\/\/doi.org\/10.3390\/s23062880","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2023,3,7]]}}}