{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,14]],"date-time":"2026-06-14T07:41:11Z","timestamp":1781422871864,"version":"3.54.1"},"reference-count":33,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2019,10,7]],"date-time":"2019-10-07T00:00:00Z","timestamp":1570406400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>Recently, video frame interpolation research developed with a convolutional neural network has shown remarkable results. However, these methods demand huge amounts of memory and run time for high-resolution videos, and are unable to process a 4K frame in a single pass. In this paper, we propose a fast 4K video frame interpolation method, based upon a multi-scale optical flow reconstruction scheme. The proposed method predicts low resolution bi-directional optical flow, and reconstructs it into high resolution. We also proposed consistency and multi-scale smoothness loss to enhance the quality of the predicted optical flow. Furthermore, we use adversarial loss to make the interpolated frame more seamless and natural. We demonstrated that the proposed method outperforms the existing state-of-the-art methods in quantitative evaluation, while it runs up to 4.39\u00d7 faster than those methods for 4K videos.<\/jats:p>","DOI":"10.3390\/sym11101251","type":"journal-article","created":{"date-parts":[[2019,10,8]],"date-time":"2019-10-08T09:00:38Z","timestamp":1570525238000},"page":"1251","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["A Fast 4K Video Frame Interpolation Using a Multi-Scale Optical Flow Reconstruction Network"],"prefix":"10.3390","volume":"11","author":[{"given":"Ha-Eun","family":"Ahn","sequence":"first","affiliation":[{"name":"Department of Electronic Engineering, Kwangnwoon University, Seoul 01897, Korea"},{"name":"Korea Electronics Technology Institute, Sungnam 13509, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinwoo","family":"Jeong","sequence":"additional","affiliation":[{"name":"Korea Electronics Technology Institute, Sungnam 13509, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Je Woo","family":"Kim","sequence":"additional","affiliation":[{"name":"Korea Electronics Technology Institute, Sungnam 13509, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Soonchul","family":"Kwon","sequence":"additional","affiliation":[{"name":"Department of Electronic Engineering, Kwangnwoon University, Seoul 01897, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jisang","family":"Yoo","sequence":"additional","affiliation":[{"name":"Department of Electronic Engineering, Kwangnwoon University, Seoul 01897, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2019,10,7]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Werlberger, M., Pock, T., Unger, M., and Bischof, H. (2011). Optical flow guided TV-L 1 video interpolation and restoration. International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, Springer.","DOI":"10.1007\/978-3-642-23094-3_20"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1235","DOI":"10.1109\/TCSVT.2013.2242631","article-title":"Multi-level video frame interpolation: Exploiting the interaction among different levels","volume":"23","author":"Yu","year":"2013","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. European Conference on Computer Vision, Springer.","DOI":"10.1007\/978-3-540-24673-2_3"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Liu, Z., Yeh, R.A., Tang, X., Liu, Y., and Agarwala, A. (2017, January 22\u201329). Video frame synthesis using deep voxel flow. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.478"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Jiang, H., Sun, D., Jampani, V., Yang, M.H., Learned-Miller, E., and Kautz, J. (2018, January 18\u201322). Super slomo: High quality estimation of multiple intermediate frames for video interpolation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00938"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Niklaus, S., Mai, L., and Liu, F. (2017, January 21\u201326). Video frame interpolation via adaptive convolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.244"},{"key":"ref_7","unstructured":"Liu, Y.L., Liao, Y.T., Lin, Y.Y., and Chuang, Y.Y. (February, January 27). Deep Video Frame Interpolation using Cyclic Frame Generation. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_8","unstructured":"Xue, T., Chen, B., Wu, J., Wei, D., and Freeman, W.T. (2017). Video enhancement with task-oriented flow. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Bao, W., Lai, W.-S., Ma, C., Zhang, X., Gao, Z., and Yang, M.-H. (2019, January 16\u221220). Depth-aware video frame interpolation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00382"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Niklaus, S., and Liu, F. (2018, January 18\u201322). Context-aware synthesis for video frame interpolation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00183"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Fourure, D., Emonet, R., Fromont, E., Muselet, D., Tr\u2019emeau, A., and Wolf, C. (2017, January 4\u20137). Residual conv-deconv grid network for semantic segmentation. Proceedings of the British Machine Vision Conference, London, UK.","DOI":"10.5244\/C.31.181"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Meyer, S., Djelouah, A., McWilliams, B., SorkineHornung, A., Gross, M., and Schroers, C. (2018, January 18\u201323). PhaseNet for video frame interpolation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00059"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Niklaus, S., Mai, L., and Liu, F. (2017, January 22\u201329). Video frame interpolation via adaptive separable convolution. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.37"},{"key":"ref_14","unstructured":"van Amersfoort, J., Shi, W., Acosta, A., Massa, F., Totz, J., Wang, Z., and Caballero, J. (2017). Frame interpolation with multi-scale deep loss functions and generative adversarial networks. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Peleg, T., Szekely, P., Sabo, D., and Sendik, O. (2019, January 16\u201320). IM-Net for High Resolution Video Frame Interpolation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00250"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014). Caffe: Convolutional architecture for fast feature embedding. arXiv.","DOI":"10.1145\/2647868.2654889"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Ahn, H.-E., Jeong, J., and Kim, J.W. (2019). A Fast 4K Video Frame Interpolation using a Hybrid Task-Based Convolutional Neural Network. Symmetry, 11.","DOI":"10.3390\/sym11050619"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional Networks for Biomedical Image Segmentation, Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_19","unstructured":"Xu, B., Wang, N., Chen, T., and Li, M. (2015). Empirical evaluation of rectified activations in convolutional network. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11\u201314 October 2016.","DOI":"10.1007\/978-3-319-46475-6_43"},{"key":"ref_21","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). Imagenet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_23","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv."},{"key":"ref_24","unstructured":"Mathieu, M., Couprie, C., and LeCun, Y. (2015). Deep multi-scale video prediction beyond mean square error. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s11263-010-0390-2","article-title":"A database and evaluation methodology for optical flow","volume":"92","author":"Baker","year":"2011","journal-title":"Int. J. Comput. Vis."},{"key":"ref_26","unstructured":"Soomro, K., Amir, R.Z., and Mubarak, S. (2012). UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv."},{"key":"ref_27","unstructured":"(2019, May 01). Vimeo. Available online: https:\/\/vimeo.com."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"LeFeuvre, J., Thiesse, J.M., Parmentier, M., Raulet, M., and Daguet, C. Ultra high definition HEVC DASH data set. Proceedings of the 5th ACM Multimedia Systems Conference, Singapore, 19 March 2014.","DOI":"10.1145\/2557642.2563672"},{"key":"ref_29","unstructured":"Song, L., Tang, X., Zhang, W., Yang, X., and Xia, P. (2013, January 3). The SJTU 4K video sequence dataset. Proceedings of the IEEE 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX), Klagenfurt am W\u00f6rthersee, Austria."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Yamanaka, J., Kuwashima, S., and Kurita, T. (2017, January 14\u201318). Fast and accurate image super resolution by deep CNN with skip connection and network in network. Proceedings of the International Conference of Neural Information Processing, Guangzhou, China.","DOI":"10.1007\/978-3-319-70096-0_23"},{"key":"ref_32","unstructured":"Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017). Automatic differentiation in pytorch. NIPS-W, Available online: https:\/\/openreview.net\/forum?id=BJJsrmfCZ."},{"key":"ref_33","unstructured":"Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., and Shelhamer, E. (2014). cuDNN: Efficient primitives for deep learning. arXiv."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/11\/10\/1251\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:28:07Z","timestamp":1760189287000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/11\/10\/1251"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,7]]},"references-count":33,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2019,10]]}},"alternative-id":["sym11101251"],"URL":"https:\/\/doi.org\/10.3390\/sym11101251","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,7]]}}}