{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:08:21Z","timestamp":1750219701990,"version":"3.41.0"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,1,11]],"date-time":"2024-01-11T00:00:00Z","timestamp":1704931200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2024,4,30]]},"abstract":"<jats:p>\n            Video frame synthesis, which consists of\n            <jats:italic>interpolation<\/jats:italic>\n            and\n            <jats:italic>extrapolation<\/jats:italic>\n            , is an essential video processing technique that can be applied to various scenarios. However, most existing methods cannot handle small objects or large motion well, especially in high-resolution videos such as 4K videos. To eliminate such limitations, we introduce a neighbor correspondence matching (NCM) algorithm for flow-based frame synthesis. Since the current frame is not available in video frame synthesis, NCM is performed in a current-frame-agnostic fashion to establish multi-scale correspondences in the spatial-temporal neighborhoods of each pixel. Based on the powerful motion representation capability of NCM, we propose a heterogeneous coarse-to-fine scheme for intermediate flow estimation. The coarse-scale and fine-scale modules are trained progressively, making NCM computationally efficient and robust to large motions. We further explore the mechanism of NCM and find that neighbor correspondence is powerful, since it provides multiple-hypotheses motion information for synthesis. Based on this analysis, we introduce a multiple-hypotheses estimation process for video frame extrapolation, resulting in a more robust framework, NCM-MH. Experimental results show that NCM and NCM-MH achieve 31.63 and 28.08 dB for interpolation and extrapolation on the most challenging X4K1000FPS benchmark, outperforming all the other state-of-the-art methods that use two reference frames as input.\n          <\/jats:p>","DOI":"10.1145\/3633780","type":"journal-article","created":{"date-parts":[[2023,11,23]],"date-time":"2023-11-23T13:55:02Z","timestamp":1700747702000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Exploring Neighbor Correspondence Matching for Multiple-hypotheses Video Frame Synthesis"],"prefix":"10.1145","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8814-9691","authenticated-orcid":false,"given":"Zhaoyang","family":"Jia","sequence":"first","affiliation":[{"name":"University of Science and Technology of China, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5383-6424","authenticated-orcid":false,"given":"Yan","family":"Lu","sequence":"additional","affiliation":[{"name":"Microsoft Research Asia, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2188-3028","authenticated-orcid":false,"given":"Houqiang","family":"Li","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,1,11]]},"reference":[{"doi-asserted-by":"publisher","key":"e_1_3_2_2_2","DOI":"10.1109\/CVPR.2019.00382"},{"doi-asserted-by":"publisher","key":"e_1_3_2_3_2","DOI":"10.1109\/TPAMI.2019.2941941"},{"doi-asserted-by":"publisher","key":"e_1_3_2_4_2","DOI":"10.1109\/CVPR.2019.00700"},{"unstructured":"Ho Kei Cheng Yu-Wing Tai and Chi-Keung Tang. 2021. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. Advances in Neural Information Processing Systems 34 (2021) 11781\u201311794.","key":"e_1_3_2_5_2"},{"doi-asserted-by":"publisher","key":"e_1_3_2_6_2","DOI":"10.1609\/aaai.v34i07.6693"},{"doi-asserted-by":"publisher","key":"e_1_3_2_7_2","DOI":"10.1109\/CVPR.2016.595"},{"doi-asserted-by":"publisher","key":"e_1_3_2_8_2","DOI":"10.5555\/861369"},{"doi-asserted-by":"publisher","key":"e_1_3_2_9_2","DOI":"10.1109\/CVPR52729.2023.00593"},{"doi-asserted-by":"publisher","key":"e_1_3_2_10_2","DOI":"10.1007\/978-3-031-19781-9_36"},{"doi-asserted-by":"publisher","key":"e_1_3_2_11_2","DOI":"10.1007\/978-3-030-58565-5_11"},{"doi-asserted-by":"publisher","key":"e_1_3_2_12_2","DOI":"10.1145\/3503161.3548163"},{"doi-asserted-by":"publisher","key":"e_1_3_2_13_2","DOI":"10.1109\/CVPR.2018.00938"},{"doi-asserted-by":"publisher","key":"e_1_3_2_14_2","DOI":"10.1109\/ICCV48922.2021.00963"},{"doi-asserted-by":"publisher","key":"e_1_3_2_15_2","DOI":"10.1145\/2980179.2980251"},{"doi-asserted-by":"publisher","key":"e_1_3_2_16_2","DOI":"10.1109\/CVPR42600.2020.00536"},{"doi-asserted-by":"publisher","key":"e_1_3_2_17_2","DOI":"10.1109\/CVPR52729.2023.00945"},{"doi-asserted-by":"publisher","key":"e_1_3_2_18_2","DOI":"10.1109\/TMM.2019.2961504"},{"doi-asserted-by":"publisher","key":"e_1_3_2_19_2","DOI":"10.1007\/978-3-030-66823-5_3"},{"doi-asserted-by":"publisher","key":"e_1_3_2_20_2","DOI":"10.1109\/ICCV.2017.478"},{"unstructured":"Ilya Loshchilov and Frank Hutter. 2018. Fixing weight decay regularization in adam. (2018).","key":"e_1_3_2_21_2"},{"doi-asserted-by":"publisher","key":"e_1_3_2_22_2","DOI":"10.1109\/CVPR.2019.01126"},{"doi-asserted-by":"publisher","key":"e_1_3_2_23_2","DOI":"10.1109\/CVPR52688.2022.00352"},{"doi-asserted-by":"publisher","key":"e_1_3_2_24_2","DOI":"10.1109\/CVPR.2018.00183"},{"doi-asserted-by":"publisher","key":"e_1_3_2_25_2","DOI":"10.1109\/CVPR42600.2020.00548"},{"doi-asserted-by":"publisher","key":"e_1_3_2_26_2","DOI":"10.1109\/ICCV.2017.37"},{"doi-asserted-by":"publisher","key":"e_1_3_2_27_2","DOI":"10.1007\/978-3-030-58568-6_7"},{"doi-asserted-by":"publisher","key":"e_1_3_2_28_2","DOI":"10.1109\/ICCV48922.2021.01427"},{"doi-asserted-by":"publisher","key":"e_1_3_2_29_2","DOI":"10.1109\/ICCV48922.2021.00661"},{"doi-asserted-by":"publisher","key":"e_1_3_2_30_2","DOI":"10.1109\/CVPR.2017.291"},{"doi-asserted-by":"publisher","key":"e_1_3_2_31_2","DOI":"10.1109\/CVPR.2016.445"},{"doi-asserted-by":"crossref","unstructured":"Xihua Sheng Jiahao Li Bin Li Li Li Dong Liu and Yan Lu. 2023. Temporal context mining for learned video compression. IEEE Transactions on Multimedia 25 (2023) 7311\u20137322.","key":"e_1_3_2_32_2","DOI":"10.1109\/TMM.2022.3220421"},{"doi-asserted-by":"publisher","key":"e_1_3_2_33_2","DOI":"10.1109\/TMM.2021.3052419"},{"doi-asserted-by":"publisher","key":"e_1_3_2_34_2","DOI":"10.1109\/ICCV48922.2021.01422"},{"key":"e_1_3_2_35_2","article-title":"UCF101: A dataset of 101 human actions classes from videos in the wild","author":"Soomro Khurram","year":"2012","unstructured":"Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv:1212.0402. Retrieved from https:\/\/arxiv.org\/abs\/1212.0402.","journal-title":"arXiv:1212.0402"},{"doi-asserted-by":"publisher","key":"e_1_3_2_36_2","DOI":"10.1109\/TCSVT.2012.2221191"},{"doi-asserted-by":"publisher","key":"e_1_3_2_37_2","DOI":"10.1109\/CVPR.2018.00931"},{"doi-asserted-by":"publisher","key":"e_1_3_2_38_2","DOI":"10.1007\/978-3-030-58536-5_24"},{"doi-asserted-by":"publisher","key":"e_1_3_2_39_2","DOI":"10.1109\/CVPR52688.2022.01729"},{"unstructured":"Jing Xiao Kangmin Xu Mengshun Hu Liang Liao Zheng Wang Chia-Wen Lin Mi Wang and Shin\u2019ichi Satoh. 2022. Progressive motion boosting for video frame interpolation. IEEE Transactions on Multimedia (2022) 1\u201314.","key":"e_1_3_2_40_2"},{"doi-asserted-by":"publisher","key":"e_1_3_2_41_2","DOI":"10.1007\/s11263-018-01144-2"},{"unstructured":"Gengshan Yang and Deva Ramanan. 2019. Volumetric correspondence networks for optical flow. Advances in Neural Information Processing Systems 32 (2019) 794\u2013805.","key":"e_1_3_2_42_2"},{"unstructured":"Zongxin Yang Yunchao Wei and Yi Yang. 2021. Collaborative video object segmentation by multi-scale foreground-background integration. IEEE Transactions on Pattern Analysis and Machine Intelligence 44 9 (2021) 4701\u20134712.","key":"e_1_3_2_43_2"},{"doi-asserted-by":"publisher","key":"e_1_3_2_44_2","DOI":"10.1109\/CVPR52729.2023.00550"},{"doi-asserted-by":"publisher","key":"e_1_3_2_45_2","DOI":"10.1109\/CVPR52688.2022.01707"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3633780","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3633780","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:35:48Z","timestamp":1750178148000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3633780"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,11]]},"references-count":44,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,4,30]]}},"alternative-id":["10.1145\/3633780"],"URL":"https:\/\/doi.org\/10.1145\/3633780","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"type":"print","value":"1551-6857"},{"type":"electronic","value":"1551-6865"}],"subject":[],"published":{"date-parts":[[2024,1,11]]},"assertion":[{"value":"2023-05-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-11","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}