{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,30]],"date-time":"2025-07-30T14:14:11Z","timestamp":1753884851308,"version":"3.41.2"},"reference-count":69,"publisher":"World Scientific Pub Co Pte Ltd","issue":"16","funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"crossref","award":["JP16H01744"],"award-info":[{"award-number":["JP16H01744"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP17H01847"],"award-info":[{"award-number":["JP17H01847"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP17H06101"],"award-info":[{"award-number":["JP17H06101"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP19H01129"],"award-info":[{"award-number":["JP19H01129"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["JP19H04137"],"award-info":[{"award-number":["JP19H04137"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100009025","name":"Accelerated Innovation Research Initiative Turning Top Science and Ideas into High-Impact Values","doi-asserted-by":"publisher","award":["JPMJAC1602"],"award-info":[{"award-number":["JPMJAC1602"]}],"id":[{"id":"10.13039\/501100009025","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002241","name":"Japan Science and Technology Agency","doi-asserted-by":"publisher","award":["JPMJMI19B2"],"award-info":[{"award-number":["JPMJMI19B2"]}],"id":[{"id":"10.13039\/501100002241","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Patt. Recogn. Artif. Intell."],"published-print":{"date-parts":[[2021,12,30]]},"abstract":"<jats:p> This paper describes a deep learning-based method for long-term video interpolation that generates intermediate frames between two music performance videos of a person playing a specific instrument. Recent advances in deep learning techniques have successfully generated realistic images with high-fidelity and high-resolution in short-term video interpolation. However, there is still room for improvement in long-term video interpolation due to lack of resolution and temporal consistency of the generated video. Particularly in music performance videos, the music and human performance motion need to be synchronized. We solved these problems by using human poses and music features essential for music performance in long-term video interpolation. By closely matching human poses with music and videos, it is possible to generate intermediate frames that synchronize with the music. Specifically, we obtain the human poses of the last frame of the first video and the first frame of the second video in the performance videos to be interpolated as key poses. Then, our encoder\u2013decoder network estimates the human poses in the intermediate frames from the obtained key poses, with the music features as the condition. In order to construct an end-to-end network, we utilize a differentiable network that transforms the estimated human poses in vector form into the human pose in image form, such as human stick figures. Finally, a video-to-video synthesis network uses the stick figures to generate intermediate frames between two music performance videos. We found that the generated performance videos were of higher quality than the baseline method through quantitative experiments. <\/jats:p>","DOI":"10.1142\/s0218001421600168","type":"journal-article","created":{"date-parts":[[2022,1,24]],"date-time":"2022-01-24T12:18:28Z","timestamp":1643026708000},"source":"Crossref","is-referenced-by-count":3,"title":["Audio-Oriented Video Interpolation Using Key Pose"],"prefix":"10.1142","volume":"35","author":[{"given":"Takayuki","family":"Nakatsuka","sequence":"first","affiliation":[{"name":"Department of Pure and Applied Physics, Waseda University, 55N Building, Room 406, 3-4-1 Okubo, Shinjuku, Tokyo 169-8555, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yukitaka","family":"Tsuchiya","sequence":"additional","affiliation":[{"name":"Department of Pure and Applied Physics, Waseda University, 55N Building, Room 406, 3-4-1 Okubo, Shinjuku, Tokyo 169-8555, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Masatoshi","family":"Hamanaka","sequence":"additional","affiliation":[{"name":"RIKEN, Nihonbashi 1-Chome Mitsui Building, 15th Floor, 1-4-1 Nihonbashi, Chuo-KU, Tokyo 103-0027, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shigeo","family":"Morishima","sequence":"additional","affiliation":[{"name":"Waseda Research Institute for Science and Engineering, 55N Building, Room 406, 3-4-1 Okubo, Shinjuku, Tokyo 169-8555, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"219","published-online":{"date-parts":[[2022,1,22]]},"reference":[{"key":"S0218001421600168BIB001","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.2977333"},{"issue":"17","key":"S0218001421600168BIB002","first-page":"26","volume":"8","author":"Alemi O.","year":"2017","journal-title":"Networks"},{"key":"S0218001421600168BIB003","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00382"},{"key":"S0218001421600168BIB004","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2019.2941941"},{"key":"S0218001421600168BIB005","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2004.840597"},{"key":"S0218001421600168BIB006","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2019.2929257"},{"key":"S0218001421600168BIB007","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00603"},{"key":"S0218001421600168BIB008","doi-asserted-by":"publisher","DOI":"10.1109\/VCIP.2017.8305029"},{"key":"S0218001421600168BIB009","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.6634"},{"key":"S0218001421600168BIB010","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i07.6693"},{"key":"S0218001421600168BIB011","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01034"},{"key":"S0218001421600168BIB012","first-page":"1174","volume-title":"Int. Conf. Machine Learning (ICML)","volume":"80","author":"Denton E. L.","year":"2018"},{"key":"S0218001421600168BIB013","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2011.73"},{"key":"S0218001421600168BIB014","doi-asserted-by":"publisher","DOI":"10.1117\/12.290336"},{"key":"S0218001421600168BIB015","first-page":"153","volume-title":"Int. Conf. Advances in Computer Entertainment (ACE)","author":"Furukawa S.","year":"2017"},{"key":"S0218001421600168BIB016","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00361"},{"key":"S0218001421600168BIB017","first-page":"2672","volume-title":"Advances in Neural Information Processing Systems (NeurIPS)","author":"Goodfellow I.","year":"2014"},{"key":"S0218001421600168BIB018","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01402"},{"key":"S0218001421600168BIB019","first-page":"3","volume-title":"Int. Symp. Computer Music Multidisciplinary Research (CMMR)","author":"Hamanaka M.","year":"2016"},{"key":"S0218001421600168BIB020","first-page":"238","volume-title":"Int. Symp. Computer Music Multidisciplinary Research (CMMR)","author":"Hamanaka M.","year":"2017"},{"key":"S0218001421600168BIB021","first-page":"19","volume-title":"ACM SIGGRAPH Emerging Technologies","author":"Hamanaka M.","year":"2019"},{"key":"S0218001421600168BIB022","doi-asserted-by":"publisher","DOI":"10.1093\/ml\/59.2.180"},{"key":"S0218001421600168BIB023","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-019-01219-8"},{"key":"S0218001421600168BIB024","first-page":"6626","volume-title":"Advances in Neural Information Processing Systems (NeurIPS)","author":"Heusel M.","year":"2017"},{"key":"S0218001421600168BIB025","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"volume-title":"Int. Conf. Learning Representations (ICLR)","year":"2020","author":"Huang R.","key":"S0218001421600168BIB026"},{"key":"S0218001421600168BIB027","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00938"},{"volume-title":"Int. Conf. Learning Representations (ICLR)","year":"2019","author":"Jolicoeur-Martineau A.","key":"S0218001421600168BIB028"},{"key":"S0218001421600168BIB029","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073658"},{"volume-title":"Int. Conf. Learning Representations (ICLR)","year":"2018","author":"Karras T.","key":"S0218001421600168BIB030"},{"key":"S0218001421600168BIB031","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00411"},{"key":"S0218001421600168BIB033","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00536"},{"key":"S0218001421600168BIB034","doi-asserted-by":"publisher","DOI":"10.1007\/s11042-012-1288-5"},{"key":"S0218001421600168BIB035","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/12513.001.0001"},{"key":"S0218001421600168BIB037","first-page":"13401","volume-title":"IEEE Conf. Computer Vision (ICCV)","author":"Li R.","year":"2021"},{"key":"S0218001421600168BIB039","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33018794"},{"key":"S0218001421600168BIB040","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2000.859068"},{"volume-title":"Int. Symp. Music Information Retrieval (ISMIR)","year":"2000","author":"Logan B.","key":"S0218001421600168BIB041"},{"key":"S0218001421600168BIB043","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00059"},{"key":"S0218001421600168BIB044","doi-asserted-by":"publisher","DOI":"10.5220\/0008876600270035"},{"key":"S0218001421600168BIB045","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00183"},{"key":"S0218001421600168BIB046","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.00548"},{"key":"S0218001421600168BIB047","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.37"},{"key":"S0218001421600168BIB048","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2011.2181492"},{"key":"S0218001421600168BIB049","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00772"},{"key":"S0218001421600168BIB050","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58568-6_7"},{"key":"S0218001421600168BIB051","first-page":"14539","volume-title":"Proc. IEEE\/CVF Int. Conf. Computer Vision","author":"Park J.","year":"2021"},{"key":"S0218001421600168BIB052","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00762"},{"key":"S0218001421600168BIB053","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.48"},{"key":"S0218001421600168BIB054","doi-asserted-by":"publisher","DOI":"10.1121\/1.1919362"},{"key":"S0218001421600168BIB055","doi-asserted-by":"publisher","DOI":"10.1142\/S0218001417580010"},{"key":"S0218001421600168BIB056","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00790"},{"volume-title":"Int. Conf. Learning Representations (ICLR)","year":"2015","author":"Simonyan K.","key":"S0218001421600168BIB057"},{"key":"S0218001421600168BIB058","doi-asserted-by":"publisher","DOI":"10.1145\/3072959.3073640"},{"key":"S0218001421600168BIB059","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.308"},{"key":"S0218001421600168BIB060","doi-asserted-by":"publisher","DOI":"10.1145\/3240508.3240526"},{"key":"S0218001421600168BIB061","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.510"},{"key":"S0218001421600168BIB062","doi-asserted-by":"publisher","DOI":"10.1117\/12.367679"},{"key":"S0218001421600168BIB063","doi-asserted-by":"publisher","DOI":"10.1109\/WACV45572.2020.9093345"},{"key":"S0218001421600168BIB064","volume-title":"Advances in Neural Information Processing Systems (NeurIPS)","volume":"32","author":"Wang T.-C.","year":"2019"},{"key":"S0218001421600168BIB065","volume-title":"Advances in Neural Information Processing Systems (NeurIPS)","volume":"31","author":"Wang T.-C.","year":"2018"},{"key":"S0218001421600168BIB066","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00917"},{"key":"S0218001421600168BIB067","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.01059"},{"key":"S0218001421600168BIB068","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01261-8_41"},{"key":"S0218001421600168BIB069","doi-asserted-by":"publisher","DOI":"10.1109\/WACV45572.2020.9093530"},{"key":"S0218001421600168BIB070","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-01249-6_13"},{"key":"S0218001421600168BIB071","doi-asserted-by":"publisher","DOI":"10.1145\/3197517.3201292"},{"key":"S0218001421600168BIB072","doi-asserted-by":"publisher","DOI":"10.1109\/ICCVW.2019.00153"},{"key":"S0218001421600168BIB073","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR48806.2021.9412698"}],"container-title":["International Journal of Pattern Recognition and Artificial Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S0218001421600168","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,2,25]],"date-time":"2022-02-25T08:19:42Z","timestamp":1645777182000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/abs\/10.1142\/S0218001421600168"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,30]]},"references-count":69,"journal-issue":{"issue":"16","published-print":{"date-parts":[[2021,12,30]]}},"alternative-id":["10.1142\/S0218001421600168"],"URL":"https:\/\/doi.org\/10.1142\/s0218001421600168","relation":{},"ISSN":["0218-0014","1793-6381"],"issn-type":[{"type":"print","value":"0218-0014"},{"type":"electronic","value":"1793-6381"}],"subject":[],"published":{"date-parts":[[2021,12,30]]},"article-number":"2160016"}}