{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,29]],"date-time":"2026-07-29T11:10:47Z","timestamp":1785323447357,"version":"3.55.0"},"reference-count":30,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,1,13]],"date-time":"2024-01-13T00:00:00Z","timestamp":1705104000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,1,13]],"date-time":"2024-01-13T00:00:00Z","timestamp":1705104000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"crossref","award":["NRF-2020R1A2C2011709"],"award-info":[{"award-number":["NRF-2020R1A2C2011709"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Virtual Reality"],"published-print":{"date-parts":[[2024,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Immersive video stored in multiview video-plus-depth format can provide viewers with vivid immersive experiences. However, rendering such video in real time in immersive environments remains a challenging task due to the high resolution and refresh rate demanded by recent extended reality displays. An essential issue in this immersive rendering is the disocclusion problem that inevitably occurs when virtual views are synthesized via the de facto standard 3D warping technique. In this paper, we present a novel virtual view synthesis framework that, from a live immersive video stream, renders stereoscopic images in real time for a freely moving virtual viewer. The main difference from previous approaches is that the surrounding background environment of the immersive video\u2019s virtual scene is progressively reproduced on the fly directly in the 3D space while the input stream is being rendered. To allow this, we propose a new 3D background modeling scheme that, based on GPU-accelerated real-time ray tracing, efficiently and incrementally builds the background model in compact 3D triangular mesh. Then, we demonstrate that the 3D background environment can effectively alleviate the critical disocclusion problem in the immersive rendering, eventually reducing spatial and temporal aliasing artifacts. It is also suggested that the 3D representation of background environment enables extension of the virtual environment of immersive video by interactively adding 3D visual effects during rendering.<\/jats:p>","DOI":"10.1007\/s10055-023-00921-w","type":"journal-article","created":{"date-parts":[[2024,1,13]],"date-time":"2024-01-13T14:01:43Z","timestamp":1705154503000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Ray tracing-based construction of 3D background model for real-time stereoscopic rendering of live immersive video"],"prefix":"10.1007","volume":"28","author":[{"given":"Youngwook","family":"Kim","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jehyeong","family":"Yun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joungil","family":"Yun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sangwoon","family":"Kwak","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5611-925X","authenticated-orcid":false,"given":"Insung","family":"Ihm","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,1,13]]},"reference":[{"issue":"9","key":"921_CR1","doi-asserted-by":"publisher","first-page":"1521","DOI":"10.1109\/JPROC.2021.3062590","volume":"109","author":"JM Boyce","year":"2021","unstructured":"Boyce JM, Dor\u00e9 R, Dziembowski A, Fleureau J, Jung J, Kroon B, Salahieh B, Vadakital VKM, Yu L (2021) MPEG immersive video coding standard. Proc IEEE 109(9):1521\u20131536","journal-title":"Proc IEEE"},{"issue":"4","key":"921_CR2","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1145\/3386569.3392485","volume":"39","author":"M Broxton","year":"2020","unstructured":"Broxton M, Flynn J, Overbeck R, Erickson D, Hedman P, Duvall M, Dourgarian J, Busch J, Whalen M, Debevec P (2020) Immersive light field video with a layered mesh representation. ACM Trans Gr 39(4):86","journal-title":"ACM Trans Gr"},{"key":"921_CR3","doi-asserted-by":"crossref","unstructured":"Chen KY, Tsung PK, Lin PC, Yang HJ, Chen LG (2010) Hybrid motion\/depth-oriented inpainting for virtual view synthesis in multiview applications. In: 2010 3DTV-conference: the true vision - capture, transmission and display of 3D video, pp. 1\u20134","DOI":"10.1109\/3DTV.2010.5506309"},{"issue":"9","key":"921_CR4","doi-asserted-by":"publisher","first-page":"1200","DOI":"10.1109\/TIP.2004.833105","volume":"13","author":"A Criminisi","year":"2004","unstructured":"Criminisi A, Perez P, Toyama K (2004) Region filling and object removal by exemplar-based image inpainting. IEEE Trans Image Process 13(9):1200\u20131212","journal-title":"IEEE Trans Image Process"},{"issue":"1","key":"921_CR5","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1007\/s11263-006-8525-1","volume":"71","author":"A Criminisi","year":"2007","unstructured":"Criminisi A, Blake A, Rother C, Shotton J, Torr PHS (2007) Efficient dense stereo with occlusions for new view-synthesis by four-state dynamic programming. Int J Comput Vis 71(1):89\u2013110","journal-title":"Int J Comput Vis"},{"key":"921_CR6","unstructured":"Dor\u00e9, R.: Technicolor 3DoF+ test materials. Standard ISO\/IEC JTC1\/SC29\/WG11 MPEG\/M42349 (2018)"},{"key":"921_CR7","unstructured":"Doyen D et al. (2017) Light field content from 16-camera rig. Standard ISO\/IEC JTC1\/SC29\/WG11 MPEG\/M40010"},{"issue":"2","key":"921_CR8","doi-asserted-by":"publisher","first-page":"2007","DOI":"10.1007\/s11063-019-10163-0","volume":"51","author":"O Elharrouss","year":"2020","unstructured":"Elharrouss O, Almaadeed N, Al-Maadeed S, Akbari Y (2020) Image inpainting: a review. Neural Process Lett 51(2):2007\u20132028","journal-title":"Neural Process Lett"},{"key":"921_CR9","doi-asserted-by":"crossref","unstructured":"Fehn, C.: Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In: Proceedings of stereoscopic displays and virtual reality systems XI\u00a0(SPIE), vol. 5291, pp. 93\u2013104 (2004)","DOI":"10.1117\/12.524762"},{"key":"921_CR10","doi-asserted-by":"crossref","unstructured":"Flynn J, Broxton M, Debevec P, DuVall M, Fyffe G, Overbeck R, Snavely N, Tucker R (2019) DeepView: view synthesis with learned gradient descent. In: 2019 IEEE\/CVF conference on computer vision and pattern recognition (CVPR), pp. 2362\u20132371 (2019)","DOI":"10.1109\/CVPR.2019.00247"},{"key":"921_CR11","doi-asserted-by":"crossref","unstructured":"Huang Y, Zhang C (2008) A layered method of visibility resolving in depth image-based rendering. In: 2008 19th international conference on pattern recognition, pp. 1\u20134","DOI":"10.1109\/ICPR.2008.4761661"},{"key":"921_CR12","doi-asserted-by":"publisher","first-page":"103147","DOI":"10.1016\/j.cviu.2020.103147","volume":"203","author":"J Jam","year":"2021","unstructured":"Jam J, Kendrick C, Walker K, Drouard V, Hsu J, Yap MH (2021) A comprehensive review of past and present image inpainting methods. Comput Vis Image Underst 203:103147","journal-title":"Comput Vis Image Underst"},{"key":"921_CR13","unstructured":"Jung J, Kroon B (2022) Common test conditions for MPEG immersive video. Standard ISO\/IED JTC1\/SC29\/WG04 MPEG\/N0203"},{"key":"921_CR14","doi-asserted-by":"crossref","unstructured":"Lee J, Kim Y, Yun J, Yun J, Cheong WS, Ihm I (2021) Disocclusion-reducing geometry for multiple RGB-D video streams. In: 2021 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops (VRW), pp. 603\u2013604","DOI":"10.1109\/VRW52623.2021.00184"},{"issue":"6","key":"921_CR15","doi-asserted-by":"publisher","first-page":"1289","DOI":"10.1109\/TPAMI.2019.2899837","volume":"42","author":"G Luo","year":"2020","unstructured":"Luo G, Zhu Y, Weng Z, Li Z (2020) A disocclusion inpainting framework for depth-based view synthesis. IEEE Trans Pattern Anal Mach Intell 42(6):1289\u20131302","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"921_CR16","doi-asserted-by":"crossref","unstructured":"Luo G, Zhu Y, Li Z, Zhang L (2016) A hole filling approach based on background reconstruction for view synthesis in 3D video. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp. 1781\u20131789","DOI":"10.1109\/CVPR.2016.197"},{"key":"921_CR17","doi-asserted-by":"crossref","unstructured":"Mark WR, McMillan L, Bishop G (1997) Post-rendering 3D warping. In: Proceedings of the 1997 symposium on interactive 3D graphics (I3D \u201997), pp. 7\u201316","DOI":"10.1145\/253284.253292"},{"issue":"4","key":"921_CR18","doi-asserted-by":"publisher","first-page":"643","DOI":"10.1109\/JPROC.2010.2091090","volume":"99","author":"K M\u00fcller","year":"2011","unstructured":"M\u00fcller K, Merkle P, Wiegand T (2011) 3-D video representation using depth maps. Proc IEEE 99(4):643\u2013656","journal-title":"Proc IEEE"},{"issue":"3","key":"921_CR19","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1109\/TMM.2011.2128862","volume":"13","author":"P Ndjiki-Nya","year":"2011","unstructured":"Ndjiki-Nya P, K\u00f6ppel M, Doshkov D, Lakshman H, Merkle P, Muller K, Wiegand T (2011) Depth image-based rendering with advanced texture synthesis for 3-D video. IEEE Trans Multimed 13(3):453\u2013465","journal-title":"IEEE Trans Multimed"},{"key":"921_CR20","unstructured":"NVIDIA: NVIDIA OptiX 7.4 Programming Guide (2021)"},{"key":"921_CR21","doi-asserted-by":"publisher","first-page":"102028","DOI":"10.1016\/j.displa.2021.102028","volume":"69","author":"Z Qin","year":"2021","unstructured":"Qin Z, Zeng Q, Zong Y, Xu F (2021) Image inpainting based on deep learning: a review. Displays 69:102028","journal-title":"Displays"},{"issue":"3","key":"921_CR22","doi-asserted-by":"publisher","first-page":"1190","DOI":"10.1109\/TIP.2017.2772858","volume":"27","author":"DMM Rahaman","year":"2018","unstructured":"Rahaman DMM, Paul M (2018) Virtual view synthesis for free viewpoint video and multiview video compression using Gaussian mixture modeling. IEEE Trans Image Process 27(3):1190\u20131201","journal-title":"IEEE Trans Image Process"},{"key":"921_CR23","unstructured":"Salahieh B et al. (2018) Kermit test sequence for windowed 6DoF activities. Standard ISO\/IEC JTC1\/SC29\/WG11 MPEG\/M43748"},{"key":"921_CR24","doi-asserted-by":"crossref","unstructured":"Schmeing M, Jiang X (2010) Depth image based rendering: A faithful approach for the disocclusion problem. In: 2010 3DTV\u2014conference: the true vision\u2014capture, transmission and display of 3D video, pp. 1\u20134","DOI":"10.1109\/3DTV.2010.5506596"},{"issue":"9","key":"921_CR25","doi-asserted-by":"publisher","first-page":"1958","DOI":"10.1016\/j.patcog.2010.09.005","volume":"44","author":"A Smolic","year":"2011","unstructured":"Smolic A (2011) 3D video and free viewpoint video\u2014from capture to display. Pattern Recognit 44(9):1958\u20131968","journal-title":"Pattern Recognit"},{"key":"921_CR26","doi-asserted-by":"crossref","unstructured":"Sun W, Au OC, Xu L, Li Y, Hu W (2012) Novel temporal domain hole filling based on background modeling for view synthesis. In: 2012 19th IEEE international conference on image processing, pp. 2721\u20132724","DOI":"10.1109\/ICIP.2012.6467461"},{"issue":"3","key":"921_CR27","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1109\/MMUL.2022.3175654","volume":"29","author":"VKM Vadakital","year":"2022","unstructured":"Vadakital VKM, Dziembowski A, Lafruit G, Thudor F, Lee G, Alface PR (2022) The MPEG immersive video standard\u2013current status and future outlook. IEEE Multimed 29(3):101\u2013111","journal-title":"IEEE Multimed"},{"issue":"2","key":"921_CR28","doi-asserted-by":"publisher","first-page":"394","DOI":"10.1109\/TBC.2014.2321671","volume":"60","author":"C Yao","year":"2014","unstructured":"Yao C, Tillo T, Zhao Y, Xiao J, Bai H, Lin C (2014) Depth map driven hole filling algorithm exploiting temporal correlation information. IEEE Trans Broadcast 60(2):394\u2013404","journal-title":"IEEE Trans Broadcast"},{"key":"921_CR29","doi-asserted-by":"crossref","unstructured":"Zhou T, Tucker R, Flynn J, Fyffe G, Snavely N (2018) Stereo magnification: learning view synthesis using multiplane images. ACM Trans Gr 37(4):65","DOI":"10.1145\/3197517.3201323"},{"key":"921_CR30","volume-title":"3D-TV system with depth-image-based rendering: architectures, techniques and challenges","year":"2013","unstructured":"Zhu C, Zhao Y, Yu L, Tanimoto M (eds) (2013) 3D-TV system with depth-image-based rendering: architectures, techniques and challenges. Springer, New York"}],"container-title":["Virtual Reality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10055-023-00921-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10055-023-00921-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10055-023-00921-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,17]],"date-time":"2024-03-17T04:17:09Z","timestamp":1710649029000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10055-023-00921-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,13]]},"references-count":30,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,3]]}},"alternative-id":["921"],"URL":"https:\/\/doi.org\/10.1007\/s10055-023-00921-w","relation":{},"ISSN":["1359-4338","1434-9957"],"issn-type":[{"value":"1359-4338","type":"print"},{"value":"1434-9957","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,13]]},"assertion":[{"value":"15 January 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 December 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 January 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"17"}}