{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T14:58:23Z","timestamp":1776351503389,"version":"3.51.2"},"reference-count":23,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T00:00:00Z","timestamp":1776297600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100010418","name":"ETRI Research and Development Support Program of MSIT\/IITP","doi-asserted-by":"publisher","award":["RS-2023-00216821"],"award-info":[{"award-number":["RS-2023-00216821"]}],"id":[{"id":"10.13039\/501100010418","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Accurate alignment of real-world object poses with their virtual counterparts using sensors, e.g. cameras, is essential for consistent interaction in mixed-reality systems. However, objects can undergo abrupt, untracked movements during periods when a tracking system is inactive, e.g., overnight, causing stored pose records to become inconsistent with the real scene and breaking user interaction in the virtual environment. Off-the-shelf 3D reconstruction networks such as MASt3R (Matching and Stereo 3D Reconstruction) method provide metrically scaled 3D point maps and pixel correspondences, but they are trained on static scenes and therefore fail to produce reliable object correspondences when the object has moved. We propose a robust pipeline that combines MASt3R\u2019s metrically scaled 3D outputs with a background-based alignment strategy to recover and apply the true pose change of moved objects. Our method first segments foreground and background and extracts 3D background point sets for a reference day and a current day. An affine transformation between these background point sets is estimated via a standard registration technique and used to express the current-day object 3D coordinates in the reference coordinate frame. Within that unified frame we compute the object pose change and apply the resulting transform to the virtual object, restoring real\u2013virtual consistency. Experiments on real scenes demonstrate that the proposed approach reliably corrects pose misalignments introduced during inactive periods and substantially improves over applying MASt3R alone, thereby enabling restored and consistent user interaction in the virtual environment.<\/jats:p>","DOI":"10.3390\/s26082453","type":"journal-article","created":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T14:05:36Z","timestamp":1776348336000},"page":"2453","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Reliable Object Pose Alignment in Mixed-Reality Environments Using Background-Referenced 3D Reconstruction"],"prefix":"10.3390","volume":"26","author":[{"given":"Gyu-Bin","family":"Shin","sequence":"first","affiliation":[{"name":"HCI Laboratory, Kookmin University, Seoul 02707, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7252-6712","authenticated-orcid":false,"given":"Bok-Deuk","family":"Song","sequence":"additional","affiliation":[{"name":"Hyper-Reality Metaverse Research Laboratory, ETRI (Electronics and Telecommunications Research Institute), Daejeon 34129, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vladimirov Blagovest","family":"Iordanov","sequence":"additional","affiliation":[{"name":"Warrior Augmentation Research Section, ETRI (Electronics and Telecommunications Research Institute), Daejeon 34129, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sangjoon","family":"Park","sequence":"additional","affiliation":[{"name":"Warrior Augmentation Research Section, ETRI (Electronics and Telecommunications Research Institute), Daejeon 34129, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Soyeon","family":"Lee","sequence":"additional","affiliation":[{"name":"Warrior Augmentation Research Section, ETRI (Electronics and Telecommunications Research Institute), Daejeon 34129, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Suk-Ho","family":"Lee","sequence":"additional","affiliation":[{"name":"Department Computer Engineering, Dongseo University, Busan 47011, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2026,4,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Leroy, V., Cabon, Y., and Revaud, J. (October, January 29). Grounding Image Matching in 3D with MASt3R. Proceedings of the European Conference on Computer Vision, ECCV 2024, Milan, Italy.","DOI":"10.1007\/978-3-031-73220-1_5"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Martinez, J., Hossain, R., Romero, J., and Little, J.J. (2017, January 22\u201329). A simple yet effective baseline for 3D human pose estimation. Proceedings of the IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy.","DOI":"10.1109\/ICCV.2017.288"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Pavllo, D., Feichtenhofer, C., Grangier, D., and Auli, M. (2019, January 16\u201320). 3D human pose estimation in video with temporal convolutions and semi-supervised training. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00794"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1325","DOI":"10.1109\/TPAMI.2013.248","article-title":"Human3.6M Large scale datasets and predictive methods for 3D human sensing","volume":"36","author":"Ionescu","year":"2014","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zheng, C., Zhu, S., Mendieta, M., Yang, T., Chen, C., and Ding, Z. (2021, January 11\u201317). 3D Human Pose Estimation with Spatial and Temporal Transformers. Proceedings of the IEEE International Conference on Computer Vision, ICCV 2021, Virtual.","DOI":"10.1109\/ICCV48922.2021.01145"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Mao, W., Liu, M., and Salzmann, M. (2021, January 23\u201328). History repeats itself: Human motion prediction via motion attention. Proceedings of the European Conference on Computer Vision, ECCV 2020, Virtual.","DOI":"10.1007\/978-3-030-58568-6_28"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Mehraban, S., Adeli, V., and Taati, B. (2024, January 4\u20138). MotionAGFormer: Enhancing 3D human pose estimation with a transformer-GCNFormer network. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, WACV 2024, Waikoloa, HI, USA.","DOI":"10.1109\/WACV57701.2024.00677"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"110116","DOI":"10.1016\/j.patcog.2023.110116","article-title":"JoyPose: Jointly learning evolutionary data augmentation and anatomy-aware global\u2013local representation for 3D human pose estimation","volume":"147","author":"Du","year":"2024","journal-title":"Pattern Recognit."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wang, J., Yang, X., Li, B., Gou, W., Yan, D., Zeng, A., Gao, Y., Wang, J., Jing, Y., and Zhang, R. (2024, January 17\u201321). FreeMan: Towards Benchmarking 3D Human Pose Estimation under Real-World Conditions. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.02075"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Neupane, B., Stankovic, V., and Stankovic, L. (2025). A Survey of the State of the Art in Monocular 3D Human Pose Estimation: Methods, Benchmarks, and Challenges. Sensors, 25.","DOI":"10.3390\/s25082409"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"105437","DOI":"10.1016\/j.imavis.2025.105437","article-title":"Markerless multi-view 3D human pose estimation: A survey","volume":"155","author":"Nogueira","year":"2025","journal-title":"Image Vis. Comput."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"De, S. (2026). Efficient Mesh Reconstruction and Texturing of Oracle Bones. Sensors, 26.","DOI":"10.3390\/s26072270"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Lu, Y., Wang, H., Ning, Q., Liu, Z., Zang, Y., Liao, Z., and Yan, Z. (2026). In-Orbit MapAnything: An Enhanced Feed-Forward Metric Framework for 3D Reconstruction of Non-Cooperative Space Targets Under Complex Lighting. Sensors, 26.","DOI":"10.3390\/s26072026"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Murai, R., Dexheimer, E., and Davison, A.J. (2025, January 11\u201315). MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA.","DOI":"10.1109\/CVPR52734.2025.01556"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Li, J., Lee, Y., Yadav, A.K., Peng, C., Chellappa, R., and Fan, D. (2025, January 4\u20136). Speedy MASt3R. Proceedings of the IEEE International Conference on Omni-layer Intelligent Systems, COINS 2025, Madison, WI, USA.","DOI":"10.1109\/COINS65080.2025.11125757"},{"key":"ref_16","unstructured":"Chen, X., Xia, T., Xu, S., Yang, J., Chai, J., and Cheng, Z. (2025). SAB3R: Semantic-Augmented Backbone in 3D Reconstruction. arXiv."},{"key":"ref_17","unstructured":"Rojas, S., Armando, M., Ghamen, B., Weinzaepfel, P., Leroy, V., and Rogez, G. (2025, January 19\u201323). HAMSt3R: Human-Aware Multi-view Stereo 3D Reconstruction. Proceedings of the IEEE International Conference on Computer Vision, ICCV 2025, Honolulu, HI, USA."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Cai, Q., Hu, X., Hou, S., Yao, L., and Huang, Y. (2024, January 20\u201324). Disentangled Diffusion-Based 3D Human Pose Estimation with Hierarchical Spatial and Temporal Denoiser. Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Vancouver, BC, Canada.","DOI":"10.1609\/aaai.v38i2.27847"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1109\/34.121791","article-title":"A Method for Registration of 3-D Shapes","volume":"14","author":"Besl","year":"1992","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"376","DOI":"10.1109\/34.88573","article-title":"Least-Squares Estimation of Transformation Parameters Between Two Point Patterns","volume":"13","author":"Umeyama","year":"1991","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"698","DOI":"10.1109\/TPAMI.1987.4767965","article-title":"Least-Squares Fitting of Two 3-D Point Sets","volume":"9","author":"Arun","year":"1987","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2241","DOI":"10.1109\/TPAMI.2015.2513405","article-title":"Go-ICP: A Globally Optimal Solution to 3D ICP Point-Set Registration","volume":"38","author":"Yang","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.Y. (2023, January 2\u20136). Segment Anything. Proceedings of the IEEE International Conference on Computer Vision, ICCV 2023, Paris, France.","DOI":"10.1109\/ICCV51070.2023.00371"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/8\/2453\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T14:07:05Z","timestamp":1776348425000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/8\/2453"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,16]]},"references-count":23,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2026,4]]}},"alternative-id":["s26082453"],"URL":"https:\/\/doi.org\/10.3390\/s26082453","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,16]]}}}