{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T12:03:51Z","timestamp":1772280231929,"version":"3.50.1"},"reference-count":52,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,2,2]],"date-time":"2026-02-02T00:00:00Z","timestamp":1769990400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,2,5]],"date-time":"2026-02-05T00:00:00Z","timestamp":1770249600000},"content-version":"vor","delay-in-days":3,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1942531"],"award-info":[{"award-number":["1942531"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Virtual Reality"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Augmented reality (AR) games, particularly those designed for head-mounted displays, have grown increasingly prevalent. However, most existing systems depend on pre-scanned, static environments and rely heavily on continuous tracking or marker-based solutions, which limit adaptability in dynamic physical spaces. This is particularly problematic for AR headsets and glasses, which typically follow the user\u2019s head movement and cannot maintain a fixed, stationary view of the scene. Moreover, continuous scene observation is neither power-efficient nor practical for wearable devices, given their limited battery and processing capabilities. A persistent challenge arises when multiple identical objects are present in the environment\u2013standard object tracking pipelines often fail to maintain consistent identities without uninterrupted observation or external sensors. These limitations hinder fluid physical-virtual interactions, especially in dynamic or occluded scenes where continuous tracking is infeasible. To address this, we introduce a novel optimization-based framework for re-identifying identical objects in AR scenes using only one partial egocentric observation frame captured by a headset. We formulate the problem as a label assignment task solved via integer programming, augmented with a Voronoi diagram-based pruning strategy to improve computational efficiency. This method reduces computation time by 50% while preserving 91% accuracy in simulated experiments. Moreover, we evaluated our approach in quantitative synthetic and quantitative real-world experiments. We also conducted three qualitative real-world experiments to demonstrate the practical utility and generalizability for enabling dynamic, markerless object interaction in AR environments. Our video demo is available at\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/youtu.be\/RwptEfLtW1U\" ext-link-type=\"uri\">https:\/\/youtu.be\/RwptEfLtW1U<\/jats:ext-link>\n                    .\n                  <\/jats:p>","DOI":"10.1007\/s10055-025-01305-y","type":"journal-article","created":{"date-parts":[[2026,2,2]],"date-time":"2026-02-02T09:04:22Z","timestamp":1770023062000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Enriching physical-virtual interaction in AR gaming by tracking identical objects via an egocentric partial observation frame"],"prefix":"10.1007","volume":"30","author":[{"given":"Liuchuan","family":"Yu","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ching-I","family":"Huang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hsueh-Cheng","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lap-Fai","family":"Yu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2026,2,2]]},"reference":[{"key":"1305_CR1","doi-asserted-by":"crossref","unstructured":"Ahmadyan A, Zhang L, Ablavatski A, Wei J, Grundmann M (2021) Objectron a large scale dataset of object-centric videos in the wild with pose annotations. pp 7822\u20137831","DOI":"10.1109\/CVPR46437.2021.00773"},{"issue":"1","key":"1305_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.29333\/cac\/11925","volume":"1","author":"N Amanatidis","year":"2022","unstructured":"Amanatidis N (2022) Augmented reality in education and educational games-implementation and evaluation: a focused literature review. Comput Child 1(1):1\u201311","journal-title":"Comput Child"},{"issue":"11","key":"1305_CR3","doi-asserted-by":"publisher","first-page":"21","DOI":"10.3991\/ijim.v15i11.20697","volume":"15","author":"B Arifitama","year":"2021","unstructured":"Arifitama B, Hanan G, Rofiqi MH (2021) Mobile augmented reality for campus visualization using markerless tracking in an Indonesian private university. Int J Interact Mob Technol 15(11):21\u201333","journal-title":"Int J Interact Mob Technol"},{"issue":"4","key":"1305_CR4","doi-asserted-by":"publisher","first-page":"355","DOI":"10.1162\/pres.1997.6.4.355","volume":"6","author":"RT Azuma","year":"1997","unstructured":"Azuma RT (1997) A survey of augmented reality. Presence Teleop Virtual Environ 6(4):355\u2013385","journal-title":"Presence Teleop Virtual Environ"},{"key":"1305_CR5","doi-asserted-by":"crossref","unstructured":"Balakrishnan S, Hameed MSS, Venkatesan K, Aswin G (2021) Interaction of spatial computing in augmented reality. In: 2021 7th International conference on advanced computing and communication systems (ICACCS), vol. 1, pp. 1900\u20131904. IEEE","DOI":"10.1109\/ICACCS51430.2021.9442010"},{"key":"1305_CR6","unstructured":"Cai M, Akcayir G, Epp CD (2021) Exploring augmented reality games in accessible learning: a systematic review. arXiv preprint arXiv:2111.08214"},{"key":"1305_CR7","doi-asserted-by":"crossref","unstructured":"Ducasse J, Kljun M, \u010copi\u010d Pucihar K (2019). In: Geroimenko V (ed) Playful ambient augmented reality systems to improve people\u2019s well-being. Springer, Cham, pp 125\u2013157","DOI":"10.1007\/978-3-030-15620-6_6"},{"issue":"5","key":"1305_CR8","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1049\/htl.2018.5079","volume":"5","author":"T Frantz","year":"2018","unstructured":"Frantz T, Jansen B, Duerinck J, Vandemeulebroucke J (2018) Augmenting microsoft\u2019s hololens with vuforia tracking for neuronavigation. Healthc Technol Lett 5(5):221\u2013225","journal-title":"Healthc Technol Lett"},{"key":"1305_CR9","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-15616-9","volume-title":"Augmented reality games II","author":"V Geroimenko","year":"2019","unstructured":"Geroimenko V (2019) Augmented reality games II. Springer, Cham"},{"key":"1305_CR10","doi-asserted-by":"crossref","unstructured":"Hartmann J, Holz C, Ofek E, Wilson, AD, Realitycheck, (2019) Blending virtual environments with situated physical reality. pp 1\u201312","DOI":"10.1145\/3290605.3300577"},{"key":"1305_CR11","doi-asserted-by":"crossref","unstructured":"Hu Y, He W, Zhang L, Li S (2020) Enhancing realism and presence with active physical reactions in augmented reality. pp 700\u2013704","DOI":"10.1145\/3441000.3441080"},{"key":"1305_CR12","doi-asserted-by":"crossref","unstructured":"Jain R, Shi J, Duan R, Zhu Z, Qian X, Ramani K (2023) Ubi-touch: Ubiquitous tangible object utilization through consistent hand-object interaction in augmented reality. pp 1\u201318","DOI":"10.1145\/3586183.3606793"},{"key":"1305_CR13","doi-asserted-by":"crossref","unstructured":"Kaimoto H, Monteiro K, Faridan M, Li J, Farajian S, Kakehi Y, Nakagaki K, Suzuki R (2022) Sketched reality: sketching bi-directional interactions between virtual and physical worlds with AR and actuated tangible UI. pp 1\u201312","DOI":"10.1145\/3526113.3545626"},{"key":"1305_CR14","doi-asserted-by":"crossref","unstructured":"Kato H, Billinghurst M (1999). Marker tracking and HMD calibration for a video-based augmented reality conferencing system, vol IWAR\u201999. IEEE, pp 85\u201394","DOI":"10.1109\/IWAR.1999.803809"},{"key":"1305_CR15","first-page":"596","volume-title":"Approximate differentiable rendering with algebraic surfaces","author":"L Keselman","year":"2022","unstructured":"Keselman L, Hebert M (2022) Approximate differentiable rendering with algebraic surfaces. Springer, Cham, pp 596\u2013614"},{"key":"1305_CR16","doi-asserted-by":"crossref","unstructured":"Kirillov A, Mintun E, Ravi N, Mao H, Rolland C, Gustafson L, Xiao T, Whitehead S, Berg AC, Lo W-Y, Doll\u00e1r P, Girshick R (2023) Segment anything. arXiv:2304.02643","DOI":"10.1109\/ICCV51070.2023.00371"},{"issue":"6","key":"1305_CR17","doi-asserted-by":"publisher","first-page":"1737","DOI":"10.3390\/s20061737","volume":"20","author":"T-Y Ko","year":"2020","unstructured":"Ko T-Y, Lee S-h (2020) Novel method of semantic segmentation applicable to augmented reality. Sensors 20(6):1737","journal-title":"Sensors"},{"issue":"13","key":"1305_CR18","doi-asserted-by":"publisher","first-page":"6006","DOI":"10.3390\/app11136006","volume":"11","author":"H Le","year":"2021","unstructured":"Le H, Nguyen M, Yan WQ, Nguyen H (2021) Augmented reality and machine learning incorporation using yolov3 and arkit. Appl Sci 11(13):6006","journal-title":"Appl Sci"},{"issue":"5","key":"1305_CR19","doi-asserted-by":"publisher","first-page":"7509","DOI":"10.1007\/s11227-021-04161-0","volume":"78","author":"T Lee","year":"2022","unstructured":"Lee T, Jung C, Lee K, Seo S (2022) A study on recognizing multi-real world object and estimating 3d position in augmented reality. J Supercomput 78(5):7509\u20137528","journal-title":"J Supercomput"},{"issue":"8","key":"1305_CR20","doi-asserted-by":"publisher","first-page":"3534","DOI":"10.1109\/TVCG.2019.2959575","volume":"27","author":"M Lee","year":"2019","unstructured":"Lee M, Norouzi N, Bruder G, Wisniewski PJ, Welch GF (2019) Mixed reality tabletop gameplay: social interaction with a virtual human capable of physical influence. IEEE Trans Visual Comput Graph 27(8):3534\u20133545","journal-title":"IEEE Trans Visual Comput Graph"},{"key":"1305_CR21","unstructured":"Lee L-H, Braud T, Zhou P, Wang L, Xu D, Lin Z, Kumar A, Bermejo C, Hui P (2021) All one needs to know about metaverse: a complete survey on technological singularity, virtual ecosystem, and research agenda. arXiv preprint arXiv:2110.05352"},{"issue":"4","key":"1305_CR22","first-page":"1","volume":"41","author":"C Li","year":"2022","unstructured":"Li C, Li W, Huang H, Yu L-F (2022) Interactive augmented reality storytelling guided by scene semantics. ACM Trans Graph 41(4):1\u201315","journal-title":"ACM Trans Graph"},{"issue":"4","key":"1305_CR23","first-page":"1","volume":"42","author":"C Li","year":"2023","unstructured":"Li C, Yu L-F (2023) Generating activity snippets by learning human-scene interactions. ACM Trans Graph (TOG) 42(4):1\u201315","journal-title":"ACM Trans Graph (TOG)"},{"key":"1305_CR24","doi-asserted-by":"crossref","unstructured":"Li W, Li C, Kim M, Huang H, Yu L-F (2023) Location-aware adaptation of augmented reality narratives","DOI":"10.1145\/3544548.3580978"},{"key":"1305_CR25","doi-asserted-by":"crossref","unstructured":"Li H, Whitmire E, Mariakakis A, Chan V, Sample AP, Patel SN (2019) Idcam: Precise item identification for AR enhanced object interactions. IEEE, pp 1\u20137","DOI":"10.1109\/RFID.2019.8719279"},{"key":"1305_CR26","doi-asserted-by":"crossref","unstructured":"Liang W, Yu X, Alghofaili R, Lang Y, Yu L-F (2021) Scene-aware behavior synthesis for virtual pets in mixed reality. pp 1\u201312","DOI":"10.1145\/3411764.3445532"},{"key":"1305_CR27","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1007\/978-3-030-15620-6_10","volume-title":"Augmented reality games II: the gamification of education, medicine and art","author":"P Lichty","year":"2019","unstructured":"Lichty P (2019) The gamification of augmented reality art. In: Geroimenko V (ed) Augmented reality games II: the gamification of education, medicine and art. Springer, Cham, pp 225\u2013246"},{"key":"1305_CR28","doi-asserted-by":"crossref","unstructured":"Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu C-Y, Berg AC (2016) Ssd: single shot multibox detector. In: Computer vision\u2013ECCV 2016: 14th European conference, Amsterdam, The Netherlands, October 11\u201314, 2016, Proceedings, Part I 14, pp. 21\u201337. Springer","DOI":"10.1007\/978-3-319-46448-0_2"},{"key":"1305_CR29","unstructured":"Liu J, Sun W, Yang H, Zeng Z, Liu C, Zheng J, Liu X, Rahmani H, Sebe N, Mian A (2024) Deep learning-based object pose estimation: a comprehensive survey. arXiv preprint arXiv:2405.07801"},{"issue":"4","key":"1305_CR30","doi-asserted-by":"publisher","first-page":"91","DOI":"10.3390\/jimaging8040091","volume":"8","author":"A Marto","year":"2022","unstructured":"Marto A, Gon\u00e7alves A (2022) Augmented reality games and presence: a systematic review. J Imaging 8(4):91","journal-title":"J Imaging"},{"key":"1305_CR31","doi-asserted-by":"crossref","unstructured":"Meinhardt T, Kirillov A, Leal-Taixe L, Feichtenhofer C (2022) Trackformer: multi-object tracking with transformers. pp 8844\u20138854","DOI":"10.1109\/CVPR52688.2022.00864"},{"key":"1305_CR32","doi-asserted-by":"crossref","unstructured":"Mooser J, You S, Neumann U (2007) Real-time object tracking for augmented reality combining graph cuts and optical flow. In: 2007 6th IEEE and ACM international symposium on mixed and augmented reality, pp. 145\u2013152. IEEE","DOI":"10.1109\/ISMAR.2007.4538839"},{"key":"1305_CR33","doi-asserted-by":"crossref","unstructured":"Nelson CR, Gabbard JL (2023) Augmented reality rehabilitative and exercise games (arregs): a systematic review and future considerations. IEEE, pp 1016\u20131025","DOI":"10.1109\/ISMAR59233.2023.00118"},{"key":"1305_CR34","doi-asserted-by":"publisher","first-page":"657","DOI":"10.1016\/j.sbspro.2013.10.385","volume":"103","author":"D Nincarean","year":"2013","unstructured":"Nincarean D, Alia MB, Halim NDA, Rahman MHA (2013) Mobile augmented reality: the potential for education. Proc Soc Behav Sci 103:657\u2013664","journal-title":"Proc Soc Behav Sci"},{"issue":"19","key":"1305_CR35","doi-asserted-by":"publisher","first-page":"2406","DOI":"10.3390\/electronics10192406","volume":"10","author":"Y Park","year":"2021","unstructured":"Park Y, Dang LM, Lee S, Han D, Moon H (2021) Multiple object tracking in deep learning approaches: a survey. Electronics 10(19):2406","journal-title":"Electronics"},{"key":"1305_CR36","doi-asserted-by":"crossref","unstructured":"Park Y, Lepetit V, Woo W (2008) Multiple 3d object tracking for augmented reality. In: 2008 7th IEEE\/ACM International symposium on mixed and augmented reality, pp. 117\u2013120. IEEE","DOI":"10.1109\/ISMAR.2008.4637336"},{"issue":"4","key":"1305_CR37","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1007\/s10055-018-0347-2","volume":"23","author":"N Pellas","year":"2019","unstructured":"Pellas N, Fotaris P, Kazanidis I, Wells D (2019) Augmenting the learning experience in primary and secondary school education: a systematic review of recent trends in augmented reality game-based learning. Virtual Real 23(4):329\u2013346","journal-title":"Virtual Real"},{"key":"1305_CR38","unstructured":"Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J, et al (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning, pp. 8748\u20138763. PMLR"},{"key":"1305_CR39","doi-asserted-by":"crossref","unstructured":"Radkowski R (2018) Hololens integration into a multi-kinect tracking environment. IEEE, pp 138\u2013142","DOI":"10.1109\/ISMAR-Adjunct.2018.00052"},{"key":"1305_CR40","doi-asserted-by":"crossref","unstructured":"Rambach J, Pagani A, Stricker, D.: [poster], (2017). augmented things: enhancing AR applications leveraging the internet of things and universal 3d object tracking. IEEE, pp 103\u2013108","DOI":"10.1109\/ISMAR-Adjunct.2017.42"},{"key":"1305_CR41","unstructured":"Ravi N, Reizenstein J, Novotny D, Gordon T, Lo W-Y, Johnson J, Gkioxari G (2020) Accelerating 3d deep learning with pytorch3d. arXiv:2007.08501"},{"key":"1305_CR42","unstructured":"Redmon J, Farhadi A (2018) Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767"},{"key":"1305_CR43","unstructured":"Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. Advances in neural information processing systems 28"},{"key":"1305_CR44","doi-asserted-by":"crossref","unstructured":"Ren H, Han S, Ding H, Zhang Z, Wang H, Wang F (2023) Focus on details: online multi-object tracking with diverse fine-grained representation. pp 11289\u201311298","DOI":"10.1109\/CVPR52729.2023.01086"},{"key":"1305_CR45","doi-asserted-by":"crossref","unstructured":"Roxas M, Hori T, Fukiage T, Okamoto Y, Oishi T (2018) Occlusion handling using semantic segmentation and visibility-based rendering for mixed reality. pp 1\u20138","DOI":"10.1145\/3281505.3281546"},{"key":"1305_CR46","doi-asserted-by":"crossref","unstructured":"Simeone AL, Velloso E, Gellersen H (2015) Substitutional reality: using the physical environment to design virtual reality experiences. pp 3307\u20133316","DOI":"10.1145\/2702123.2702389"},{"key":"1305_CR47","doi-asserted-by":"crossref","unstructured":"Simony M, Milzy S, Amendey K, Gross H-M (2018) Complex-yolo An Euler-region-proposal for real-time 3d object detection on point clouds","DOI":"10.1007\/978-3-030-11009-3_11"},{"issue":"1","key":"1305_CR48","doi-asserted-by":"publisher","first-page":"146","DOI":"10.3390\/s23010146","volume":"23","author":"TA Syed","year":"2022","unstructured":"Syed TA, Siddiqui MS, Abdullah HB, Jan S, Namoun A, Alzahrani A, Nadeem A, Alkhodre AB (2022) In-depth review of augmented reality: tracking technologies, development tools, AR displays, collaborative AR, and security concerns. Sensors 23(1):146","journal-title":"Sensors"},{"key":"1305_CR49","doi-asserted-by":"crossref","unstructured":"Tahara T, Seno T, Narita G, Ishikawa T, Retargetable AR (2020) Context-aware augmented reality in indoor scenes based on 3d scene graph. IEEE, pp 249\u2013255","DOI":"10.1109\/ISMAR-Adjunct51615.2020.00072"},{"issue":"9","key":"1305_CR50","doi-asserted-by":"publisher","first-page":"1435","DOI":"10.1007\/s11548-021-02432-y","volume":"16","author":"L Tanzi","year":"2021","unstructured":"Tanzi L, Piazzolla P, Porpiglia F, Vezzetti E (2021) Real-time deep learning semantic segmentation during intra-operative surgery for 3d augmented reality assistance. Int J Comput Assist Radiol Surg 16(9):1435\u20131445","journal-title":"Int J Comput Assist Radiol Surg"},{"key":"1305_CR51","doi-asserted-by":"crossref","unstructured":"Weerasinghe M, Quigley A, Ducasse J, \u010copi\u010d Pucihar K, Kljun M (2019). In: Geroimenko V (ed) Educational augmented reality games. Springer, Cham, pp 3\u201332","DOI":"10.1007\/978-3-030-15620-6_1"},{"key":"1305_CR52","doi-asserted-by":"crossref","unstructured":"Zhang H, Han B, Ip CY, Mohapatra, P.: Slimmer, (2020). In: Accelerating 3d semantic segmentation for mobile augmented reality. IEEE, pp 603\u2013612","DOI":"10.1109\/MASS50613.2020.00079"}],"container-title":["Virtual Reality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10055-025-01305-y","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10055-025-01305-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10055-025-01305-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T09:55:46Z","timestamp":1772272546000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10055-025-01305-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,2]]},"references-count":52,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,3]]}},"alternative-id":["1305"],"URL":"https:\/\/doi.org\/10.1007\/s10055-025-01305-y","relation":{},"ISSN":["1434-9957"],"issn-type":[{"value":"1434-9957","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,2]]},"assertion":[{"value":"23 August 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 December 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 February 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This work did not involve any human participants.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participants"}},{"value":"The authors declare no conflict of competing for financial interests or personal relationships that could have appeared to influence the work reported in this paper","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"51"}}