{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,5]],"date-time":"2025-12-05T12:18:38Z","timestamp":1764937118515,"version":"3.41.0"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2019,9,9]],"date-time":"2019-09-09T00:00:00Z","timestamp":1567987200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Shenzhen Science and Technology Program","award":["No. JCYJ20170413162617606"],"award-info":[{"award-number":["No. JCYJ20170413162617606"]}]},{"name":"SoC robotics initiative","award":["C-251-000-065-001"],"award-info":[{"award-number":["C-251-000-065-001"]}]},{"name":"Research Grants Council of Hong Kong Special Administrative Region","award":["Project No. CUHK14225616, Project No. CUHK14201918"],"award-info":[{"award-number":["Project No. CUHK14225616, Project No. CUHK14201918"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."],"published-print":{"date-parts":[[2019,9,9]]},"abstract":"<jats:p>Augmented virtual reality (AVR) takes portions of the physical world into the VR world to enable VR users to access physical objects. State-of-the-art solutions mainly focus on extracting and showing physical objects in the VR world. In this work, we go beyond previous solutions and propose a novel approach to realize AVR. We first analyze the physical environment in the user's egocentric view through depth sensing and deep learning, then acquire the layout and geometry of the surrounding objects, and further explore their affordances. Based on the above information, we create visual guidance (hollowed guiding path) and hybrid user interfaces (augmented physical notepad, LR finger slider, and LRRL finger slider) to augment the AVR interaction. Empirical evaluations showed that the participants responded positively to our AVR techniques.<\/jats:p>","DOI":"10.1145\/3351263","type":"journal-article","created":{"date-parts":[[2019,9,10]],"date-time":"2019-09-10T15:58:26Z","timestamp":1568131106000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Enhancing Augmented VR Interaction via Egocentric Scene Analysis"],"prefix":"10.1145","volume":"3","author":[{"given":"Yang","family":"Tian","sequence":"first","affiliation":[{"name":"The Chinese University of Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chi-Wing","family":"Fu","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, and Key Laboratory of Computer Vision and Virtual Reality Technology, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shengdong","family":"Zhao","sequence":"additional","affiliation":[{"name":"National University of Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ruihui","family":"Li","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiao","family":"Tang","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaowei","family":"Hu","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pheng-Ann","family":"Heng","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,9,9]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858226"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/CW.2009.39"},{"key":"e_1_2_2_3_1","volume-title":"Proc. of Virtual Reality International Conference (VRIC","author":"Bruder Gerd","year":"2010","unstructured":"Gerd Bruder , Frank Steinicke , Dimitar Valkov , and Klaus Hinrichs . 2010 . Augmented virtual studio for architectural exploration . In Proc. of Virtual Reality International Conference (VRIC 2010). 43--50. Gerd Bruder, Frank Steinicke, Dimitar Valkov, and Klaus Hinrichs. 2010. Augmented virtual studio for architectural exploration. In Proc. of Virtual Reality International Conference (VRIC 2010). 43--50."},{"key":"e_1_2_2_4_1","volume-title":"Where's My Drink? Enabling Peripheral Real World Interactions While Using HMDs. arXiv preprint arXiv:1502.04744","author":"Budhiraja Pulkit","year":"2015","unstructured":"Pulkit Budhiraja , Rajinder Sodhi , Brett Jones , Kevin Karsch , Brian Bailey , and David Forsyth . 2015. Where's My Drink? Enabling Peripheral Real World Interactions While Using HMDs. arXiv preprint arXiv:1502.04744 ( 2015 ). Pulkit Budhiraja, Rajinder Sodhi, Brett Jones, Kevin Karsch, Brian Bailey, and David Forsyth. 2015. Where's My Drink? Enabling Peripheral Real World Interactions While Using HMDs. arXiv preprint arXiv:1502.04744 (2015)."},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173663"},{"key":"e_1_2_2_6_1","volume-title":"Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON). 1--4.","author":"Clergeaud D.","year":"2017","unstructured":"D. Clergeaud and P. Guitton . 2017 . Design of an annotation system for taking notes in virtual reality. In 2017 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON). 1--4. D. Clergeaud and P. Guitton. 2017. Design of an annotation system for taking notes in virtual reality. In 2017 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON). 1--4."},{"volume-title":"2017 14th Conference on Computer and Robot Vision (CRV). 217--224","author":"Desai A. P.","key":"e_1_2_2_7_1","unstructured":"A. P. Desai , L. Pe\u00f1a-Castillo , and O. Meruvia-Pastor . 2017. A Window to Your Smartphone: Exploring Interaction and Communication in Immersive VR with Augmented Virtuality . In 2017 14th Conference on Computer and Robot Vision (CRV). 217--224 . A. P. Desai, L. Pe\u00f1a-Castillo, and O. Meruvia-Pastor. 2017. A Window to Your Smartphone: Exploring Interaction and Communication in Immersive VR with Augmented Virtuality. In 2017 14th Conference on Computer and Robot Vision (CRV). 217--224."},{"key":"e_1_2_2_8_1","volume-title":"https:\/\/www.cdc.gov\/nchs\/data\/series\/sr_11\/sr11_249.pdf {Online","author":"Data","year":"2019","unstructured":"Data from centers for disease control and prevention. 2019. https:\/\/www.cdc.gov\/nchs\/data\/series\/sr_11\/sr11_249.pdf {Online ; accessed 27- March - 2019 }. Data from centers for disease control and prevention. 2019. https:\/\/www.cdc.gov\/nchs\/data\/series\/sr_11\/sr11_249.pdf {Online; accessed 27-March-2019}."},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3275495.3275515"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2999508.2999531"},{"key":"e_1_2_2_11_1","volume-title":"International Conference on Artificial Reality and Telexistence. 87--95","author":"Gao Lei","year":"2017","unstructured":"Lei Gao , Huidong Bai , Thammathip Piumsomboon , G Lee , Robert W Lindeman , and Mark Billinghurst . 2017 . Real-time visual representations for mixed reality remote collaboration . In International Conference on Artificial Reality and Telexistence. 87--95 . Lei Gao, Huidong Bai, Thammathip Piumsomboon, G Lee, Robert W Lindeman, and Mark Billinghurst. 2017. Real-time visual representations for mixed reality remote collaboration. In International Conference on Artificial Reality and Telexistence. 87--95."},{"key":"e_1_2_2_12_1","doi-asserted-by":"crossref","unstructured":"J. Garcia Estrada and A. L. Simeone. 2017. Recommender system for physical object substitution in VR. In 2017 IEEE Virtual Reality (VR). 359--360.  J. Garcia Estrada and A. L. Simeone. 2017. Recommender system for physical object substitution in VR. In 2017 IEEE Virtual Reality (VR). 359--360.","DOI":"10.1109\/VR.2017.7892325"},{"key":"e_1_2_2_13_1","volume-title":"http:\/\/global.xfyun.cn\/ {Online","author":"FLYTEK","year":"2019","unstructured":"i FLYTEK voice recognition SDK. 2019. http:\/\/global.xfyun.cn\/ {Online ; accessed 25- February - 2019 }. iFLYTEK voice recognition SDK. 2019. http:\/\/global.xfyun.cn\/ {Online; accessed 25-February-2019}."},{"key":"e_1_2_2_14_1","volume-title":"International Conference on Artificial Reality and Telexistence. 63--70","author":"Khan Humayun","year":"2017","unstructured":"Humayun Khan , Gun Lee , Simon Hoermann , Rory Clifford , Mark Billinghurst , and Robert W Lindeman . 2017 . Evaluating the Effects of Hand-gesture-based Interaction with Virtual Content in a 360 Movie . In International Conference on Artificial Reality and Telexistence. 63--70 . Humayun Khan, Gun Lee, Simon Hoermann, Rory Clifford, Mark Billinghurst, and Robert W Lindeman. 2017. Evaluating the Effects of Hand-gesture-based Interaction with Virtual Content in a 360 Movie. In International Conference on Artificial Reality and Telexistence. 63--70."},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2993369.2996330"},{"volume-title":"Enhancing Immersive Cinematic Experience with Augmented Virtuality. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). 115--116","author":"Lee G. A.","key":"e_1_2_2_16_1","unstructured":"G. A. Lee , J. Chen , M. Billinghurst , and R. Lindeman . 2016 . Enhancing Immersive Cinematic Experience with Augmented Virtuality. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). 115--116 . G. A. Lee, J. Chen, M. Billinghurst, and R. Lindeman. 2016. Enhancing Immersive Cinematic Experience with Augmented Virtuality. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). 115--116."},{"key":"e_1_2_2_17_1","volume-title":"https:\/\/pointclouds.org\/ {Online","author":"Library Point Cloud","year":"2019","unstructured":"Point Cloud Library . 2019. https:\/\/pointclouds.org\/ {Online ; accessed 27- April - 2019 }. Point Cloud Library. 2019. https:\/\/pointclouds.org\/ {Online; accessed 27-April-2019}."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3102163.3102175"},{"key":"e_1_2_2_19_1","volume-title":"https:\/\/docs.opencv.org\/3.1.0\/d5\/dae\/tutorial_aruco_detection.html {Online","author":"Marker ARUCO","year":"2019","unstructured":"ARUCO Marker . 2019. https:\/\/docs.opencv.org\/3.1.0\/d5\/dae\/tutorial_aruco_detection.html {Online ; accessed 27- April - 2019 }. ARUCO Marker. 2019. https:\/\/docs.opencv.org\/3.1.0\/d5\/dae\/tutorial_aruco_detection.html {Online; accessed 27-April-2019}."},{"key":"e_1_2_2_20_1","volume-title":"RULA: A Survey Method for the Investigation of Work-Related Upper Limb Disorders. Applied ergonomics 24 (05","author":"Mcatamney Lynn","year":"1993","unstructured":"Lynn Mcatamney and E. Nigel Corlett . 1993 . RULA: A Survey Method for the Investigation of Work-Related Upper Limb Disorders. Applied ergonomics 24 (05 1993), 91--99. Lynn Mcatamney and E. Nigel Corlett. 1993. RULA: A Survey Method for the Investigation of Work-Related Upper Limb Disorders. Applied ergonomics 24 (05 1993), 91--99."},{"volume-title":"2017 IEEE International Conference on Robotics and Automation (ICRA). 4628--4635","author":"McCormac J.","key":"e_1_2_2_21_1","unstructured":"J. McCormac , A. Handa , A. Davison , and S. Leutenegger . 2017. SemanticFusion: Dense 3D semantic mapping with convolutional neural networks . In 2017 IEEE International Conference on Robotics and Automation (ICRA). 4628--4635 . J. McCormac, A. Handa, A. Davison, and S. Leutenegger. 2017. SemanticFusion: Dense 3D semantic mapping with convolutional neural networks. In 2017 IEEE International Conference on Robotics and Automation (ICRA). 4628--4635."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702382"},{"key":"e_1_2_2_23_1","volume":"201","author":"Mendes D.","unstructured":"D. Mendes , F. Fonseca , B. Ara\u00f9jo , A. Ferreira , and J. Jorge. 201 4. Mid-air interactions above stereoscopic interactive tables. In 2014 IEEE Symposium on 3D User Interfaces (3DUI). 3--10. D. Mendes, F. Fonseca, B. Ara\u00f9jo, A. Ferreira, and J. Jorge. 2014. Mid-air interactions above stereoscopic interactive tables. In 2014 IEEE Symposium on 3D User Interfaces (3DUI). 3--10.","journal-title":"J. Jorge."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/VRAIS.1993.380805"},{"key":"e_1_2_2_25_1","volume-title":"A taxonomy of real and virtual world display integration. Mixed reality: Merging real and virtual worlds 1","author":"Milgram Paul","year":"1999","unstructured":"Paul Milgram and Herman Colquhoun . 1999. A taxonomy of real and virtual world display integration. Mixed reality: Merging real and virtual worlds 1 ( 1999 ), 1--26. Paul Milgram and Herman Colquhoun. 1999. A taxonomy of real and virtual world display integration. Mixed reality: Merging real and virtual worlds 1 (1999), 1--26."},{"key":"e_1_2_2_26_1","doi-asserted-by":"crossref","unstructured":"D. Nahon G. Subileau and B. Capel. 2015. \"Never Blind VR\" enhancing the virtual reality headset experience with augmented virtuality. In 2015 IEEE Virtual Reality (VR). 347--348.  D. Nahon G. Subileau and B. Capel. 2015. \"Never Blind VR\" enhancing the virtual reality headset experience with augmented virtuality. In 2015 IEEE Virtual Reality (VR). 347--348.","DOI":"10.1109\/VR.2015.7223438"},{"key":"e_1_2_2_27_1","unstructured":"Don Norman. 2013. The design of everyday things: Revised and expanded edition. Basic books.   Don Norman. 2013. The design of everyday things: Revised and expanded edition. Basic books."},{"volume-title":"Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180)","author":"Poupyrev I.","key":"e_1_2_2_28_1","unstructured":"I. Poupyrev , N. Tomokazu , and S. Weghorst . 1998. Virtual Notepad: handwriting in immersive VR . In Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180) . 126--132. I. Poupyrev, N. Tomokazu, and S. Weghorst. 1998. Virtual Notepad: handwriting in immersive VR. In Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180). 126--132."},{"volume-title":"Stronger. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 6517--6525","author":"Redmon J.","key":"e_1_2_2_29_1","unstructured":"J. Redmon and A. Farhadi . 2017. YOLO9000: Better, Faster , Stronger. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 6517--6525 . J. Redmon and A. Farhadi. 2017. YOLO9000: Better, Faster, Stronger. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 6517--6525."},{"key":"e_1_2_2_30_1","volume-title":"https:\/\/software.intel.com\/en-us\/realsense-sdk-windows-eol {Online","author":"Intel","year":"2019","unstructured":"Intel RealSense SDK. 2019. https:\/\/software.intel.com\/en-us\/realsense-sdk-windows-eol {Online ; accessed 27- April - 2019 }. Intel RealSense SDK. 2019. https:\/\/software.intel.com\/en-us\/realsense-sdk-windows-eol {Online; accessed 27-April-2019}."},{"key":"e_1_2_2_31_1","volume-title":"https:\/\/github.com\/ValveSoftware\/openvr {Online","author":"VR SDK.","year":"2019","unstructured":"Open VR SDK. 2019. https:\/\/github.com\/ValveSoftware\/openvr {Online ; accessed 27- April - 2019 }. OpenVR SDK. 2019. https:\/\/github.com\/ValveSoftware\/openvr {Online; accessed 27-April-2019}."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702389"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025566"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2993369.2993372"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/3DUI.2009.4811218"},{"volume-title":"2011 IEEE International Symposium on VR Innovation. 349--350","author":"Suma E. A.","key":"e_1_2_2_36_1","unstructured":"E. A. Suma , D. M. Krum , and M. Bolas . 2011. Sharing space in mixed and virtual reality environments using a low-cost depth sensor . In 2011 IEEE International Symposium on VR Innovation. 349--350 . E. A. Suma, D. M. Krum, and M. Bolas. 2011. Sharing space in mixed and virtual reality environments using a low-cost depth sensor. In 2011 IEEE International Symposium on VR Innovation. 349--350."},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2671015.2671123"},{"key":"e_1_2_2_38_1","volume-title":"https:\/\/github.com\/IntelRealSense\/librealsense\/tree\/master\/wrappers\/opencv\/latency-tool {Online","author":"Latency","year":"2019","unstructured":"Latency tool from the librealsense SDK. 2019. https:\/\/github.com\/IntelRealSense\/librealsense\/tree\/master\/wrappers\/opencv\/latency-tool {Online ; accessed 27- April - 2019 }. Latency tool from the librealsense SDK. 2019. https:\/\/github.com\/IntelRealSense\/librealsense\/tree\/master\/wrappers\/opencv\/latency-tool {Online; accessed 27-April-2019}."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025783"}],"container-title":["Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3351263","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3351263","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:25:51Z","timestamp":1750206351000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3351263"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,9]]},"references-count":39,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,9,9]]}},"alternative-id":["10.1145\/3351263"],"URL":"https:\/\/doi.org\/10.1145\/3351263","relation":{},"ISSN":["2474-9567"],"issn-type":[{"type":"electronic","value":"2474-9567"}],"subject":[],"published":{"date-parts":[[2019,9,9]]},"assertion":[{"value":"2019-09-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}