{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,18]],"date-time":"2026-01-18T21:41:02Z","timestamp":1768772462863,"version":"3.49.0"},"reference-count":39,"publisher":"MDPI AG","issue":"9","license":[{"start":{"date-parts":[[2023,4,27]],"date-time":"2023-04-27T00:00:00Z","timestamp":1682553600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Work Enhancement Based on Visual Scene Perception","award":["6142222200403"],"award-info":[{"award-number":["6142222200403"]}]},{"name":"Work Enhancement Based on Visual Scene Perception","award":["SYFD062003"],"award-info":[{"award-number":["SYFD062003"]}]},{"name":"National Key Laboratory Foundation of Human Factor Engineering","award":["6142222200403"],"award-info":[{"award-number":["6142222200403"]}]},{"name":"National Key Laboratory Foundation of Human Factor Engineering","award":["SYFD062003"],"award-info":[{"award-number":["SYFD062003"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Mixed reality (MR) registers virtual information and real objects and is an effective way to supplement astronaut training. Spatial anchors are generally used to perform virtual\u2013real fusion in static scenes but cannot handle movable objects. To address this issue, we propose a smart task assistance method based on object detection and point cloud alignment. Specifically, both fixed and movable objects are detected automatically. In parallel, poses are estimated with no dependence on preset spatial position information. Firstly, YOLOv5s is used to detect the object and segment the point cloud of the corresponding structure, called the partial point cloud. Then, an iterative closest point (ICP) algorithm between the partial point cloud and the template point cloud is used to calculate the object\u2019s pose and execute the virtual\u2013real fusion. The results demonstrate that the proposed method achieves automatic pose estimation for both fixed and movable objects without background information and preset spatial anchors. Most volunteers reported that our approach was practical, and it thus expands the application of astronaut training.<\/jats:p>","DOI":"10.3390\/s23094344","type":"journal-article","created":{"date-parts":[[2023,4,28]],"date-time":"2023-04-28T02:02:23Z","timestamp":1682647343000},"page":"4344","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Smart Task Assistance in Mixed Reality for Astronauts"],"prefix":"10.3390","volume":"23","author":[{"given":"Qingwei","family":"Sun","sequence":"first","affiliation":[{"name":"Department of Aerospace Science and Technology, Space Engineering University, Beijing 101416, China"},{"name":"China Astronaut Research and Training Center, Beijing 100094, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wei","family":"Chen","sequence":"additional","affiliation":[{"name":"China Astronaut Research and Training Center, Beijing 100094, China"},{"name":"National Key Laboratory of Human Factor Engineering, China Astronaut Research and Training Center, Beijing 100094, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiangang","family":"Chao","sequence":"additional","affiliation":[{"name":"China Astronaut Research and Training Center, Beijing 100094, China"},{"name":"National Key Laboratory of Human Factor Engineering, China Astronaut Research and Training Center, Beijing 100094, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wanhong","family":"Lin","sequence":"additional","affiliation":[{"name":"China Astronaut Research and Training Center, Beijing 100094, China"},{"name":"National Key Laboratory of Human Factor Engineering, China Astronaut Research and Training Center, Beijing 100094, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhenying","family":"Xu","sequence":"additional","affiliation":[{"name":"China Astronaut Research and Training Center, Beijing 100094, China"},{"name":"National Key Laboratory of Human Factor Engineering, China Astronaut Research and Training Center, Beijing 100094, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ruizhi","family":"Cao","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Beihang University, Beijing 100083, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,4,27]]},"reference":[{"key":"ref_1","first-page":"26","article-title":"Research on Scene Understanding Method in Mixed Reality for Astronaut Training","volume":"26","author":"Chang","year":"2020","journal-title":"Manned Spacefl."},{"key":"ref_2","first-page":"72","article-title":"3D Semantic Reconstruction of Spacecraft Cabin Structures","volume":"27","author":"Qingwei","year":"2021","journal-title":"Manned Spacefl."},{"key":"ref_3","unstructured":"(2023, April 23). Vuforia Chalk. Available online: https:\/\/chalk.vuforia.com."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.vrih.2019.12.002","article-title":"A smart assistance system for cable assembly by combining wearable augmented reality with portable visual inspection","volume":"2","author":"Zheng","year":"2020","journal-title":"Virtual Real. Intell. Hardw."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"657","DOI":"10.1016\/j.cirp.2012.05.010","article-title":"Augmented reality applications in design and manufacturing","volume":"61","author":"Nee","year":"2012","journal-title":"CIRP Ann."},{"key":"ref_6","unstructured":"Microsoft (2022, May 09). Azure Spatial Anchors. Available online: https:\/\/learn.microsoft.com\/en-us\/azure\/spatial-anchors\/."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1007\/s00170-009-2012-0","article-title":"AR\/RP-based tangible interactions for collaborative design evaluation of digital products","volume":"45","author":"Lee","year":"2009","journal-title":"J. Adv. Manuf. Technol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"106195","DOI":"10.1016\/j.cie.2019.106195","article-title":"Augmented reality in support of intelligent manufacturing\u2014A systematic literature review","volume":"140","author":"Egger","year":"2020","journal-title":"Comput Ind. Eng."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23\u201328). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.81"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1904","DOI":"10.1109\/TPAMI.2015.2389824","article-title":"Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition","volume":"37","author":"He","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Girshick, R. (2015, January 7\u201313). Fast R-CNN. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks","volume":"39","author":"Ren","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017, January 22\u201329). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Redmon, J., and Farhadi, A. (2017, January 21\u201326). YOLO9000: Better, Faster, Stronger. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.690"},{"key":"ref_16","unstructured":"Redmon, J., and Farhadi, A. (2018). YOLOv3: An Incremental Improvement. arXiv."},{"key":"ref_17","unstructured":"Bochkovskiy, A., Wang, C.-Y., and Liao, H.-Y.M. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A.C. (2016, January 11\u201314). SSD: Single Shot MultiBox Detector. Proceedings of the 2016 European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46448-0_2"},{"key":"ref_19","unstructured":"Ultralytics (2022, March 10). Yolov5. Available online: https:\/\/github.com\/ultralytics\/yolov5."},{"key":"ref_20","unstructured":"Leibe, B., Matas, J., Sebe, N., and Welling, M. (2018, January 18\u201323). Real-Time Seamless Single Shot 6D Object Pose Prediction. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Rad, M., and Lepetit, V. (2017, January 22\u201329). BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.413"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Rad, M., Oberweger, M., and Lepetit, V. Feature Mapping for Learning Fast and Accurate 3D Pose Inference from Synthetic Images. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00490"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Rusu, R.B., Blodow, N., and Beetz, M. (2009, January 12\u201317). Fast Point Feature Histograms (FPFH) for 3D registration. Proceedings of the 2009 IEEE International Conference on Robotics and Automation (ICRA), Kobe, Japan.","DOI":"10.1109\/ROBOT.2009.5152473"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"431","DOI":"10.1016\/j.isprsjprs.2017.06.012","article-title":"A novel binary shape context for 3D local surface description","volume":"130","author":"Dong","year":"2017","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Hoda\u0148, T., Zabulis, X., Lourakis, M., Obdr\u017e\u00e1lek, \u0160., and Matas, J. (October, January 28). Detection and fine 3D pose estimation of texture-less objects in RGB-D images. Proceedings of the 2015 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany.","DOI":"10.1109\/IROS.2015.7354005"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., and Navab, N. (2012, January 5\u20139). Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. Proceedings of the 11th Asian Conference on Computer Vision (ACCV), Daejeon, Korea.","DOI":"10.1007\/978-3-642-33885-4_60"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Aoki, Y., Goforth, H., Srivatsan, R.A., and Lucey, S. (2019, January 15\u201320). PointNetLK: Robust & Efficient Point Cloud Registration Using PointNet. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00733"},{"key":"ref_28","unstructured":"Sarode, V., Li, X., Goforth, H., Aoki, Y., Srivatsan, R.A., Lucey, S., and Choset, H. (2019). PCRNet: Point cloud registration network using PointNet encoding. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3212","DOI":"10.1109\/TPAMI.2020.3047388","article-title":"PVNet: Pixel-Wise Voting Network for 6DoF Object Pose Estimation","volume":"44","author":"Peng","year":"2022","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"He, Y., Sun, W., Huang, H., Liu, J., Fan, H., and Sun, J. (2020, January 14\u201319). PVN3D: A deep point-wise 3D keypoints voting network for 6DoF pose estimation. Proceedings of the 2020 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01165"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Wang, C., Mart\u00edn-Mart\u00edn, R., Xu, D., Lv, J., Lu, C., Fei-Fei, L., Savarese, S., and Zhu, Y. (August, January 31). 6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.","DOI":"10.1109\/ICRA40945.2020.9196679"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"101887","DOI":"10.1016\/j.rcim.2019.101887","article-title":"Deep learning-based smart task assistance in wearable augmented reality","volume":"63","author":"Park","year":"2020","journal-title":"Robot Comput. Integr. Manuf."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"106585","DOI":"10.1016\/j.cie.2020.106585","article-title":"Deep learning-based mobile augmented reality for task assistance using 3D spatial mapping and snapshot-based RGB-D data","volume":"146","author":"Park","year":"2020","journal-title":"Comput Ind. Eng."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1109\/34.121791","article-title":"A method for registration of 3-D shapes","volume":"14","author":"Besl","year":"1992","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_35","unstructured":"Ungureanu, D., Bogo, F., Galliani, S., Sama, P., Duan, X., Meekhof, C., St\u00fchmer, J., Cashman, T.J., Tekin, B., and Sch\u00f6nberger, J.L. (2020). HoloLens 2 Research Mode as a Tool for Computer Vision Research. arXiv."},{"key":"ref_36","unstructured":"Eberly, D. (2022, July 13). Conversion of Left\u2013Handed Coordinates to Right\u2013Handed Coordinates. Available online: https:\/\/www.docin.com\/p-763518275.html."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_38","unstructured":"Chen, Y., and Medioni, G. (1991, January 9\u201311). Object modeling by registration of multiple range images. Proceedings of the 1991 IEEE International Conference on Robotics and Automation (ICRA), Sacramento, CA, USA."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zhou, Q.-Y., Park, J., and Koltun, V. (2016, January 11\u201314). Fast Global Registration. Proceedings of the 2016 European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-319-46475-6_47"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/9\/4344\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:24:56Z","timestamp":1760124296000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/9\/4344"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,27]]},"references-count":39,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["s23094344"],"URL":"https:\/\/doi.org\/10.3390\/s23094344","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,27]]}}}