{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T11:18:47Z","timestamp":1773141527279,"version":"3.50.1"},"reference-count":40,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2022,12,6]],"date-time":"2022-12-06T00:00:00Z","timestamp":1670284800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Science and Technology Council, Taiwan","award":["MOST 110-2221-E-002-110-MY3"],"award-info":[{"award-number":["MOST 110-2221-E-002-110-MY3"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper presents the development of a visual-perception system on a dual-arm mobile robot for human-robot interaction. This visual system integrates three subsystems. Hand gesture recognition is utilized to trigger human-robot interaction. Engagement and intention of the participants are detected and quantified through a cognitive system. Visual servoing uses YOLO to identify the object to be tracked and hybrid, model-based tracking to follow the object\u2019s geometry. The proposed visual-perception system is implemented in the developed dual-arm mobile robot, and experiments are conducted to validate the proposed method\u2019s effects on human-robot interaction applications.<\/jats:p>","DOI":"10.3390\/s22239545","type":"journal-article","created":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T05:50:52Z","timestamp":1670392252000},"page":"9545","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Development of a Visual Perception System on a Dual-Arm Mobile Robot for Human-Robot Interaction"],"prefix":"10.3390","volume":"22","author":[{"given":"Wei-Ting","family":"Weng","sequence":"first","affiliation":[{"name":"Department of Mechanical Engineering, National Taiwan University, Taipei 10617, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Han-Pang","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, National Taiwan University, Taipei 10617, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0280-630X","authenticated-orcid":false,"given":"Yu-Lin","family":"Zhao","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, National Taiwan University, Taipei 10617, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4585-5881","authenticated-orcid":false,"given":"Chun-Yeon","family":"Lin","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, National Taiwan University, Taipei 10617, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,12,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Koceska, N., Koceski, S., Zobel, P.B., Trajkovik, V., and Garcia, N. (2019). A telemedicine robot system for assisted and independent living. Sensors, 19.","DOI":"10.3390\/s19040834"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1007\/s12369-018-0492-5","article-title":"A study on the deployment of a service robot in an elderly care center","volume":"11","author":"Portugal","year":"2019","journal-title":"Int. J. Soc. Robot."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Scholtz, J. (2003, January 6\u20139). Theory and Evaluation of Human Robot Interactions. Proceedings of the 36th Annual Hawaii International Conference on System Sciences, Big Island, HI, USA.","DOI":"10.1109\/HICSS.2003.1174284"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1007\/s10846-019-01028-8","article-title":"Toward a sociable and dependable elderly care robot: Design, implementation and user study","volume":"98","author":"Lee","year":"2020","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_5","first-page":"7195","article-title":"Implementation and user acceptance of social service robot for an elderly care program in a daycare facility","volume":"12","author":"Takanokura","year":"2021","journal-title":"J. Ambient Intell. Humaniz. Comput."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Frennert, S. (2020, January 19\u201324). Expectations and Sensemaking: Older People and Care Robots. Proceedings of the International Conference on Human-Computer Interaction, Copenhagen, Denmark.","DOI":"10.1007\/978-3-030-50232-4_14"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Kramer, R.K., Majidi, C., Sahai, R., and Wood, R.J. (2011, January 25\u201330). Soft Curvature Sensors for Joint Angle Proprioception. Proceedings of the 2011 IEEE\/RSJ International Conference on Intelligent Robots and Systems, San Francisco, CA, USA.","DOI":"10.1109\/IROS.2011.6094701"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"3631","DOI":"10.1109\/JSEN.2014.2330998","article-title":"Flexible optical fiber bending transducer for application in glove-based sensors","volume":"14","author":"Fujiwara","year":"2014","journal-title":"IEEE Sens. J."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Devineau, G., Moutarde, F., Xi, W., and Yang, J. (2018, January 15\u201319). Deep Learning for Hand Gesture Recognition on Skeletal Data. Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition, Xi\u2019an, China.","DOI":"10.1109\/FG.2018.00025"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"5809769","DOI":"10.1155\/2018\/5809769","article-title":"Kinect sensor-based long-distance hand gesture recognition and fingertip detection with depth information","volume":"2018","author":"Ma","year":"2018","journal-title":"J. Sens."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ahmed, S., Kallu, K.D., Ahmed, S., and Cho, S.H. (2021). Hand Gestures Recognition Using Radar Sensors for Human-Computer-Interaction: A Review. Remote Sens., 13.","DOI":"10.3390\/rs13030527"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1446","DOI":"10.1038\/s41467-022-29197-5","article-title":"Noncontact human-machine interaction based on hand-responsive infrared structural color","volume":"13","author":"An","year":"2022","journal-title":"Nat. Commun."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"238","DOI":"10.1109\/70.760345","article-title":"2 1\/2 D visual servoing","volume":"15","author":"Malis","year":"1999","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1109\/MRA.2006.250573","article-title":"Visual servo control. I. Basic approaches","volume":"13","author":"Chaumette","year":"2006","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1109\/MRA.2007.339609","article-title":"Visual servo control. II. Advanced approaches [Tutorial]","volume":"14","author":"Chaumette","year":"2007","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Sun, X.Y., Zhu, X.J., Wang, P.Y., and Chen, H. (2018, January 19\u201323). A Review of Robot Control with Visual Servoing. Proceedings of the 2018 IEEE 8th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems, Tianjin, China.","DOI":"10.1109\/CYBER.2018.8688060"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Claudio, G., Spindler, F., and Chaumette, F. (2017, January 2). Vision-Based Manipulation with the Humanoid Robot Romeo. Proceedings of the 2016 IEEE-RAS 16th International Conference on Humanoid Robots, Cancun, Mexico.","DOI":"10.1109\/HUMANOIDS.2016.7803290"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Fantacci, C., Vezzani, G., Pattacini, U., Tikhanoff, V., and Natale, L. (2018, January 21\u201325). Markerless Visual Servoing on Unknown Objects for Humanoid Robot Platforms. Proceedings of the 2018 IEEE International Conference on Robotics and Automation, Brisbane, QLD, Australia.","DOI":"10.1109\/ICRA.2018.8462914"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhang, Y.D., Wang, T.T., and Zhang, Y.M. (2019, January 8\u201312). Tracking with the CAD Model of Object for Visual Servoing. Proceedings of the 2019 IEEE\/ASME International Conference on Advanced Intelligent Mechatronics, Hong Kong, China.","DOI":"10.1109\/AIM.2019.8868837"},{"key":"ref_20","unstructured":"Sonntag, D. (2016, January 17\u201319). Persuasive AI Technologies for Healthcare Systems. Proceedings of the 2016 AAAI Fall Symposium Series, Arlington, VA, USA."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1726","DOI":"10.1016\/j.robot.2013.05.007","article-title":"Human-aware robot navigation: A survey","volume":"61","author":"Kruse","year":"2013","journal-title":"Robot. Auton. Syst."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kim, M., Oh, K., Choi, J., Jung, J., and Kim, Y. (2011). User-Centered HRI: HRI Research Methodology for Designers. Mixed Reality and Human-Robot Interaction, Springer.","DOI":"10.1007\/978-94-007-0582-1_2"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1007\/s12599-015-0389-x","article-title":"Service Robots","volume":"57","author":"Sprenger","year":"2015","journal-title":"Bus. Inf. Syst. Eng."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"525","DOI":"10.1007\/s11633-018-1115-1","article-title":"Current researches and future development trend of intelligent robot: A review","volume":"15","author":"Wang","year":"2018","journal-title":"Int. J. Autom. Comput."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1016\/j.archger.2011.02.003","article-title":"Designing robots for the elderly: Appearance issue and beyond","volume":"54","author":"Wu","year":"2012","journal-title":"Arch. Gerontol. Geriatr."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1016\/j.techfore.2014.09.005","article-title":"The emergence of care robotics\u2014A patent and publication analysis","volume":"92","author":"Goeldner","year":"2015","journal-title":"Technol. Forecast. Soc. Chang."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1007\/s12369-015-0328-5","article-title":"Increasing the acceptance of assistive robots for older people through marketing strategies based on stakeholder needs","volume":"8","author":"Glende","year":"2016","journal-title":"Int. J. Soc. Robot."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1007\/s12369-016-0384-5","article-title":"People\u2019s attitudes to robots in caring for the elderly","volume":"9","author":"Hudson","year":"2017","journal-title":"Int. J. Soc. Robot."},{"key":"ref_29","unstructured":"Can, W.S.R., and Seibt, S.D.J. (2016). Social Robotics, Elderly Care, and Human Dignity: A Recognition-Theoretical Approach. What Social Robots Can and Should Do: Proceedings of Robophilosophy 2016\/TRANSOR 2016, IOS Press."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., and Sheikh, Y. (2017, January 21\u201326). Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.143"},{"key":"ref_31","unstructured":"(2021, September 10). Openpose. Available online: https:\/\/github.com\/CMU-Perceptual-Computing-Lab\/openpose."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1016\/j.artint.2005.03.005","article-title":"Explorations in engagement for humans and robots","volume":"166","author":"Sidner","year":"2005","journal-title":"Artif. Intell."},{"key":"ref_33","unstructured":"Lu, S.-R., and Huang, H.P. (2020). Implementation of Pre-Engagement Detection on Human-Robot Interaction in Complex Environments. [Master\u2019s Thesis, National Taiwan University]."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Yang, T.-Y., Chen, Y.-T., Lin, Y.-Y., and Chuang, Y.-Y. (2019, January 15\u201320). FSA-Net: Learning Fine-Grained Structure Aggregation for Head Pose Estimation from a Single Image. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00118"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1016\/j.imavis.2017.01.010","article-title":"Going deeper into action recognition: A survey","volume":"60","author":"Herath","year":"2017","journal-title":"Image Vis. Comput."},{"key":"ref_36","unstructured":"Feichtenhofer, C., Fan, H., Malik, J., and He, K. (November, January 27). Slowfast Networks for Video Recognition. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_37","unstructured":"Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., and Natsev, P. (2017). The kinetics human action video dataset. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1177\/1354066115581059","article-title":"Emotions and the communication of intentions in face-to-face diplomacy","volume":"22","author":"Wong","year":"2016","journal-title":"Eur. J. Int. Relat."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., and Lee, D.-H. (2013, January 3\u20137). Challenges in Representation Learning: A Report on Three Machine Learning Contests. Proceedings of the International conference on neural information processing, Daegu, Republic of Korea.","DOI":"10.1007\/978-3-642-42051-1_16"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"651","DOI":"10.1109\/70.538972","article-title":"A tutorial on visual servo control","volume":"12","author":"Hutchinson","year":"1996","journal-title":"IEEE Trans. Robot. Autom."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/23\/9545\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:35:01Z","timestamp":1760146501000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/23\/9545"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,6]]},"references-count":40,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["s22239545"],"URL":"https:\/\/doi.org\/10.3390\/s22239545","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,6]]}}}