{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,24]],"date-time":"2025-12-24T12:18:36Z","timestamp":1766578716305,"version":"build-2065373602"},"reference-count":45,"publisher":"MDPI AG","issue":"24","license":[{"start":{"date-parts":[[2021,12,14]],"date-time":"2021-12-14T00:00:00Z","timestamp":1639440000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/P012779\/1","FAIR-SPACE EP\/R026092\/1"],"award-info":[{"award-number":["EP\/P012779\/1","FAIR-SPACE EP\/R026092\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000780","name":"European Commission","doi-asserted-by":"publisher","award":["FETOPEN829186","ICT 871803","ICT 871767","FETOPEN 899626"],"award-info":[{"award-number":["FETOPEN829186","ICT 871803","ICT 871767","FETOPEN 899626"]}],"id":[{"id":"10.13039\/501100000780","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Human operators have the trend of increasing physical and mental workloads when performing teleoperation tasks in uncertain and dynamic environments. In addition, their performances are influenced by subjective factors, potentially leading to operational errors or task failure. Although agent-based methods offer a promising solution to the above problems, the human experience and intelligence are necessary for teleoperation scenarios. In this paper, a truncated quantile critics reinforcement learning-based integrated framework is proposed for human\u2013agent teleoperation that encompasses training, assessment and agent-based arbitration. The proposed framework allows for an expert training agent, a bilateral training and cooperation process to realize the co-optimization of agent and human. It can provide efficient and quantifiable training feedback. Experiments have been conducted to train subjects with the developed algorithm. The performances of human\u2013human and human\u2013agent cooperation modes are also compared. The results have shown that subjects can complete the tasks of reaching and picking and placing with the assistance of an agent in a shorter operational time, with a higher success rate and less workload than human\u2013human cooperation.<\/jats:p>","DOI":"10.3390\/s21248341","type":"journal-article","created":{"date-parts":[[2021,12,14]],"date-time":"2021-12-14T22:06:10Z","timestamp":1639519570000},"page":"8341","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["A Novel Training and Collaboration Integrated Framework for Human\u2013Agent Teleoperation"],"prefix":"10.3390","volume":"21","author":[{"given":"Zebin","family":"Huang","sequence":"first","affiliation":[{"name":"Department of Bioengineering, Imperial College London, London SW7 2BX, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4588-8501","authenticated-orcid":false,"given":"Ziwei","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, Imperial College London, London SW7 2BX, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8937-8485","authenticated-orcid":false,"given":"Weibang","family":"Bai","sequence":"additional","affiliation":[{"name":"Department of Computing, Imperial College London, London SW7 2BX, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yanpei","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Bioengineering, Imperial College London, London SW7 2BX, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lichao","family":"Sun","sequence":"additional","affiliation":[{"name":"School of Education, Communication & Society, King\u2019s College London, London SE5 9RJ, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9361-4340","authenticated-orcid":false,"given":"Bo","family":"Xiao","sequence":"additional","affiliation":[{"name":"Department of Computing, Imperial College London, London SW7 2BX, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eric M.","family":"Yeatman","sequence":"additional","affiliation":[{"name":"Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2BX, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"103412","DOI":"10.1016\/j.robot.2019.103412","article-title":"Virtual-joint based motion similarity criteria for human-robot kinematics mapping","volume":"125","author":"Chen","year":"2020","journal-title":"Robot. Auton. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Gao, A., Murphy, R.R., Chen, W., Dagnino, G., Fischer, P., Gutierrez, M.G., Kundrat, D., Nelson, B.J., Shamsudhin, N., and Su, H. (2021). Progress in robotics for combating infectious diseases. Sci. Robot., 6.","DOI":"10.1126\/scirobotics.abf1462"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"4889","DOI":"10.1109\/TII.2019.2951329","article-title":"Adaptive Fault-Tolerant Prescribed-Time Control for Teleoperation Systems with Position Error Constraints","volume":"16","author":"Wang","year":"2020","journal-title":"IEEE Trans. Ind. Informatics"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Boukhnifer, M., and Ferreira, A. (2008, January 22\u201326). Fault tolerant control of a bilateral teleoperated micromanipulation system. Proceedings of the 2008 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Nice, France.","DOI":"10.1109\/IROS.2008.4650863"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zheng, L., and Yi, R. (2009, January 19\u201322). Fault diagnosis system for the inspection robot in power transmission lines maintenance. Proceedings of the 2009 International Conference on Optical Instruments and Technology: Optoelectronic Imaging and Process Technology, International Society for Optics and Photonics, Shanghai, China.","DOI":"10.1117\/12.837984"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Kettwich, C., Schrank, A., and Oehl, M. (2021). Teleoperation of Highly Automated Vehicles in Public Transport: User-Centered Design of a Human-Machine Interface for Remote-Operation and Its Expert Usability Evaluation. Multimodal Technol. Interact., 5.","DOI":"10.3390\/mti5050026"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1109\/LRA.2019.2948128","article-title":"Knowledge Driven Orbit-to-Ground Teleoperation of a Robot Coworker","volume":"5","author":"Schmaus","year":"2020","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1016\/j.actaastro.2018.12.018","article-title":"Fixed-time velocity reconstruction scheme for space teleoperation systems: Exp Barrier Lyapunov Function approach","volume":"157","author":"Wang","year":"2019","journal-title":"Acta Astronaut."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2785","DOI":"10.1109\/TFUZZ.2020.3007438","article-title":"Event-Triggered Prescribed-Time Fuzzy Control for Space Teleoperation Systems Subject to Multiple Constraints and Uncertainties","volume":"29","author":"Wang","year":"2021","journal-title":"IEEE Trans. Fuzzy Syst."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1080\/03091902.2020.1772391","article-title":"A review of haptic feedback in tele-operated robotic surgery","volume":"44","year":"2020","journal-title":"J. Med Eng. Technol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"499","DOI":"10.1108\/01439910410566362","article-title":"Methods for haptic feedback in teleoperated robot-assisted surgery","volume":"31","author":"Okamura","year":"2004","journal-title":"Ind. Robot. Int. J."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"2469","DOI":"10.1109\/LRA.2021.3062009","article-title":"Design and Evaluation of a Foot-Controlled Robotic System for Endoscopic Surgery","volume":"6","author":"Huang","year":"2021","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2282","DOI":"10.1007\/s10439-021-02766-3","article-title":"A Three-Limb Teleoperated Robotic System with Foot Control for Flexible Endoscopic Surgery","volume":"49","author":"Huang","year":"2021","journal-title":"Ann. Biomed. Eng."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1109\/MRA.2007.905502","article-title":"Field and service applications\u2014Exploring deep sea by teleoperated robot\u2014An Underwater Parallel Robot with High Navigation Capabilities","volume":"14","author":"Saltaren","year":"2007","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Sun, Y., Chen, X., Wang, Z., Qin, H., and Jing, R. (2021). Adaptive interval type-2 fuzzy control for multi-legged underwater robot with input saturation and full-state constraints. Int. J. Syst. Sci., 1\u201316.","DOI":"10.1080\/00207721.2020.1869346"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"338","DOI":"10.1109\/TOH.2018.2818134","article-title":"A Systematic Review of Multilateral Teleoperation Systems","volume":"11","author":"Shahbazi","year":"2018","journal-title":"IEEE Trans. Haptics"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wang, Z., Bai, W., Chen, Z., Xiao, B., Liang, B., and Yeatman, E. (2021, January 13\u201316). Multiple-Pilot Collaboration for Advanced RemoteIntervention using Reinforcement Learning. Proceedings of the 2021 47th Annual Conference of the IEEE Industrial Electronics Society (IECON), Toronto, ON, Canada.","DOI":"10.1109\/IECON48115.2021.9589570"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"3256","DOI":"10.1109\/LRA.2021.3062604","article-title":"A Framework for Customizable Multi-User Teleoperated Control","volume":"6","author":"Munawar","year":"2021","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"9846","DOI":"10.1109\/TIE.2019.2893864","article-title":"Cooperative Trajectory Planning for Haptic Shared Control Between Driver and Automation in Highway Driving","volume":"66","author":"Benloucif","year":"2019","journal-title":"IEEE Trans. Ind. Electron."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1962","DOI":"10.1109\/TCST.2018.2842211","article-title":"Driver-Automation Cooperation Oriented Approach for Shared Control of Lane Keeping Assist Systems","volume":"27","author":"Sentouh","year":"2019","journal-title":"IEEE Trans. Control. Syst. Technol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"3826","DOI":"10.1109\/TCYB.2020.2977374","article-title":"Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications","volume":"50","author":"Nguyen","year":"2020","journal-title":"IEEE Trans. Cybern."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Bhagat, S., Banerjee, H., Ho Tse, Z.T., and Ren, H. (2019). Deep Reinforcement Learning for Soft, Flexible Robots: Brief Review with Impending Challenges. Robotics, 8.","DOI":"10.3390\/robotics8010004"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"178450","DOI":"10.1109\/ACCESS.2020.3027923","article-title":"Review of Deep Reinforcement Learning-Based Object Grasping: Techniques, Open Challenges, and Recommendations","volume":"8","author":"Mohammed","year":"2020","journal-title":"IEEE Access"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Goldberg, K., and Abbeel, P. (2018, January 21\u201325). Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.","DOI":"10.1109\/ICRA.2018.8461249"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1146\/annurev-control-100819-063206","article-title":"Recent Advances in Robot Learning from Demonstration","volume":"3","author":"Ravichandar","year":"2020","journal-title":"Annu. Rev. Control. Robot. Auton. Syst."},{"key":"ref_26","unstructured":"Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., and Vanhoucke, V. (2018). QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. arXiv."},{"key":"ref_27","unstructured":"OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., and Powell, G. (2019). Solving Rubik\u2019s Cube with a Robot Hand. arXiv."},{"key":"ref_28","unstructured":"Leibo, J.Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel, T. (2017). Multi-Agent Reinforcement Learning in Sequential Social Dilemmas. arXiv."},{"key":"ref_29","unstructured":"Xie, A., Losey, D.P., Tolsma, R., Finn, C., and Sadigh, D. (2020). Learning Latent Representations to Influence Multi-Agent Interaction. arXiv."},{"key":"ref_30","unstructured":"Lanctot, M., Zambaldi, V., Gruslys, A., Lazaridou, A., Tuyls, K., Perolat, J., Silver, D., and Graepel, T. (2017). A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I., Korjus, K., Aru, J., Aru, J., and Vicente, R. (2017). Multiagent Cooperation and Competition with Deep Reinforcement Learning. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0172395"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"3612","DOI":"10.1109\/LRA.2020.2976272","article-title":"Safety Augmented Value Estimation From Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Tasks","volume":"5","author":"Thananjeyan","year":"2020","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_33","unstructured":"Zhang, T., Xu, H., Wang, X., Wu, Y., Keutzer, K., Gonzalez, J.E., and Tian, Y. (2020). Multi-Agent Collaboration via Reward Attribution Decomposition. arXiv."},{"key":"ref_34","unstructured":"Wang, C., P\u00e9rez-D\u2019Arpino, C., Xu, D., Fei-Fei, L., Liu, C.K., and Savarese, S. (2021). Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration. arXiv."},{"key":"ref_35","unstructured":"Lee, J. (2017). A Survey of Robot Learning from Demonstrations for Human-Robot Collaboration. arXiv."},{"key":"ref_36","unstructured":"Vogt, D., Stepputtis, S., Grehl, S., Jung, B., and Ben Amor, H. (June, January 29). A System for Learning Continuous Human-Robot Interactions from Human-Human Demonstrations. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Singapore."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"453","DOI":"10.21037\/atm.2016.12.24","article-title":"Innovations in surgery simulation: A review of past, current and future techniques","volume":"4","author":"Badash","year":"2016","journal-title":"Ann. Transl. Med."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"620","DOI":"10.3138\/jvme.34.5.620","article-title":"Teaching Surgery to the Veterinary Novice: The Ohio State University Experience","volume":"34","author":"Smeak","year":"2007","journal-title":"J. Vet. Med. Educ."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1109\/LRA.2019.2891311","article-title":"Robot-Assisted Training in Laparoscopy Using Deep Reinforcement Learning","volume":"4","author":"Tan","year":"2019","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_40","unstructured":"Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. (2018). Hindsight Experience Replay. arXiv."},{"key":"ref_41","unstructured":"Kuznetsov, A., Shvechikov, P., Grishin, A., and Vetrov, D. (2020, January 12\u201318). Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics. Proceedings of the International Conference on Machine Learning, PMLR, Vienna, Austria."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Johnen, B., and Kuhlenkoetter, B. (2016, January 16\u201318). A Dynamic Time Warping Algorithm for Industrial Robot Motion Analysis. Proceedings of the 2016 Annual Conference on Information Science and Systems (CISS), Princeton, NJ, USA.","DOI":"10.1109\/CISS.2016.7460470"},{"key":"ref_43","unstructured":"Coumans, E., and Bai, Y. (2021, November 01). PyBullet, a Python Module for Physics Simulation for Games, Robotics and Machine Learning 2016\u20132018. Available online: http:\/\/pybullet.org."},{"key":"ref_44","unstructured":"Raffin, A., Hill, A., Ernestus, M., Gleave, A., Kanervisto, A., and Dormann, N. (2021, November 01). Stable Baselines3. Available online: https:\/\/github.com\/DLR-RM\/stable-baselines3."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Jarrass\u00e9, N., Charalambous, T., and Burdet, E. (2012). A framework to describe, analyze and generate interactive motor behaviors. PLoS ONE, 7.","DOI":"10.1371\/journal.pone.0049945"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/24\/8341\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:47:23Z","timestamp":1760168843000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/24\/8341"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,14]]},"references-count":45,"journal-issue":{"issue":"24","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["s21248341"],"URL":"https:\/\/doi.org\/10.3390\/s21248341","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,12,14]]}}}