{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T16:05:35Z","timestamp":1783526735804,"version":"3.55.0"},"reference-count":57,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2022,6,3]],"date-time":"2022-06-03T00:00:00Z","timestamp":1654214400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Natural Science Foundation of China","award":["51875138"],"award-info":[{"award-number":["51875138"]}]},{"name":"National Natural Science Foundation of China","award":["201506120130"],"award-info":[{"award-number":["201506120130"]}]},{"name":"China Scholarship Council (CSC)","award":["51875138"],"award-info":[{"award-number":["51875138"]}]},{"name":"China Scholarship Council (CSC)","award":["201506120130"],"award-info":[{"award-number":["201506120130"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Most robots are programmed to carry out specific tasks routinely with minor variations. However, more and more applications from SMEs require robots work alongside their counterpart human workers. To smooth the collaboration task flow and improve the collaboration efficiency, a better way is to formulate the robot to surmise what kind of assistance a human coworker needs and naturally take the right action at the right time. This paper proposes a prediction-based human-robot collaboration model for assembly scenarios. An embedded learning from demonstration technique enables the robot to understand various task descriptions and customized working preferences. A state-enhanced convolutional long short-term memory (ConvLSTM)-based framework is formulated for extracting the high-level spatiotemporal features from the shared workspace and predicting the future actions to facilitate the fluent task transition. This model allows the robot to adapt itself to predicted human actions and enables proactive assistance during collaboration. We applied our model to the seats assembly experiment for a scale model vehicle and it can obtain a human worker\u2019s intentions, predict a coworker\u2019s future actions, and provide assembly parts correspondingly. It has been verified that the proposed framework yields higher smoothness and shorter idle times, and meets more working styles, compared to the state-of-the-art methods without prediction awareness.<\/jats:p>","DOI":"10.3390\/s22114279","type":"journal-article","created":{"date-parts":[[2022,6,3]],"date-time":"2022-06-03T10:33:01Z","timestamp":1654252381000},"page":"4279","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["Prediction-Based Human-Robot Collaboration in Assembly Tasks Using a Learning from Demonstration Model"],"prefix":"10.3390","volume":"22","author":[{"given":"Zhujun","family":"Zhang","sequence":"first","affiliation":[{"name":"State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150001, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8543-9455","authenticated-orcid":false,"given":"Gaoliang","family":"Peng","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150001, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weitian","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Automotive Engineering, Clemson University, Greenville, SC 29607, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yi","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Automotive Engineering, Clemson University, Greenville, SC 29607, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yunyi","family":"Jia","sequence":"additional","affiliation":[{"name":"Department of Automotive Engineering, Clemson University, Greenville, SC 29607, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1810-5412","authenticated-orcid":false,"given":"Shaohui","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,6,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"104667","DOI":"10.1016\/j.ssci.2020.104667","article-title":"Safety bounds in human robot interaction: A survey","volume":"127","author":"Zacharaki","year":"2020","journal-title":"Saf. Sci."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"688","DOI":"10.1016\/j.procs.2019.09.104","article-title":"Industry 5.0 and human-robot co-working","volume":"158","author":"Demir","year":"2019","journal-title":"Procedia Comput. Sci."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"112067","DOI":"10.1016\/j.sna.2020.112067","article-title":"Design of a linear walking stage based on two types of piezoelectric actuators","volume":"332","author":"Cai","year":"2021","journal-title":"Sens. Actuators A Phys."},{"key":"ref_4","unstructured":"Gurgul, M. (2018). Industrial Robots and Cobots: Everything You Need to Know about Your Future Co-Worker, Micha\u0142 Gurgul."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"107567","DOI":"10.1016\/j.ijpe.2019.107567","article-title":"Sustainable development in Asian manufacturing SMEs: Progress and directions","volume":"225","author":"Ndubisi","year":"2020","journal-title":"Int. J. Prod. Econ."},{"key":"ref_6","first-page":"100107","article-title":"Adoption of digital technologies of smart manufacturing in SMEs","volume":"16","author":"Ghobakhloo","year":"2019","journal-title":"J. Ind. Inf. Integr."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zawadzki, P., \u017bywicki, K., Grajewski, D., and G\u00f3rski, F. (2018, January 17\u201318). Efficiency of Automatic Design in the Production Preparation Process for an Intelligent Factory. Proceedings of the International Conference on Intelligent Systems in Production Engineering and Maintenance, Wroclaw, Poland.","DOI":"10.1007\/978-3-319-97490-3_52"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1016\/j.cogr.2021.06.001","article-title":"Substantial capabilities of robotics in enhancing industry 4.0 implementation","volume":"1","author":"Javaid","year":"2021","journal-title":"Cogn. Robot."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"26754","DOI":"10.1109\/ACCESS.2017.2773127","article-title":"Working together: A review on safe human-robot collaboration in industrial environments","volume":"5","author":"Becerra","year":"2017","journal-title":"IEEE Access"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1142\/9789811222849_0004","article-title":"Human-robot collaboration for advanced manufacturing by learning from multi-modal human demonstrations","volume":"Volume 3","author":"Wang","year":"2021","journal-title":"Manufacturing In The Era of 4th Industrial Revolution: A World Scientific Reference"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1109\/THMS.2018.2883176","article-title":"Controlling Object Hand-Over in Human-Robot Collaboration via Natural Wearable Sensing","volume":"49","author":"Wang","year":"2018","journal-title":"IEEE Trans. Hum.-Mach. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Matheson, E., Minto, R., Zampieri, E.G.G., Faccio, M., and Rosati, G. (2019). Human\u2013robot collaboration in manufacturing applications: A review. Robotics, 8.","DOI":"10.3390\/robotics8040100"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Zhu, Z., and Hu, H. (2018). Robot learning from demonstration in robotic assembly: A survey. Robotics, 7.","DOI":"10.3390\/robotics7020017"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1146\/annurev-control-100819-063206","article-title":"Recent advances in robot learning from demonstration","volume":"3","author":"Ravichandar","year":"2020","journal-title":"Annu. Rev. Control. Robot. Auton. Syst."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"3033","DOI":"10.1109\/TCYB.2019.2905157","article-title":"Making sense of spatio-temporal preserving representations for EEG-based human intention recognition","volume":"50","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Cybern."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Chen, Y., Wang, W., Krovi, V., and Jia, Y. (2020\u201324, January 24). Enabling Robot to Assist Human in Collaborative Assembly using Convolutional Neural Networks. Proceedings of the 2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.","DOI":"10.1109\/IROS45743.2020.9340735"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"687031","DOI":"10.3389\/frobt.2021.687031","article-title":"A bayesian deep neural network for safe visual servoing in human\u2013robot interaction","volume":"8","author":"Shi","year":"2021","journal-title":"Front. Robot. AI"},{"key":"ref_18","first-page":"41","article-title":"Application of dynamically scaled safety zones based on the ISO\/TS 15066: 2016 for collaborative robotics","volume":"21","author":"Scalera","year":"2020","journal-title":"Int. J. Mech. Control"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Himmelsbach, U.B., Wendt, T.M., Hangst, N., Gawron, P., and Stiglmeier, L. (2021). Human\u2013Machine Differentiation in Speed and Separation Monitoring for Improved Efficiency in Human\u2013Robot Collaboration. Sensors, 21.","DOI":"10.3390\/s21217144"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Wang, W., Chen, Y., Jia, Y., and Peng, G. (2019). Prediction of Human Actions in Assembly Process by a Spatial-Temporal End-to-End Learning Model, SAE International.","DOI":"10.4271\/2019-01-0509"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Mainprice, J., and Berenson, D. (2013, January 3\u20137). Human-robot collaborative manipulation planning using early prediction of human motion. Proceedings of the 2013 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan.","DOI":"10.1109\/IROS.2013.6696368"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Ragaglia, M., Zanchettin, A.M., and Rocco, P. (2015, January 27\u201331). Safety-aware trajectory scaling for human-robot collaboration with prediction of human occupancy. Proceedings of the 2015 International Conference on Advanced Robotics (ICAR), Istanbul, Turkey.","DOI":"10.1109\/ICAR.2015.7251438"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"609","DOI":"10.1007\/s40684-020-00214-6","article-title":"Model-based human robot collaboration system for small batch Assembly with a virtual fence","volume":"7","author":"Lee","year":"2020","journal-title":"Int. J. Precis. Eng. Manuf. Technol."},{"key":"ref_24","unstructured":"Belingardi, G., Heydaryan, S., and Chiabert, P. (2017, January 4\u20136). Application of speed and separation monitoring method in human-robot collaboration: Industrial case study. Proceedings of the 17th International Scientific Conference on Industrial Systems, Novi Sad, Serbia."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Hawkins, K.P., Vo, N., Bansal, S., and Bobick, A.F. (2013, January 15\u201317). Probabilistic human action prediction and wait-sensitive planning for responsive human-robot collaboration. Proceedings of the 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids), Atlanta, GA, USA.","DOI":"10.1109\/HUMANOIDS.2013.7030020"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"287","DOI":"10.1016\/j.jmsy.2017.04.009","article-title":"Human motion prediction for human-robot collaboration","volume":"44","author":"Liu","year":"2017","journal-title":"J. Manuf. Syst."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"3934","DOI":"10.1109\/TII.2018.2882741","article-title":"Prediction of human activity patterns for human\u2013robot collaborative assembly tasks","volume":"15","author":"Zanchettin","year":"2018","journal-title":"IEEE Trans. Ind. Informatics"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1016\/j.jmsy.2021.12.001","article-title":"Human-robot collaboration empowered by hidden semi-Markov model for operator behaviour prediction in a smart assembly system","volume":"62","author":"Lin","year":"2022","journal-title":"J. Manuf. Syst."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1007\/s10514-017-9655-8","article-title":"Unsupervised early prediction of human reaching for human\u2013robot collaboration in shared workspaces","volume":"42","author":"Luo","year":"2018","journal-title":"Auton. Robots"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Huang, C.-M., and Mutlu, B. (2016, January 7\u201310). Anticipatory robot control for efficient human-robot collaboration. Proceedings of the 2016 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI), Christchurch, New Zealand.","DOI":"10.1109\/HRI.2016.7451737"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"102137","DOI":"10.1016\/j.rcim.2021.102137","article-title":"EEG based arm movement intention recognition towards enhanced safety in symbiotic Human-Robot Collaboration","volume":"70","author":"Buerkle","year":"2021","journal-title":"Robot. Comput. Integr. Manuf."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"272","DOI":"10.1016\/j.procir.2019.04.080","article-title":"Deep learning-based human motion prediction considering context awareness for human-robot collaboration in manufacturing","volume":"83","author":"Liu","year":"2019","journal-title":"Procedia CIRP"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Muthugala, M.A., Srimal, P.H.D., and Jayasekara, A.G. (2017). Enhancing interpretation of ambiguous voice instructions based on the environment and the user\u2019s intention for improved human-friendly robot navigation. Appl. Sci., 7.","DOI":"10.3390\/app7080821"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Luo, R.C., and Mai, L. (2019, January 3\u20138). Human intention inference and on-line human hand motion prediction for human-robot collaboration. Proceedings of the 2019 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.","DOI":"10.1109\/IROS40897.2019.8968192"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Wang, W., Li, R., Chen, Y., and Jia, Y. (2018, January 5\u20138). Human Intention Prediction in Human-Robot Collaborative Tasks. Proceedings of the Companion of the 2018 ACM\/IEEE International Conference on Human-Robot Interaction, Chicago, IL, USA.","DOI":"10.1145\/3173386.3177025"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"4289","DOI":"10.1109\/LRA.2018.2865034","article-title":"Operator awareness in human\u2013robot collaboration through wearable vibrotactile feedback","volume":"3","author":"Casalino","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Mikolov, T., Karafi\u00e1t, M., Burget, L., Cernock\u00fd, J., and Khudanpur, S. (2010, January 26\u201330). Recurrent neural network based language model. Proceedings of the Interspeech, Chiba, Japan.","DOI":"10.21437\/Interspeech.2010-343"},{"key":"ref_38","first-page":"46","article-title":"Kalman and extended kalman filters: Concept, derivation and properties","volume":"43","author":"Ribeiro","year":"2004","journal-title":"Inst. Syst. Robot."},{"key":"ref_39","unstructured":"Latombe, J.-C. (2012). Robot Motion Planning, Springer Science & Business Media."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y. (2009, January 12\u201317). ROS: An open-source Robot Operating System. Proceedings of the ICRA Workshop on Open Source Software, Kobe, Japan.","DOI":"10.1109\/MRA.2010.936956"},{"key":"ref_41","first-page":"802","article-title":"Convolutional LSTM network: A machine learning approach for precipitation nowcasting","volume":"28","author":"Shi","year":"2015","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"2222","DOI":"10.1109\/TNNLS.2016.2582924","article-title":"LSTM: A search space odyssey","volume":"28","author":"Greff","year":"2016","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Islam, M.M.M., and Kim, J.-M. (2019). Vision-Based Autonomous Crack Detection of Concrete Structures Using a Fully Convolutional Encoder\u2013Decoder Network. Sensors, 19.","DOI":"10.3390\/s19194251"},{"key":"ref_44","unstructured":"Welch, G., and Bishop, G. (1995). An Introduction to the Kalman Filter, University of North Carolina at Chapel Hill."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"115144","DOI":"10.1016\/j.compstruct.2021.115144","article-title":"Impact identification of composite cylinder based on improved deep metric learning model and weighted fusion Tikhonov regularized total least squares","volume":"283","author":"Li","year":"2022","journal-title":"Compos. Struct."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Albawi, S., Mohammed, T.A., and Al-Zawi, S. (2017, January 21\u201323). Understanding of a convolutional neural network. Proceedings of the 2017 International Conference on Engineering and Technology (ICET), Antalya, Turkey.","DOI":"10.1109\/ICEngTechnol.2017.8308186"},{"key":"ref_47","unstructured":"Ioffe, S., and Szegedy, C. (2015, January 7\u20139). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the International Conference on Machine Learning, PMLR, Lille, France."},{"key":"ref_48","unstructured":"Wang, L., Liu, S., Liu, H., and Wang, X.V. (2020, January 1\u20134). Overview of human-robot collaboration in manufacturing. Proceedings of the 5th International Conference on the Industry 4.0 Model for Advanced Manufacturing, Belgrade, Serbia."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Kunze, L., Roehm, T., and Beetz, M. (2011, January 9\u201313). Towards semantic robot description languages. Proceedings of the 2011 IEEE International Conference on Robotics and Automation, Shanghai, China.","DOI":"10.1109\/ICRA.2011.5980170"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/MRA.2011.2181749","article-title":"Moveit![ros topics]","volume":"19","author":"Chitta","year":"2012","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1109\/MRA.2012.2205651","article-title":"The open motion planning library","volume":"19","author":"Sucan","year":"2012","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"883","DOI":"10.1177\/0278364915577958","article-title":"Fast marching tree: A fast marching sampling-based method for optimal motion planning in many dimensions","volume":"34","author":"Janson","year":"2015","journal-title":"Int. J. Rob. Res."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1016\/j.jmsy.2018.01.006","article-title":"Data-driven smart manufacturing","volume":"48","author":"Tao","year":"2018","journal-title":"J. Manuf. Syst."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1186\/s40537-019-0197-0","article-title":"A survey on image data augmentation for deep learning","volume":"6","author":"Shorten","year":"2019","journal-title":"J. Big Data"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Fredriksson, T., Mattos, D.I., Bosch, J., and Olsson, H.H. (2020, January 25\u201327). Data labeling: An empirical investigation into industrial challenges and mitigation strategies. Proceedings of the International Conference on Product-Focused Software Process Improvement, Turin, Italy.","DOI":"10.1007\/978-3-030-64148-1_13"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"2733","DOI":"10.12928\/telkomnika.v16i6.9638","article-title":"Speech recognition application for the speech impaired using the android-based google cloud speech API","volume":"16","author":"Anggraini","year":"2018","journal-title":"Telkomnika"},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1109\/THMS.2019.2904558","article-title":"Evaluating fluency in human\u2013robot collaboration","volume":"49","author":"Hoffman","year":"2019","journal-title":"IEEE Trans. Human-Mach. Syst."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/11\/4279\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:24:22Z","timestamp":1760138662000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/11\/4279"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,3]]},"references-count":57,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2022,6]]}},"alternative-id":["s22114279"],"URL":"https:\/\/doi.org\/10.3390\/s22114279","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,3]]}}}