{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:57:06Z","timestamp":1760144226024,"version":"build-2065373602"},"reference-count":47,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2024,4,1]],"date-time":"2024-04-01T00:00:00Z","timestamp":1711929600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Union","award":["871352","810346","964854"],"award-info":[{"award-number":["871352","810346","964854"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>The objective of this paper is to propose a framework for a robot to learn multiple Sensory-Motor Contingencies from human demonstrations and reproduce them. Sensory-Motor Contingencies are a concept that describes intelligent behavior of animals and humans in relation to their environment. They have been used to design control and planning algorithms for robots capable of interacting and adapting autonomously. However, enabling a robot to autonomously develop Sensory-Motor Contingencies is challenging due to the complexity of action and perception signals. This framework leverages tools from Learning from Demonstrations to have the robot memorize various sensory phases and corresponding motor actions through an attention mechanism. This generates a metric in the perception space, used by the robot to determine which sensory-motor memory is contingent to the current context. The robot generalizes the memorized actions to adapt them to the present perception. This process creates a discrete lattice of continuous Sensory-Motor Contingencies that can control a robot in loco-manipulation tasks. Experiments on a 7-dof collaborative robotic arm with a gripper, and on a mobile manipulator demonstrate the functionality and versatility of the framework.<\/jats:p>","DOI":"10.3390\/robotics13040058","type":"journal-article","created":{"date-parts":[[2024,4,1]],"date-time":"2024-04-01T05:44:36Z","timestamp":1711950276000},"page":"58","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Exploring Saliency for Learning Sensory-Motor Contingencies in Loco-Manipulation Tasks"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0033-314X","authenticated-orcid":false,"given":"Elisa","family":"Stefanini","sequence":"first","affiliation":[{"name":"Soft Robotics for Human Cooperation and Rehabilitation, Fondazione Istituto Italiano di Tecnologia, Via Morego 30, 16163 Genova, Italy"},{"name":"Centro di Ricerca \u201cE. Piaggio\u201d, Dipartimento di Ingegneria dell\u2019Informazione, Universit\u00e1 di Pisa, Largo L. Lazzarino 1, 56122 Pisa, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gianluca","family":"Lentini","sequence":"additional","affiliation":[{"name":"Soft Robotics for Human Cooperation and Rehabilitation, Fondazione Istituto Italiano di Tecnologia, Via Morego 30, 16163 Genova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Giorgio","family":"Grioli","sequence":"additional","affiliation":[{"name":"Soft Robotics for Human Cooperation and Rehabilitation, Fondazione Istituto Italiano di Tecnologia, Via Morego 30, 16163 Genova, Italy"},{"name":"Centro di Ricerca \u201cE. Piaggio\u201d, Dipartimento di Ingegneria dell\u2019Informazione, Universit\u00e1 di Pisa, Largo L. Lazzarino 1, 56122 Pisa, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Manuel Giuseppe","family":"Catalano","sequence":"additional","affiliation":[{"name":"Soft Robotics for Human Cooperation and Rehabilitation, Fondazione Istituto Italiano di Tecnologia, Via Morego 30, 16163 Genova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Antonio","family":"Bicchi","sequence":"additional","affiliation":[{"name":"Soft Robotics for Human Cooperation and Rehabilitation, Fondazione Istituto Italiano di Tecnologia, Via Morego 30, 16163 Genova, Italy"},{"name":"Centro di Ricerca \u201cE. Piaggio\u201d, Dipartimento di Ingegneria dell\u2019Informazione, Universit\u00e1 di Pisa, Largo L. Lazzarino 1, 56122 Pisa, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,4,1]]},"reference":[{"key":"ref_1","first-page":"95","article-title":"Predictive feed-forward sensory control during grasping and manipulation in man","volume":"14","author":"Johansson","year":"1993","journal-title":"Biomed. Res."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"98","DOI":"10.3389\/fnbot.2019.00098","article-title":"Sensorimotor contingencies as a key drive of development: From babies to robots","volume":"13","author":"Jacquey","year":"2019","journal-title":"Front. Neurorobot."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"285","DOI":"10.3389\/fpsyg.2013.00285","article-title":"A dynamical systems account of sensorimotor contingencies","volume":"4","author":"Buhrmann","year":"2013","journal-title":"Front. Psychol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1023\/A:1012699224677","article-title":"What it is like to see: A sensorimotor theory of perceptual experience","volume":"129","year":"2001","journal-title":"Synthese"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Maye, A., and Engel, A.K. (2011, January 3\u201313). A discrete computational model of sensorimotor contingencies for object perception and control of behavior. Proceedings of the 2011 IEEE International Conference on Robotics and Automation, IEEE, Shanghai, China.","DOI":"10.1109\/ICRA.2011.5979919"},{"key":"ref_6","unstructured":"Maye, A., and Engel, A.K. (2012). Proceedings of the From Animals to Animats 12: 12th International Conference on Simulation of Adaptive Behavior, SAB 2012, Odense, Denmark, 27\u201330 August 2012, Springer. Proceedings 12."},{"key":"ref_7","unstructured":"Hoffmann, M., Schmidt, N.M., Pfeifer, R., Engel, A.K., and Maye, A. (2012). Proceedings of the From Animals to Animats 12: 12th International Conference on Simulation of Adaptive Behavior, SAB 2012, Odense, Denmark, 27\u201330 August 2012, Springer. Proceedings 12."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"624610","DOI":"10.3389\/fnhum.2021.624610","article-title":"Socializing sensorimotor contingencies","volume":"15","author":"Krause","year":"2021","journal-title":"Front. Hum. Neurosci."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Gibson, J.J. (2014). The Ecological Approach to Visual Perception: Classic Edition, Psychology Press.","DOI":"10.4324\/9781315740218"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1177\/1059712313497975","article-title":"Extending sensorimotor contingency theory: Prediction, planning, and action generation","volume":"21","author":"Maye","year":"2013","journal-title":"Adapt. Behav."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Ard\u00f3n, P., Pairet, \u00c8., Lohan, K.S., Ramamoorthy, S., and Petrick, R.P.A. (2021). Building Affordance Relations for Robotic Agents\u2014A Review. arXiv.","DOI":"10.24963\/ijcai.2021\/590"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Montesano, L., Lopes, M., Bernardino, A., and Santos-Victor, J. (November, January 29). Modeling affordances using bayesian networks. Proceedings of the 2007 IEEE\/RSJ International Conference on Intelligent Robots and Systems, IEEE, San Diego, CA, USA.","DOI":"10.1109\/IROS.2007.4399511"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"740","DOI":"10.1016\/j.robot.2011.05.009","article-title":"Object\u2013action complexes: Grounded abstractions of sensory\u2013motor processes","volume":"59","author":"Geib","year":"2011","journal-title":"Robot. Auton. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Dogar, M.R., Ugur, E., Sahin, E., and Cakmak, M. (2008, January 19\u201323). Using learned affordances for robotic behavior development. Proceedings of the 2008 IEEE International Conference on Robotics and Automation, Pasadena, CA, USA.","DOI":"10.1109\/ROBOT.2008.4543794"},{"key":"ref_15","unstructured":"Ard\u00f3n, P., Pairet, \u00c8., Lohan, K.S., Ramamoorthy, S., and Petrick, R. (2020). Affordances in robotic tasks\u2013a survey. arXiv."},{"key":"ref_16","unstructured":"Datteri, E., Teti, G., Laschi, C., Tamburrini, G., Dario, G., and Guglielmelli, E. (2003, January 27\u201331). Expected perception: An anticipation-based perception-action scheme in robots. Proceedings of the 2003 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453), Las Vegas, NV, USA."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1428","DOI":"10.1109\/LRA.2023.3346758","article-title":"Adaptive shape servoing of elastic rods using parameterized regression features and auto-tuning motion controls","volume":"9","author":"Qi","year":"2023","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_18","unstructured":"Yang, C., Zhou, P., and Qi, J. (2023). Integrating visual foundation models for enhanced robot manipulation and motion planning: A layered approach. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1238","DOI":"10.1177\/0278364913495721","article-title":"Reinforcement learning in robotics: A survey","volume":"32","author":"Kober","year":"2013","journal-title":"Int. J. Robot. Res."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Ghadirzadeh, A., B\u00fctepage, J., Maki, A., Kragic, D., and Bj\u00f6rkman, M. (2016, January 9\u201314). A sensorimotor reinforcement learning framework for physical Human-Robot Interaction. Proceedings of the 2016 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Republic of Korea.","DOI":"10.1109\/IROS.2016.7759417"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Hay, N., Stark, M., Schlegel, A., Wendelken, C., Park, D., Purdy, E., Silver, T., Phoenix, D.S., and George, D. (2018). Behavior Is Everything: Towards Representing Concepts with Sensorimotor Contingencies. Proc. AAAI Conf. Artif. Intell., 32.","DOI":"10.1609\/aaai.v32i1.11547"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"2419","DOI":"10.1007\/s10994-021-05961-4","article-title":"Challenges of real-world reinforcement learning: Definitions, benchmarks and analysis","volume":"110","author":"Levine","year":"2021","journal-title":"Mach. Learn."},{"key":"ref_23","unstructured":"Maye, A., Trendafilov, D., Polani, D., and Engel, A. (2015, January 2). A visual attention mechanism for autonomous robots controlled by sensorimotor contingencies. Proceedings of the IROS 2015 Workshop on Sensorimotor Contingencies For Robotics, Hamburg, Germany."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1146\/annurev-control-100819-063206","article-title":"Recent advances in robot learning from demonstration","volume":"3","author":"Ravichandar","year":"2020","journal-title":"Annu. Rev. Control. Robot. Auton. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Correia, A., and Alexandre, L.A. (2023). A Survey of Demonstration Learning. arXiv.","DOI":"10.2139\/ssrn.4390650"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"23441","DOI":"10.1007\/s00521-021-06449-x","article-title":"Human\u2013robot skill transmission for mobile robot via learning by demonstration","volume":"35","author":"Li","year":"2021","journal-title":"Neural Comput. Appl."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"8036","DOI":"10.1109\/LRA.2022.3187258","article-title":"A Hybrid Learning and Optimization Framework to Achieve Physically Interactive Tasks With Mobile Manipulators","volume":"7","author":"Zhao","year":"2022","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1007\/s10514-015-9502-8","article-title":"Human\u2013robot planning and learning for marine data collection","volume":"40","author":"Somers","year":"2016","journal-title":"Auton. Robot."},{"key":"ref_29","unstructured":"Zeng, A., Florence, P., Tompson, J., Welker, S., Chien, J., Attarian, M., Armstrong, T., Krasin, I., Duong, D., and Sindhwani, V. (2021, January 8\u201311). Transporter networks: Rearranging the visual world for robotic manipulation. Proceedings of the Conference on Robot Learning, PMLR, London, UK."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Goldberg, K., and Abbeel, P. (2018, January 21\u201325). Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE, Brisbane, Australia.","DOI":"10.1109\/ICRA.2018.8461249"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Zhu, Y., Wang, Z., Merel, J., Rusu, A., Erez, T., Cabi, S., Tunyasuvunakool, S., Kram\u00e1r, J., Hadsell, R., and de Freitas, N. (2018). Reinforcement and imitation learning for diverse visuomotor skills. arXiv.","DOI":"10.15607\/RSS.2018.XIV.009"},{"key":"ref_32","unstructured":"Li, Y., Song, J., and Ermon, S. (2017). Advances in Neural Information Processing Systems 30 (NIPS 2017), Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4\u20139 December 2017, Neural Inforamation Processing Systems."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Pan, Y., Cheng, C.A., Saigol, K., Lee, K., Yan, X., Theodorou, E., and Boots, B. (2017). Agile autonomous driving using end-to-end deep imitation learning. arXiv.","DOI":"10.15607\/RSS.2018.XIV.056"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1088","DOI":"10.1109\/LRA.2018.2795643","article-title":"Dronet: Learning to fly by driving","volume":"3","author":"Loquercio","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Calandra, R., Gopalan, N., Seyfarth, A., Peters, J., and Deisenroth, M.P. (2014, January 16\u201321). Bayesian Gait Optimization for Bipedal Locomotion. Proceedings of the LION, Learning and Intelligent Optimization: 8th International Conference, Lion 8, Gainesville, FL, USA.","DOI":"10.1007\/978-3-319-09584-4_25"},{"key":"ref_36","unstructured":"Gopalan, N., Moorman, N., Natarajan, M., and Gombolay, M. (July, January 27). Negative Result for Learning from Demonstration: Challenges for End-Users Teaching Robots with Task and Motion Planning Abstractions. Proceedings of the Robotics: Science and Systems (RSS), New York, NY, USA."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Maye, A., and Engel, A.K. (2013, January 3\u20137). Context-dependent dynamic weighting of information from multiple sensory modalities. Proceedings of the 2013 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan.","DOI":"10.1109\/IROS.2013.6696754"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"939","DOI":"10.1017\/S0140525X01000115","article-title":"A sensorimotor account of vision and visual consciousness","volume":"24","year":"2001","journal-title":"Behav. Brain Sci."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1162\/NECO_a_00393","article-title":"Dynamical movement primitives: Learning attractor models for motor behaviors","volume":"25","author":"Ijspeert","year":"2013","journal-title":"Neural Comput."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"80716","DOI":"10.1109\/ACCESS.2020.2988796","article-title":"Unsupervised K-means clustering algorithm","volume":"8","author":"Sinaga","year":"2020","journal-title":"IEEE Access"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1016\/0167-6393(93)90083-W","article-title":"Automatic segmentation and labeling of speech based on hidden Markov models","volume":"12","author":"Brugnara","year":"1993","journal-title":"Speech Commun."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Zhang, H., Han, X., Zhang, W., and Zhou, W. (2018, January 12\u201315). Complex sequential tasks learning with Bayesian inference and Gaussian mixture model. Proceedings of the 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, Kuala Lumpur, Malaysia.","DOI":"10.1109\/ROBIO.2018.8665229"},{"key":"ref_43","unstructured":"Escudero-Rodrigo, D., and Alquezar, R. (2015). Artificial Intelligence Research and Development, IOS Press."},{"key":"ref_44","unstructured":"Koenig, N., and Howard, A. (October, January 28). Design and use paradigms for Gazebo, an open-source multi-robot simulator. Proceedings of the 2004 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), Sendai, Japan."},{"key":"ref_45","first-page":"401","article-title":"On a Measure of Divergence between Two Multinomial Populations","volume":"7","author":"Bhattacharyya","year":"1946","journal-title":"Sankhya Indian J. Stat."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Ajoudani, A. (2016). Transferring Human Impedance Regulation Skills to Robots, Springer.","DOI":"10.1007\/978-3-319-24205-7"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"768","DOI":"10.1177\/0278364913518998","article-title":"Adaptive synergies for the design and control of the Pisa\/IIT SoftHand","volume":"33","author":"Catalano","year":"2014","journal-title":"Int. J. Robot. Res."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/13\/4\/58\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:22:00Z","timestamp":1760106120000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/13\/4\/58"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,1]]},"references-count":47,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2024,4]]}},"alternative-id":["robotics13040058"],"URL":"https:\/\/doi.org\/10.3390\/robotics13040058","relation":{},"ISSN":["2218-6581"],"issn-type":[{"type":"electronic","value":"2218-6581"}],"subject":[],"published":{"date-parts":[[2024,4,1]]}}}