{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T04:48:07Z","timestamp":1780634887443,"version":"3.54.1"},"reference-count":35,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T00:00:00Z","timestamp":1660694400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Priority Research Area DigiWorld PSP","award":["U1U\/P06\/NO\/02.30"],"award-info":[{"award-number":["U1U\/P06\/NO\/02.30"]}]},{"name":"Priority Research Area DigiWorld PSP","award":["K\/PBI\/000171"],"award-info":[{"award-number":["K\/PBI\/000171"]}]},{"DOI":"10.13039\/501100014434","name":"Polish National Agency for Academic Exchange","doi-asserted-by":"publisher","award":["U1U\/P06\/NO\/02.30"],"award-info":[{"award-number":["U1U\/P06\/NO\/02.30"]}],"id":[{"id":"10.13039\/501100014434","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100014434","name":"Polish National Agency for Academic Exchange","doi-asserted-by":"publisher","award":["K\/PBI\/000171"],"award-info":[{"award-number":["K\/PBI\/000171"]}],"id":[{"id":"10.13039\/501100014434","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Multi-purpose service robots must execute their tasks reliably in different situations, as well as learn from humans and explain their plans to them. We address these issues by introducing a knowledge representation scheme to facilitate skill generalization and explainability. This scheme allows representing knowledge of the robot\u2019s understanding of a scene and performed task. We also present techniques for extracting this knowledge from raw data. Such knowledge representation and extraction methods have not been explored adequately in previous research. Our approach does not require any prior knowledge or 3D models of the objects involved. Moreover, the representation scheme is easy to understand for humans. The system is modular so that new recognition or reasoning routines can be added without changing the basic architecture. We developed a computer vision system and a task reasoning module that works with our knowledge representation. The efficacy of our approach is demonstrated with two different tasks: hanging items on pegs and stacking one item on another. A formalization of our knowledge representation scheme is presented, showing how the system is capable of learning from a few demonstrations.<\/jats:p>","DOI":"10.3390\/s22166156","type":"journal-article","created":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T22:53:30Z","timestamp":1660776810000},"page":"6156","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["An Approach to Task Representation Based on Object Features and Affordances"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0931-2476","authenticated-orcid":false,"given":"Paul","family":"Gajewski","sequence":"first","affiliation":[{"name":"Institute of Computer Science, AGH University of Science and Technology, 30-059 Krakow, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3798-9209","authenticated-orcid":false,"given":"Bipin","family":"Indurkhya","sequence":"additional","affiliation":[{"name":"Institute of Philosophy, Jagiellonian University, 31-007 Krakow, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,8,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1080\/01691864.2022.2029720","article-title":"Explainable autonomous robots: A survey and perspective","volume":"36","author":"Sakai","year":"2022","journal-title":"Adv. Robot."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Gajewski, P., Ferreira, P., Bartels, G., Wang, C., Guerin, F., Indurkhya, B., Beetz, M., and Sniezynski, B. (2019, January 20\u201324). Adapting Everyday Manipulation Skills to Varied Scenarios. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8793590"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Mitrevski, A., Ploger, P.G., and Lakemeyer, G. (October, January 27). Ontology-Assisted Generalisation of Robot Action Execution Knowledge. Proceedings of the 2021 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic.","DOI":"10.1109\/IROS51168.2021.9636791"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Beetz, M., Bessler, D., Haidu, A., Pomarlan, M., Bozcuoglu, A.K., and Bartels, G. (2018, January 21\u201325). Know Rob 2.0\u2014A 2nd Generation Knowledge Processing Framework for Cognition-Enabled Robotic Agents. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia.","DOI":"10.1109\/ICRA.2018.8460964"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Paulius, D., Huang, Y., Milton, R., Buchanan, W.D., Sam, J., and Sun, Y. (2016, January 9\u201314). Functional object-oriented network for manipulation learning. Proceedings of the 2016 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea.","DOI":"10.1109\/IROS.2016.7759413"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Lin, Y., Tremblay, J., Tyree, S., Vela, P.A., and Birchfield, S. (October, January 27). Multi-view Fusion for Multi-level Robotic Scene Understanding. Proceedings of the 2021 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic.","DOI":"10.1109\/IROS51168.2021.9635994"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"639","DOI":"10.1007\/s00422-015-0666-2","article-title":"Learning to grasp and extract affordances: The Integrated Learning of Grasps and Affordances (ILGA) model","volume":"109","author":"Bonaiuto","year":"2015","journal-title":"Biol. Cybern."},{"key":"ref_8","unstructured":"Gibson, J.J., and Carmichael, L. (1966). The Senses Considered as Perceptual Systems, Houghton Mifflin."},{"key":"ref_9","first-page":"111","article-title":"Coordinated control programs for movements of the hand","volume":"10","author":"Arbib","year":"1985","journal-title":"Exp. Brain Res."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1177\/0278364917710318","article-title":"Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection","volume":"37","author":"Levine","year":"2018","journal-title":"Int. J. Robot. Res."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Kazhoyan, G., and Beetz, M. (2017, January 24\u201328). Programming robotic agents with action descriptions. Proceedings of the 2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.","DOI":"10.1109\/IROS.2017.8202144"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1007\/s10845-021-01867-z","article-title":"A review of motion planning algorithms for intelligent robots","volume":"33","author":"Zhou","year":"2022","journal-title":"J. Intell. Manuf."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Brandao, M., Canal, G., Krivic, S., and Magazzeni, D. (June, January 30). Towards providing explanations for robot motion planning. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi\u2019an, China.","DOI":"10.1109\/ICRA48506.2021.9562003"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Aparicio, J., and Goldberg, K. (2017, January 12\u201316). Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics. Proceedings of the Robotics: Science and Systems XIII, Cambridge, MA, USA.","DOI":"10.15607\/RSS.2017.XIII.058"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Takeuchi, K., Yanokura, I., Kakiuchi, Y., Okada, K., and Inaba, M. (June, January 30). Automatic Hanging Point Learning from Random Shape Generation and Physical Function Validation. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi\u2019an, China.","DOI":"10.1109\/ICRA48506.2021.9561484"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1007\/s10015-013-0105-1","article-title":"Bayesian learning of tool affordances based on generalization of functional feature to estimate effects of unseen tools","volume":"18","author":"Jain","year":"2013","journal-title":"Artif. Life Robot."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Montesano, L., Lop, M., Bernardino, A., and Santos-Victor, J. (November, January 29). Modeling affordances using Bayesian networks. Proceedings of the 2007 IEEE\/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, USA.","DOI":"10.1109\/IROS.2007.4399511"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"725","DOI":"10.1609\/icaps.v29i1.3541","article-title":"Open-World Reasoning for Service Robots","volume":"29","author":"Jiang","year":"2021","journal-title":"ICAPS"},{"key":"ref_19","unstructured":"Be\u00dfler, D., Porzel, R., Pomarlan, M., Beetz, M., Malaka, R., and Bateman, J. (September, January 29). A formal model of affordances for flexible robotic task execution. Proceedings of the ECAI\u20142020, Santiago de Compostela, Spain."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"103763","DOI":"10.1016\/j.robot.2021.103763","article-title":"Ontology-based knowledge management with verbal interaction for command interpretation and execution by home service robots","volume":"140","author":"Miura","year":"2021","journal-title":"Robot. Auton. Syst."},{"key":"ref_21","unstructured":"Be\u00dfler, D., Koralewski, S., and Beetz, M. (2018, January 27). Knowledge Representation for Cognition-and Learning-enabled Robot Manipulation. Proceedings of the 11th International Workshop on Cognitive Robotics, Tempe, AZ, USA."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Vassiliades, A., Bassiliades, N., Gouidis, F., and Patkos, T. (2020, January 7\u201310). A knowledge retrieval framework for household objects and actions with external knowledge. Proceedings of the International Conference on Semantic Systems, Amsterdam, The Netherlands.","DOI":"10.1007\/978-3-030-59833-4_3"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.robot.2019.03.005","article-title":"A Survey of Knowledge Representation in Service Robotics","volume":"118","author":"Paulius","year":"2019","journal-title":"Robot. Auton. Syst."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Chiatti, A., Motta, E., and Daga, E. (2020, January 12\u201318). Towards a Framework for Visual Intelligence in Service Robotics: Epistemic Requirements and Gap Analysis. Proceedings of the Seventeenth International Conference on Principles of Knowledge Representation and Reasoning, Rhodes, Greece.","DOI":"10.24963\/kr.2020\/93"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1007\/s13218-015-0364-1","article-title":"Open-EASE: A Cloud-Based Knowledge Service for Autonomous Learning","volume":"29","author":"Tenorth","year":"2015","journal-title":"K\u00fcnstl. Intell."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Daruna, A., Liu, W., Kira, Z., and Chetnova, S. (2019, January 20\u201324). RoboCSE: Robot Common Sense Embedding. Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8794070"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"674292","DOI":"10.3389\/frobt.2021.674292","article-title":"Modeling and Learning Constraints for Creative Tool Use","volume":"8","author":"Fitzgerald","year":"2021","journal-title":"Front. Robot. AI"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Mitrevski, A., Ploger, P.G., and Lakemeyer, G. (2020\u201324, January 24). Representation and Experience-Based Learning of Explainable Models for Robot Action Execution. Proceedings of the 2020 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.","DOI":"10.1109\/IROS45743.2020.9341470"},{"key":"ref_29","first-page":"1","article-title":"Building the Foundation of Robot Explanation Generation Using Behavior Trees","volume":"10","author":"Han","year":"2021","journal-title":"ACM Trans. Hum.-Robot Interact."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Kazhoyan, G., and Beetz, M. (2019, January 3\u20138). Executing Underspecified Actions in Real World Based on Online Projection. Proceedings of the 2019 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.","DOI":"10.1109\/IROS40897.2019.8967867"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1109\/TAMD.2015.2488284","article-title":"Bootstrapping the Semantics of Tools: Affordance Analysis of Real World Objects on a Per-part Basis","volume":"8","author":"Schoeler","year":"2016","journal-title":"IEEE Trans. Cogn. Dev. Syst."},{"key":"ref_32","unstructured":"Guerin, F. (2022). Projection: A mechanism for human-like reasoning in Artificial Intelligence. J. Exp. Theor. Artif. Intell., 1\u201325."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"927","DOI":"10.1016\/j.robot.2008.08.005","article-title":"Towards 3D Point cloud based object maps for household environments","volume":"56","author":"Rusu","year":"2008","journal-title":"Robot. Auton. Syst."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1145\/358669.358692","article-title":"Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography","volume":"24","author":"Fischler","year":"1981","journal-title":"Commun. ACM"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1007\/s13218-010-0059-6","article-title":"Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments","volume":"24","author":"Rusu","year":"2010","journal-title":"K\u00fcnstl. Intell."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/16\/6156\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:11:08Z","timestamp":1760141468000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/16\/6156"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,17]]},"references-count":35,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2022,8]]}},"alternative-id":["s22166156"],"URL":"https:\/\/doi.org\/10.3390\/s22166156","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,17]]}}}