{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,3]],"date-time":"2025-03-03T05:36:13Z","timestamp":1740980173337,"version":"3.38.0"},"reference-count":42,"publisher":"SAGE Publications","issue":"5","license":[{"start":{"date-parts":[[2019,4,12]],"date-time":"2019-04-12T00:00:00Z","timestamp":1555027200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Adaptive Behavior"],"published-print":{"date-parts":[[2019,10]]},"abstract":"<jats:p> Recent advances in deep learning, in particular in convolutional neural networks (CNNs), have been widely used in robotics for object classification and action recognition, among others, with very high performance. Nevertheless, this high performance, mostly in classification tasks, is rarely accompanied by reasoning processes that consider the relationships between objects, actions, and effects. In this article, we used three CNNs to classify objects, actions, and effects that were trained with the CERTH-SOR3D dataset that has more than 20,000 RGB-D videos. This dataset involves 14 objects, 13 actions, and in this article was augmented with seven effects. The probabilistic vector output of each trained CNN was combined into a Bayesian network (BN) to capture the relationships between objects, actions, and effects. It is shown that by probabilistically combining information from the three classifiers, it is possible to improve the classification performance of each CNN or to level the same performance with less training data. In particular, the recognition performance improved from 71.2% to 79.7% for actions, 85.0%\u201386.7% for objects, and 77.0%\u201382.1% for effects. In the article, it is also shown that with missing information, the model can still produce reasonable classification performance. In particular, the system can be used for reasoning purposes in robotics, as it can make action planning with information from object and effects or it can predict effects with information from objects and actions. <\/jats:p>","DOI":"10.1177\/1059712319839057","type":"journal-article","created":{"date-parts":[[2019,4,12]],"date-time":"2019-04-12T09:38:03Z","timestamp":1555061883000},"page":"295-306","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":2,"title":["Enhancing object, action, and effect recognition using probabilistic affordances"],"prefix":"10.1177","volume":"27","author":[{"given":"Esteban","family":"Jaramillo-Cabrera","sequence":"first","affiliation":[{"name":"Instituto Nacional de Astrof\u00edsica, \u00d3ptica y Electr\u00f3nica (INAOE), Puebla, Mexico"},{"name":"Universidad Tecnol\u00f3gica de Pereira (UTP), Pereira, Colombia"}]},{"given":"Eduardo F","family":"Morales","sequence":"additional","affiliation":[{"name":"Instituto Nacional de Astrof\u00edsica, \u00d3ptica y Electr\u00f3nica (INAOE), Puebla, Mexico"}]},{"given":"Jose","family":"Martinez-Carranza","sequence":"additional","affiliation":[{"name":"Instituto Nacional de Astrof\u00edsica, \u00d3ptica y Electr\u00f3nica (INAOE), Puebla, Mexico"},{"name":"University of Bristol, Bristol, UK"}]}],"member":"179","published-online":{"date-parts":[[2019,4,12]]},"reference":[{"key":"bibr1-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2015.7178146"},{"key":"bibr2-1059712319839057","unstructured":"Cakmak M., Dogar M., Ugur E., Sahin E. (2007). Affordances as a framework for robot control. Paper presented at the proceedings of the 7th international conference on epigenetic robotics, epirob\u201907. Retrieved from https:\/\/www.semanticscholar.org\/paper\/Affordances-as-a-Framework-for-Robot-Control-Cakmak-Dogar\/d406239de17bbb4f66d4c835f2a5f1be977b4131"},{"key":"bibr3-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/TAMD.2011.2106782"},{"key":"bibr4-1059712319839057","doi-asserted-by":"crossref","unstructured":"Chu V., Akgun B., Thomaz A. (2016). Learning haptic affordances from demonstration and human-guided exploration. Paper presented at 2016 IEEE Haptics Symposium (HAPTICS). Retrieved from https:\/\/ieeexplore.ieee.org\/document\/7463165","DOI":"10.1109\/HAPTICS.2016.7463165"},{"key":"bibr5-1059712319839057","doi-asserted-by":"crossref","unstructured":"Chu V., Fitzgerald T., Thomaz A. (2016). Learning object affordances by leveraging the combination of human-guidance and self-exploration. Paper presented at 2016 11th ACM\/IEEE international conference on Human-Robot Interaction (HRI). Retrieved from https:\/\/ieeexplore.ieee.org\/document\/7451755","DOI":"10.1109\/HRI.2016.7451755"},{"key":"bibr6-1059712319839057","unstructured":"Chu V., Thomaz A. (2014). Understanding the Role of Haptics in Affordances. Retrieved from http:\/\/theaffordances.appspot.com\/workshops\/RSS\/archive\/CR2.pap.pdf"},{"key":"bibr7-1059712319839057","doi-asserted-by":"crossref","unstructured":"Chu V., Thomaz A. (2016). Learning and grounding haptic affordances using demonstration and human-guided exploration. Paper presented at 2016 11th ACM\/IEEE international conference on Human-Robot Interaction (HRI). Retrieved from https:\/\/ieeexplore.ieee.org\/document\/7451878","DOI":"10.1109\/HRI.2016.7451878"},{"key":"bibr8-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1177\/0278364917693691"},{"key":"bibr9-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/HUMANOIDS.2017.8246915"},{"key":"bibr10-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2018.8460902"},{"key":"bibr11-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2007.4399469"},{"volume-title":"The senses considered as perceptual systems","year":"1966","author":"Gibson J.","key":"bibr12-1059712319839057"},{"key":"bibr13-1059712319839057","doi-asserted-by":"publisher","DOI":"10.2307\/1574154"},{"volume-title":"The ecological approach to visual perception","year":"1979","author":"Gibson J.","key":"bibr14-1059712319839057"},{"key":"bibr15-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/TCDS.2016.2594134"},{"key":"bibr16-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2010.08.002"},{"journal-title":"Anticipating the Future by Constructing Human Activities using Object Affordances","year":"2015","author":"Koppula H.","key":"bibr17-1059712319839057"},{"volume-title":"Human Activity Learning using Object Affordances from RGB-D Videos","year":"2012","author":"Koppula H.","key":"bibr18-1059712319839057"},{"key":"bibr19-1059712319839057","doi-asserted-by":"crossref","unstructured":"Koppula H., Saxena A. (2013). Anticipating human activities using object affordances for reactive robotic response. Paper presented at IEEE Transactions on Pattern Analysis and Machine Intelligence. Retrieved from https:\/\/ieeexplore.ieee.org\/document\/7102751","DOI":"10.15607\/RSS.2013.IX.006"},{"key":"bibr20-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10578-9_54"},{"volume-title":"ImageNet Classification with Deep Convolutional Neural Networks","year":"2012","author":"Krizhevsky A.","key":"bibr21-1059712319839057"},{"key":"bibr22-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298958"},{"key":"bibr23-1059712319839057","doi-asserted-by":"crossref","unstructured":"Lopes M., Melo F., Montesano L. (2007). Affordance-based imitation learning in robots. Paper presented at IEEE\/RSJ international conference on intelligent robots and systems. Retrieved from https:\/\/ieeexplore.ieee.org\/abstract\/document\/4399517","DOI":"10.1109\/IROS.2007.4399517"},{"key":"bibr24-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2015.7353481"},{"key":"bibr25-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/TCDS.2016.2614992"},{"key":"bibr26-1059712319839057","doi-asserted-by":"crossref","unstructured":"Moldovan B., Moreno P., Otterlo M. (2013). On the use of probabilistic relational affordance models for sequential manipulation tasks in robotics. Paper presented at 2013 IEEE International Conference on Robotics and Automation. Retrieved from http:\/\/vislab.isr.ist.utl.pt\/wp-content\/uploads\/2013\/12\/icra2013_planningWithLeuven.pdf","DOI":"10.1109\/ICRA.2013.6630737"},{"key":"bibr27-1059712319839057","doi-asserted-by":"crossref","unstructured":"Moldovan B., Moreno P., Otterlo M., Santos-Victor J., Raedt L. (2012). Learning relational affordance models for robots in multi-object manipulation tasks. Paper presented at 2012 IEEE International Conference on Robotics and Automation. Retrieved from https:\/\/ieeexplore.ieee.org\/document\/6225042","DOI":"10.1109\/ICRA.2012.6225042"},{"key":"bibr28-1059712319839057","unstructured":"Moldovan B., Otterlo M., Moreno P., Santos-Victor J. (2011). Statistical Relational Learning of Object Affordances for Robotic Manipulation. Retrieved from http:\/\/ilp11.doc.ic.ac.uk\/short_papers\/ilp2011_submission_27.pdf"},{"key":"bibr29-1059712319839057","doi-asserted-by":"crossref","unstructured":"Moldovan B., Raedt L. (2014). Occluded object search by relational affordances. Paper presented at 2014 IEEE International Conference on Robotics and Automation (ICRA). Retrieved from https:\/\/ieeexplore.ieee.org\/abstract\/document\/6906605","DOI":"10.1109\/ICRA.2014.6906605"},{"key":"bibr30-1059712319839057","doi-asserted-by":"crossref","unstructured":"Montesano L., Lopes M. (2009). Learning grasping affordances from local visual descriptors. Paper presented at International Conference on Development and Learning. Retrieved from https:\/\/ieeexplore.ieee.org\/document\/5175529","DOI":"10.1109\/DEVLRN.2009.5175529"},{"key":"bibr31-1059712319839057","doi-asserted-by":"crossref","unstructured":"Montesano L., Lopes M., Bernardino A., Santos-Victor J. (2007). Modeling affordances using Bayesian networks. Paper presented at IEEE\/RSJ International Conference on Intelligent Robots and Systems. Retrieved from https:\/\/ieeexplore.ieee.org\/document\/4399511","DOI":"10.1109\/IROS.2007.4399511"},{"key":"bibr32-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2007.914848"},{"key":"bibr33-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2017.8206484"},{"key":"bibr34-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1145\/301153.301168"},{"key":"bibr35-1059712319839057","doi-asserted-by":"crossref","unstructured":"Norman D. (1999). The design of everyday things (originally published: The psychology of everyday things). Retrieved from http:\/\/www.nixdell.com\/classes\/HCI-and-Design-Spring-2017\/The-Design-of-Everyday-Things-Revised-and-Expanded-Edition.pdf","DOI":"10.15358\/9783800648108"},{"key":"bibr36-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2015.7139361"},{"volume-title":"A multi-scale CNN for affordance segmentation in RGB images in European Conference on Computer Vision","year":"2016","author":"Roy A.","key":"bibr37-1059712319839057"},{"key":"bibr38-1059712319839057","unstructured":"Simonyan K., Zisserman A. (2014). Two-stream convolutional networks for action recognition in videos, NIPS. Retrieved from https:\/\/papers.nips.cc\/paper\/5353-two-stream-convolutional-networks-for-action-recognition-in-videos"},{"key":"bibr39-1059712319839057","doi-asserted-by":"crossref","unstructured":"Thermos S., Papadopoulos G., Daras P., Potamianos G. (2017). Deep affordance-grounded sensorimotor object recognition. Paper presented at 2017 IEEE conference on computer vision and pattern recognition (CVPR). Retrieved from https:\/\/ieeexplore.ieee.org\/document\/8099496","DOI":"10.1109\/CVPR.2017.13"},{"key":"bibr40-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2007.363571"},{"key":"bibr41-1059712319839057","doi-asserted-by":"publisher","DOI":"10.1177\/1059712317726357"},{"volume-title":"Reasoning about object affordances in a knowledge base representation, In European conference on computer vision","year":"2014","author":"Zhu Y.","key":"bibr42-1059712319839057"}],"container-title":["Adaptive Behavior"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1059712319839057","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1059712319839057","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1059712319839057","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,2]],"date-time":"2025-03-02T10:45:06Z","timestamp":1740912306000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1059712319839057"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,4,12]]},"references-count":42,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2019,10]]}},"alternative-id":["10.1177\/1059712319839057"],"URL":"https:\/\/doi.org\/10.1177\/1059712319839057","relation":{},"ISSN":["1059-7123","1741-2633"],"issn-type":[{"type":"print","value":"1059-7123"},{"type":"electronic","value":"1741-2633"}],"subject":[],"published":{"date-parts":[[2019,4,12]]}}}