{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T12:44:51Z","timestamp":1778071491652,"version":"3.51.4"},"reference-count":49,"publisher":"SAGE Publications","issue":"10","license":[{"start":{"date-parts":[[2011,8,1]],"date-time":"2011-08-01T00:00:00Z","timestamp":1312156800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of Robotics Research"],"published-print":{"date-parts":[[2011,9]]},"abstract":"<jats:p>Recognizing manipulations performed by a human and the transfer and execution of this by a robot is a difficult problem. We address this in the current study by introducing a novel representation of the relations between objects at decisive time points during a manipulation. Thereby, we encode the essential changes in a visual scenery in a condensed way such that a robot can recognize and learn a manipulation without prior object knowledge. To achieve this we continuously track image segments in the video and construct a dynamic graph sequence. Topological transitions of those graphs occur whenever a spatial relation between some segments has changed in a discontinuous way and these moments are stored in a transition matrix called the semantic event chain (SEC). We demonstrate that these time points are highly descriptive for distinguishing between different manipulations. Employing simple sub-string search algorithms, SECs can be compared and type-similar manipulations can be recognized with high confidence. As the approach is generic, statistical learning can be used to find the archetypal SEC of a given manipulation class. The performance of the algorithm is demonstrated on a set of real videos showing hands manipulating various objects and performing different actions. In experiments with a robotic arm, we show that the SEC can be learned by observing human manipulations, transferred to a new scenario, and then reproduced by the machine.<\/jats:p>","DOI":"10.1177\/0278364911410459","type":"journal-article","created":{"date-parts":[[2011,8,1]],"date-time":"2011-08-01T22:02:11Z","timestamp":1312236131000},"page":"1229-1249","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":141,"title":["Learning the semantics of object\u2013action relations by observation"],"prefix":"10.1177","volume":"30","author":[{"given":"Eren Erdal","family":"Aksoy","sequence":"first","affiliation":[{"name":"Bernstein Center for Computational Neuroscience, University of G\u00f6ttingen, III. Physikalisches Institut, G\u00f6ttingen, Germany"}]},{"given":"Alexey","family":"Abramov","sequence":"additional","affiliation":[{"name":"Bernstein Center for Computational Neuroscience, University of G\u00f6ttingen, III. Physikalisches Institut, G\u00f6ttingen, Germany"}]},{"given":"Johannes","family":"D\u00f6rr","sequence":"additional","affiliation":[{"name":"Bernstein Center for Computational Neuroscience, University of G\u00f6ttingen, III. Physikalisches Institut, G\u00f6ttingen, Germany"}]},{"given":"Kejun","family":"Ning","sequence":"additional","affiliation":[{"name":"Bernstein Center for Computational Neuroscience, University of G\u00f6ttingen, III. Physikalisches Institut, G\u00f6ttingen, Germany"}]},{"given":"Babette","family":"Dellen","sequence":"additional","affiliation":[{"name":"Bernstein Center for Computational Neuroscience, University of G\u00f6ttingen, III. Physikalisches Institut, G\u00f6ttingen, Germany"},{"name":"Institut de Rob\u00f2tica i Inform\u00e0tica Industrial (CSIC-UPC), Barcelona, Spain"}]},{"given":"Florentin","family":"W\u00f6rg\u00f6tter","sequence":"additional","affiliation":[{"name":"Bernstein Center for Computational Neuroscience, University of G\u00f6ttingen, III. Physikalisches Institut, G\u00f6ttingen, Germany"}]}],"member":"179","published-online":{"date-parts":[[2011,8,1]]},"reference":[{"key":"bibr1-0278364911410459","unstructured":"Abramov A, Aksoy EE, D\u00f6rr J, Pauwels K, W\u00f6rg\u00f6tter F, Dellen B (2010) 3D semantic representation of actions from efficient stereo-image-sequence segmentation on GPUs. In 3DPVT."},{"key":"bibr2-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2010.5509319"},{"key":"bibr3-0278364911410459","first-page":"270","author":"Belhumeur PN","year":"1996","journal-title":"IEEE CVPR"},{"key":"bibr4-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1016\/S1364-6613(02)02016-8"},{"key":"bibr5-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2004.1389828"},{"key":"bibr6-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102365"},{"key":"bibr7-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1145\/1228716.1228751"},{"key":"bibr8-0278364911410459","first-page":"727","volume-title":"Proceedings of the Seventeenth International Conference on Machine Learning","author":"Dan Pelleg AM","year":"2000"},{"key":"bibr9-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-03832-7_18"},{"key":"bibr10-0278364911410459","doi-asserted-by":"publisher","DOI":"10.3390\/s91109355"},{"key":"bibr11-0278364911410459","doi-asserted-by":"publisher","DOI":"10.5244\/C.23.96"},{"key":"bibr12-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2003.1211479"},{"key":"bibr13-0278364911410459","first-page":"127","volume-title":"Perceiving, acting, and knowing","author":"Gibson JJ","year":"1977"},{"key":"bibr14-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2010.144"},{"key":"bibr15-0278364911410459","volume-title":"Proceedings of AAAI","author":"Hakeem A","year":"2005"},{"key":"bibr16-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1016\/0167-2789(90)90087-6"},{"key":"bibr17-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1007\/s00221-009-1953-8"},{"key":"bibr18-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-008-0137-5"},{"key":"bibr19-0278364911410459","first-page":"487","volume-title":"Proceedings of the 15th British Machine Vision Conference","author":"Hongeng S","year":"2004"},{"key":"bibr20-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2002.1014739"},{"key":"bibr21-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-88688-4_25"},{"key":"bibr22-0278364911410459","author":"Kr\u00fcger N","year":"2010","journal-title":"Robotics and Autonomous Systems"},{"key":"bibr23-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2007.4409105"},{"key":"bibr24-0278364911410459","first-page":"773","volume-title":"Proceedings of the 19th International Joint Conference on Artificial Intelligence","author":"Liao L","year":"2005"},{"key":"bibr25-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1023\/B:VISI.0000029664.99615.94"},{"key":"bibr26-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1007\/11550822_77"},{"key":"bibr27-0278364911410459","first-page":"195","volume-title":"Machine Intelligence","volume":"4","author":"McCarthy J","year":"1969"},{"key":"bibr28-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1145\/1409635.1409641"},{"key":"bibr29-0278364911410459","volume-title":"Geometric Invariance in Computer Vision","author":"Mundy J","year":"1992"},{"key":"bibr30-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1007\/11957959_1"},{"key":"bibr31-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1007\/BF01421486"},{"key":"bibr32-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-007-0122-4"},{"key":"bibr33-0278364911410459","author":"Ning K","year":"2010","journal-title":"IEEE Trans Robotics"},{"key":"bibr34-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2006.264"},{"key":"bibr35-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1109\/IRDS.2002.1043877"},{"key":"bibr36-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2007.09.009"},{"key":"bibr37-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2008.4563090"},{"key":"bibr38-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-74272-2_13"},{"key":"bibr39-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.neuro.27.070203.144230"},{"key":"bibr40-0278364911410459","first-page":"213","volume-title":"International Conference on Computer Vision Theory and Applications","author":"Sabatini S","year":"2007"},{"key":"bibr41-0278364911410459","volume-title":"Proceedings of the International Conference on Cognitive Systems (Cogsys 2008)","author":"Shylo N","year":"2009"},{"key":"bibr42-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2003.1238663"},{"key":"bibr43-0278364911410459","first-page":"606","volume-title":"Proceedings 18th European Conference on Artificial Intelligence","author":"Sridhar M","year":"2008"},{"key":"bibr44-0278364911410459","unstructured":"Sumsi MF (2008) Theory and Algorithms on the Median Graph. Application to Graph-based Classification and Clustering. PhD thesis, Universitat Autonoma de Barcelona."},{"key":"bibr45-0278364911410459","volume-title":"Animal Intelligence","author":"Thorndike E","year":"1911"},{"key":"bibr46-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1364\/JOSAA.20.001407"},{"key":"bibr47-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1162\/jocn.1991.3.1.71"},{"key":"bibr48-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1163\/156855307782506156"},{"key":"bibr49-0278364911410459","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2008.06.011"}],"container-title":["The International Journal of Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0278364911410459","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/0278364911410459","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T10:17:11Z","timestamp":1777457831000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/0278364911410459"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2011,8,1]]},"references-count":49,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2011,9]]}},"alternative-id":["10.1177\/0278364911410459"],"URL":"https:\/\/doi.org\/10.1177\/0278364911410459","relation":{},"ISSN":["0278-3649","1741-3176"],"issn-type":[{"value":"0278-3649","type":"print"},{"value":"1741-3176","type":"electronic"}],"subject":[],"published":{"date-parts":[[2011,8,1]]}}}