{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T14:08:55Z","timestamp":1769004535349,"version":"3.49.0"},"reference-count":72,"publisher":"SAGE Publications","issue":"12-14","license":[{"start":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T00:00:00Z","timestamp":1638316800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"International Max-Planck Research School for Intelligent Systems"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of Robotics Research"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:p> In this article, we propose deep visual reasoning, which is a convolutional recurrent neural network that predicts discrete action sequences from an initial scene image for sequential manipulation problems that arise, for example, in task and motion planning (TAMP). Typical TAMP problems are formalized by combining reasoning on a symbolic, discrete level (e.g., first-order logic) with continuous motion planning such as nonlinear trajectory optimization. The action sequences represent the discrete decisions on a symbolic level, which, in turn, parameterize a nonlinear trajectory optimization problem. Owing to the great combinatorial complexity of possible discrete action sequences, a large number of optimization\/motion planning problems have to be solved to find a solution, which limits the scalability of these approaches. To circumvent this combinatorial complexity, we introduce deep visual reasoning: based on a segmented initial image of the scene, a neural network directly predicts promising discrete action sequences such that ideally only one motion planning problem has to be solved to find a solution to the overall TAMP problem. Our method generalizes to scenes with many and varying numbers of objects, although being trained on only two objects at a time. This is possible by encoding the objects of the scene and the goal in (segmented) images as input to the neural network, instead of a fixed feature vector. We show that the framework can not only handle kinematic problems such as pick-and-place (as typical in TAMP), but also tool-use scenarios for planar pushing under quasi-static dynamic models. Here, the image-based representation enables generalization to other shapes than during training. Results show runtime improvements of several orders of magnitudes by, in many cases, removing the need to search over the discrete action sequences. <\/jats:p>","DOI":"10.1177\/02783649211056967","type":"journal-article","created":{"date-parts":[[2021,12,31]],"date-time":"2021-12-31T17:53:39Z","timestamp":1640973219000},"page":"1435-1466","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":13,"title":["Learning to solve sequential physical reasoning problems from a scene image"],"prefix":"10.1177","volume":"40","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8258-1659","authenticated-orcid":false,"given":"Danny","family":"Driess","sequence":"first","affiliation":[{"name":"Learning and Intelligent Systems, TU Berlin, Germany"},{"name":"Max-Planck Institute for Intelligent Systems, Stuttgart, Germany"}]},{"given":"Jung-Su","family":"Ha","sequence":"additional","affiliation":[{"name":"Learning and Intelligent Systems, TU Berlin, Germany"},{"name":"Max-Planck Institute for Intelligent Systems, Stuttgart, Germany"}]},{"given":"Marc","family":"Toussaint","sequence":"additional","affiliation":[{"name":"Learning and Intelligent Systems, TU Berlin, Germany"},{"name":"Max-Planck Institute for Intelligent Systems, Stuttgart, Germany"}]}],"member":"179","published-online":{"date-parts":[[2021,12,31]]},"reference":[{"key":"bibr1-02783649211056967","first-page":"8289","author":"Amos B","year":"2018","journal-title":"Advances in Neural Information Processing Systems"},{"key":"bibr2-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/IROS40897.2019.8967717"},{"key":"bibr3-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1177\/0278364911404092"},{"key":"bibr4-02783649211056967","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2017.XIII.031"},{"key":"bibr5-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2016.7487165"},{"key":"bibr6-02783649211056967","author":"Chitnis R","year":"2020","journal-title":"arXiv preprint arXiv:2007.13202"},{"key":"bibr7-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1177\/0278364918761570"},{"key":"bibr8-02783649211056967","author":"de Silva L","year":"2013","journal-title":"arXiv preprint arXiv:1307.1482"},{"key":"bibr9-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9197414"},{"key":"bibr10-02783649211056967","volume-title":"International Conference on Learning Representations ICLR","author":"Dosovitskiy A","year":"2017"},{"key":"bibr11-02783649211056967","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2020.XVI.003"},{"key":"bibr12-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9560934"},{"key":"bibr13-02783649211056967","volume-title":"Proceedings of the Annual Conference on Robot Learning (CoRL)","author":"Driess D","year":"2021"},{"key":"bibr14-02783649211056967","volume-title":"RSS Workshop on Robust Task and Motion Planning","author":"Driess D","year":"2019"},{"key":"bibr15-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9197291"},{"key":"bibr16-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/IROS40897.2019.8967858"},{"key":"bibr17-02783649211056967","volume-title":"Conference on Robot Learning","author":"Ebert F","year":"2017"},{"key":"bibr18-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2017.7989324"},{"key":"bibr19-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2016.7487173"},{"key":"bibr20-02783649211056967","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)","author":"Garrett C","year":"2016"},{"key":"bibr21-02783649211056967","author":"Garrett CR","year":"2020","journal-title":"arXiv preprint arXiv:2010.01083"},{"key":"bibr22-02783649211056967","author":"Garrett CR","year":"2019","journal-title":"arXiv preprint arXiv:1911.04577"},{"key":"bibr23-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9196840"},{"key":"bibr24-02783649211056967","first-page":"8927","author":"Ha JS","year":"2018","journal-title":"Advances in Neural Information Processing Systems"},{"key":"bibr25-02783649211056967","author":"Hartmann VN","year":"2020","journal-title":"arXiv preprint arXiv:2003.07754"},{"key":"bibr26-02783649211056967","author":"Hartmann VN","year":"2021","journal-title":"arXiv preprint arXiv:2106.02489"},{"key":"bibr27-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.322"},{"key":"bibr28-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2018.8461175"},{"key":"bibr29-02783649211056967","volume-title":"Proceedings of the Workshop on Algorithmic Foundation Robotics (WAFR)","author":"Hogan FR","year":"2016"},{"key":"bibr30-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2018.8460730"},{"key":"bibr31-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2019.2901898"},{"key":"bibr32-02783649211056967","volume-title":"Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)","author":"Kaelbling LP","year":"2011"},{"key":"bibr33-02783649211056967","author":"Kase K","year":"2020","journal-title":"arXiv preprint arXiv:2003.03726"},{"key":"bibr34-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.12106"},{"key":"bibr35-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1177\/0278364919848837"},{"key":"bibr36-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9197409"},{"key":"bibr37-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1177\/0278364914545811"},{"key":"bibr38-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2012.6385972"},{"key":"bibr39-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04180-8_65"},{"key":"bibr40-02783649211056967","author":"Lange S","year":"2012","journal-title":"Proceedings of IJCNN"},{"key":"bibr41-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9197468"},{"key":"bibr42-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2014.6943079"},{"key":"bibr43-02783649211056967","author":"Manuelli L","year":"2019","journal-title":"arXiv preprint arXiv:1903.06684"},{"key":"bibr44-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.1985.1087242"},{"key":"bibr45-02783649211056967","author":"Mukherjee S","year":"2020","journal-title":"arXiv preprint arXiv:2011.08694"},{"key":"bibr46-02783649211056967","author":"Okada M","year":"2017","journal-title":"arXiv preprint arXiv:1706.09597"},{"key":"bibr47-02783649211056967","author":"Orthey A","year":"2020","journal-title":"arXiv preprint arXiv:2007.09435"},{"key":"bibr48-02783649211056967","first-page":"06170","volume":"1707","author":"Pascanu R","year":"2017","journal-title":"CoRR"},{"key":"bibr49-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2019.8793736"},{"key":"bibr50-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2019.8793260"},{"key":"bibr51-02783649211056967","author":"Qin Z","year":"2019","journal-title":"arXiv preprint arXiv:1910.11977"},{"key":"bibr52-02783649211056967","author":"Racani\u00e8re S","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"bibr53-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2019.2895845"},{"key":"bibr54-02783649211056967","volume-title":"International Conference on Machine Learning","author":"Silver D","year":"2017"},{"key":"bibr55-02783649211056967","author":"Silver T","year":"2020","journal-title":"arXiv preprint arXiv:2009.05613"},{"key":"bibr56-02783649211056967","author":"Simeonov A","year":"2020","journal-title":"arXiv preprint arXiv:2011.08177"},{"key":"bibr57-02783649211056967","first-page":"4739","volume-title":"International Conference on Machine Learning (ICML)","author":"Srinivas A","year":"2018"},{"key":"bibr58-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2014.6906922"},{"key":"bibr59-02783649211056967","author":"Suh H","year":"2020","journal-title":"arXiv preprint arXiv:2002.09093"},{"key":"bibr60-02783649211056967","first-page":"2154","author":"Tamar A","year":"2016","journal-title":"Advances in Neural Information Processing Systems"},{"key":"bibr61-02783649211056967","first-page":"1930","volume-title":"Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI)","author":"Toussaint M","year":"2015"},{"key":"bibr62-02783649211056967","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2018.XIV.044"},{"key":"bibr63-02783649211056967","author":"Toussaint M","year":"2020","journal-title":"arXiv preprint arXiv:2002.12780"},{"key":"bibr64-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2017.7989464"},{"key":"bibr65-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2020.3026963"},{"key":"bibr66-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2018.8594027"},{"key":"bibr67-02783649211056967","first-page":"2746","author":"Watter M","year":"2015","journal-title":"Advances in Neural Information Processing Systems"},{"key":"bibr68-02783649211056967","doi-asserted-by":"publisher","DOI":"10.1109\/LRA.2019.2894861"},{"key":"bibr69-02783649211056967","volume-title":"Conference on Robot Learning","author":"Wilson M","year":"2019"},{"key":"bibr70-02783649211056967","doi-asserted-by":"publisher","DOI":"10.15607\/RSS.2019.XV.001"},{"key":"bibr71-02783649211056967","author":"Xu D","year":"2020","journal-title":"arXiv preprint arXiv:2011.08424"},{"key":"bibr72-02783649211056967","author":"Zhu Y","year":"2020","journal-title":"arXiv preprint arXiv:2012.07277"}],"container-title":["The International Journal of Robotics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649211056967","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/02783649211056967","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/02783649211056967","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,2]],"date-time":"2025-03-02T08:52:31Z","timestamp":1740905551000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/02783649211056967"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12]]},"references-count":72,"journal-issue":{"issue":"12-14","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["10.1177\/02783649211056967"],"URL":"https:\/\/doi.org\/10.1177\/02783649211056967","relation":{},"ISSN":["0278-3649","1741-3176"],"issn-type":[{"value":"0278-3649","type":"print"},{"value":"1741-3176","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12]]}}}