{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T05:58:04Z","timestamp":1670392684540},"reference-count":50,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2022,12,6]],"date-time":"2022-12-06T00:00:00Z","timestamp":1670284800000},"content-version":"vor","delay-in-days":339,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,11,28]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Abstraction is a core tenet of human cognition and communication. When composing natural language instructions, humans naturally evoke abstraction to convey complex procedures in an efficient and concise way. Yet, interpreting and grounding abstraction expressed in NL has not yet been systematically studied in NLP, with no accepted benchmarks specifically eliciting abstraction in NL. In this work, we set the foundation for a systematic study of processing and grounding abstraction in NLP. First, we deliver a novel abstraction elicitation method and present Hexagons, a 2D instruction-following game. Using Hexagons we collected over 4k naturally occurring visually-grounded instructions rich with diverse types of abstractions. From these data, we derive an instruction-to-execution task and assess different types of neural models. Our results show that contemporary models and modeling practices are substantially inferior to human performance, and that model performance is inversely correlated with the level of abstraction, showing less satisfying performance on higher levels of abstraction. These findings are consistent across models and setups, confirming that abstraction is a challenging phenomenon deserving further attention and study in NLP\/AI research.<\/jats:p>","DOI":"10.1162\/tacl_a_00522","type":"journal-article","created":{"date-parts":[[2022,12,6]],"date-time":"2022-12-06T16:47:59Z","timestamp":1670345279000},"page":"1341-1356","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Draw Me a Flower: Processing and Grounding Abstraction in Natural Language"],"prefix":"10.1162","volume":"10","author":[{"given":"Royi","family":"Lachmy","sequence":"first","affiliation":[{"name":"Bar-Ilan University, Ramat Gan, Israel"},{"name":"Allen Institute for Artificial Intelligence, Tel Aviv, Israel. royi.lachmy@biu.ac.il"}]},{"given":"Valentina","family":"Pyatkin","sequence":"additional","affiliation":[{"name":"Bar-Ilan University, Ramat Gan, Israel"},{"name":"Allen Institute for Artificial Intelligence, Tel Aviv, Israel. valpyatkin@gmail.com"}]},{"given":"Avshalom","family":"Manevich","sequence":"additional","affiliation":[{"name":"Bar-Ilan University, Ramat Gan, Israel. avshalomman@gmail.com"}]},{"given":"Reut","family":"Tsarfaty","sequence":"additional","affiliation":[{"name":"Bar-Ilan University, Ramat Gan, Israel"},{"name":"Allen Institute for Artificial Intelligence, Tel Aviv, Israel. reut.tsarfaty@biu.ac.il"}]}],"member":"281","published-online":{"date-parts":[[2022,11,28]]},"reference":[{"issue":"4","key":"2022120616374412200_bib1","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1177\/002383099103400404","article-title":"The HCRC map task corpus","volume":"34","author":"Anderson","year":"1991","journal-title":"Language and Speech"},{"key":"2022120616374412200_bib2","doi-asserted-by":"publisher","first-page":"3674","DOI":"10.1109\/CVPR.2018.00387","article-title":"Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments","volume-title":"2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Anderson","year":"2018"},{"issue":"2","key":"2022120616374412200_bib3","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1080\/08993408.2020.1866939","article-title":"A principled approach to designing computational thinking concepts and practices assessments for upper elementary grades","volume":"31","author":"Basu","year":"2021","journal-title":"Computer Science Education"},{"key":"2022120616374412200_bib4","article-title":"Towards a dataset for human computer communication via grounded language acquisition","volume-title":"Symbiotic Cognitive Systems, Papers from the 2016 AAAI Workshop, Phoenix, Arizona, USA, February 13, 2016","author":"Bisk","year":"2016"},{"key":"2022120616374412200_bib5","first-page":"5028","article-title":"Learning interpretable spatial operations in a rich 3d blocks world","volume-title":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2\u20137, 2018","author":"Bisk","year":"2018"},{"key":"2022120616374412200_bib6","doi-asserted-by":"publisher","first-page":"751","DOI":"10.18653\/v1\/N16-1089","article-title":"Natural language communication with robots","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Bisk","year":"2016"},{"issue":"5","key":"2022120616374412200_bib7","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1177\/1745691613497964","article-title":"There are many ways to see the forest for the trees: A tour guide for abstraction","volume":"8","author":"Burgoon","year":"2013","journal-title":"Perspectives on Psychological Science"},{"key":"2022120616374412200_bib8","doi-asserted-by":"publisher","first-page":"12530","DOI":"10.1109\/CVPR.2019.01282","article-title":"Touchdown: Natural language navigation and spatial reasoning in visual street environments","volume-title":"2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Chen","year":"2019"},{"key":"2022120616374412200_bib9","article-title":"BabyAI: A platform to study the sample efficiency of grounded language learning","volume-title":"International Conference on Learning Representations","author":"Chevalier-Boisvert","year":"2018"},{"issue":"1","key":"2022120616374412200_bib10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0010-0277(86)90010-7","article-title":"Referring as a collaborative process","volume":"22","author":"Clark","year":"1986","journal-title":"Cognition"},{"key":"2022120616374412200_bib11","volume-title":"Demystifying Computational Thinking for Non-computer Scientists","author":"Cuny","year":"2010"},{"issue":"1","key":"2022120616374412200_bib12","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1145\/63238.63239","article-title":"Computing as a discipline","volume":"32","author":"Denning","year":"1989","journal-title":"Communication of the ACM"},{"key":"2022120616374412200_bib13","doi-asserted-by":"publisher","DOI":"10.1145\/1283920.1283927","article-title":"The humble programmer","author":"Dijkstra","year":"1972","journal-title":"ACM Turing Award Lectures"},{"key":"2022120616374412200_bib14","article-title":"Ew dijkstra quotes","author":"Dijkstra","year":"1  2021"},{"key":"2022120616374412200_bib15","doi-asserted-by":"publisher","first-page":"351","DOI":"10.18653\/v1\/P18-1033","article-title":"Improving text-to-SQL evaluation methodology","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Finegan-Dollak","year":"2018"},{"key":"2022120616374412200_bib16","doi-asserted-by":"publisher","first-page":"2051","DOI":"10.18653\/v1\/P18-1191","article-title":"Large-scale QA-SRL parsing","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"FitzGerald","year":"2018"},{"key":"2022120616374412200_bib17","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1145\/3017680.3017801","article-title":"Multiple levels of abstraction in algorithmic problem solving","volume-title":"Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education","author":"Ginat","year":"2017"},{"key":"2022120616374412200_bib18","doi-asserted-by":"publisher","first-page":"864","DOI":"10.18653\/v1\/2022.acl-short.96","article-title":"(un)solving morphological inflection: Lemma overlap artificially inflates models\u2019 performance","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Goldman","year":"2022"},{"key":"2022120616374412200_bib19","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1163\/9789004368811_003","article-title":"Logic and conversation","volume-title":"Speech Acts","author":"Grice","year":"1975"},{"issue":"1","key":"2022120616374412200_bib20","doi-asserted-by":"publisher","first-page":"38","DOI":"10.3102\/0013189X12463051","article-title":"Computational thinking in k\u201312: A review of the state of the field","volume":"42","author":"Grover","year":"2013","journal-title":"Educational Researcher"},{"key":"2022120616374412200_bib21","unstructured":"Kilem L.\n              Gwet\n            \n          . 2015. On Krippendorff\u2019s alpha coefficient. Accessed: 1 June 2022."},{"key":"2022120616374412200_bib22","doi-asserted-by":"publisher","first-page":"1895","DOI":"10.18653\/v1\/P19-1184","article-title":"The PhotoBook dataset: Building common ground through visually- grounded dialogue","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Haber","year":"2019"},{"issue":"2","key":"2022120616374412200_bib23","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1023\/B:EAIT.0000027926.99053.6f","article-title":"High-school students\u2019 attitudes regarding procedural abstraction","volume":"9","author":"Haberman","year":"2004","journal-title":"Education and Information Technologies"},{"key":"2022120616374412200_bib24","article-title":"Deberta: Decoding-enhanced bert with disentangled attention","volume-title":"International Conference on Learning Representations","author":"He","year":"2020"},{"key":"2022120616374412200_bib25","article-title":"Span- based semantic parsing for compositional generalization","author":"Herzig","year":"2020","journal-title":"arXiv preprint arXiv:2009.06040. Version 2"},{"key":"2022120616374412200_bib26","doi-asserted-by":"publisher","first-page":"2589","DOI":"10.18653\/v1\/2020.acl-main.232","article-title":"Learning to execute instructions in a Minecraft dialogue","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Jayannavar","year":"2020"},{"key":"2022120616374412200_bib27","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1162\/tacl_a_00037","article-title":"Planning, inference, and pragmatics in sequential language games","volume":"6","author":"Khani","year":"2018","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2022120616374412200_bib28","doi-asserted-by":"crossref","first-page":"6495","DOI":"10.18653\/v1\/P19-1651","article-title":"CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Kim","year":"2019"},{"key":"2022120616374412200_bib29","doi-asserted-by":"publisher","first-page":"174","DOI":"10.1145\/1822090.1822140","article-title":"Teaching abstraction in introductory courses","volume-title":"Proceedings of the Fifteenth Annual Conference on Innovation and Technology in Computer Science Education","author":"Koppelman","year":"2010"},{"key":"2022120616374412200_bib30","volume-title":"Content Analysis: An Introduction to Its Methodology (2nd ed.), Chapter 11","author":"Krippendorff","year":"2004"},{"key":"2022120616374412200_bib31","article-title":"Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing","author":"Liu","year":"2021","journal-title":"arXiv preprint arXiv:2107.13586. Version 1"},{"key":"2022120616374412200_bib32","doi-asserted-by":"publisher","first-page":"1456","DOI":"10.18653\/v1\/P16-1138","article-title":"Simpler context-dependent logical forms via model projections","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Long","year":"2016"},{"key":"2022120616374412200_bib33","first-page":"1475","article-title":"Walk the talk: Connecting language, knowledge, and action in route instructions","volume-title":"Proceedings of the 21st National Conference on Artificial Intelligence - Volume 2","author":"MacMahon","year":"2006"},{"key":"2022120616374412200_bib34","doi-asserted-by":"publisher","first-page":"2667","DOI":"10.18653\/v1\/D18-1287","article-title":"Mapping instructions to actions in 3D environments with visual goal prediction","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Misra","year":"2018"},{"key":"2022120616374412200_bib35","doi-asserted-by":"publisher","first-page":"6450","DOI":"10.18653\/v1\/D19-1681","article-title":"RUN through the streets: A new dataset and baseline models for realistic urban navigation","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Paz-Argaman","year":"2019"},{"key":"2022120616374412200_bib36","doi-asserted-by":"publisher","first-page":"44","DOI":"10.18653\/v1\/W17-2806","article-title":"Communication with robots using multilayer recurrent networks","volume-title":"Proceedings of the First Workshop on Language Grounding for Robotics","author":"Pi\u0161l","year":"2017"},{"key":"2022120616374412200_bib37","doi-asserted-by":"publisher","first-page":"2804","DOI":"10.18653\/v1\/2020.emnlp-main.224","article-title":"QADiscourse - discourse relations as QA pairs: Representation, crowdsourcing and baselines","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Pyatkin","year":"2020"},{"issue":"140","key":"2022120616374412200_bib38","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2022120616374412200_bib39","doi-asserted-by":"publisher","DOI":"10.4324\/9780429453755-5","article-title":"Designing an assessment of computational thinking abilities for young children","volume-title":"STEM in Early Childhood Education: How Science, Technology, Engineering, and Mathematics Strengthen Learning","author":"Relkin","year":"2019"},{"key":"2022120616374412200_bib40","doi-asserted-by":"publisher","first-page":"7008","DOI":"10.18653\/v1\/2020.acl-main.626","article-title":"Controlled crowdsourcing for high-quality QA-SRL annotation","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Roit","year":"2020"},{"issue":"1","key":"2022120616374412200_bib41","first-page":"34","article-title":"Computational thinking task design and assessment.","volume":"36","author":"Ructtinger","year":"2017","journal-title":"Scan: The Journal For Educators"},{"key":"2022120616374412200_bib42","doi-asserted-by":"publisher","first-page":"142","DOI":"10.1016\/j.edurev.2017.09.003","article-title":"Demystifying computational thinking","volume":"22","author":"Shute","year":"2017","journal-title":"Educational Research Review"},{"key":"2022120616374412200_bib43","doi-asserted-by":"publisher","first-page":"2119","DOI":"10.18653\/v1\/D19-1218","article-title":"Executing instructions in situated collaborative interactions","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Suhr","year":"2019"},{"issue":"01","key":"2022120616374412200_bib44","doi-asserted-by":"publisher","first-page":"7120","DOI":"10.1609\/aaai.v33i01.33017120","article-title":"A natural language corpus of common grounding under continuous and partially-observable context","volume":"33","author":"Udagawa","year":"2019","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2022120616374412200_bib45","doi-asserted-by":"publisher","first-page":"929","DOI":"10.18653\/v1\/P17-1086","article-title":"Naturalizing a programming language via interactive learning","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Wang","year":"2017"},{"key":"2022120616374412200_bib46","doi-asserted-by":"publisher","first-page":"2368","DOI":"10.18653\/v1\/P16-1224","article-title":"Learning language games through interaction","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Wang","year":"2016"},{"issue":"2","key":"2022120616374412200_bib47","first-page":"7","article-title":"Computational thinking\u2019s influence on research and education for all","volume":"25","author":"Wing","year":"2017","journal-title":"Italian Journal of Educational Technology"},{"key":"2022120616374412200_bib48","article-title":"Computational thinking\u2014what and why?","author":"Wing","year":"2011","journal-title":"CMU Research Notebook"},{"key":"2022120616374412200_bib49","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language processing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Wolf","year":"2020"},{"key":"2022120616374412200_bib50","first-page":"5180","article-title":"Beyond goldfish memory: Long-term open-domain conversation","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Jing","year":"2022"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00522\/2061241\/tacl_a_00522.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00522\/2061241\/tacl_a_00522.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,6]],"date-time":"2022-12-06T16:48:18Z","timestamp":1670345298000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00522\/114048\/Draw-Me-a-Flower-Processing-and-Grounding"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":50,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00522","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}