{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T11:25:57Z","timestamp":1783337157898,"version":"3.54.6"},"reference-count":46,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2019,11]]},"abstract":"<jats:p> Adversarial evaluation stress-tests a model\u2019s understanding of natural language. Because past approaches expose superficial patterns, the resulting adversarial examples are limited in complexity and diversity. We propose human- in-the-loop adversarial generation, where human authors are guided to break models. We aid the authors with interpretations of model predictions through an interactive user interface. We apply this generation framework to a question answering task called Quizbowl, where trivia enthusiasts craft adversarial questions. The resulting questions are validated via live human\u2013computer matches: Although the questions appear ordinary to humans, they systematically stump neural and information retrieval models. The adversarial questions cover diverse phenomena from multi-hop reasoning to entity type distractors, exposing open challenges in robust question answering. <\/jats:p>","DOI":"10.1162\/tacl_a_00279","type":"journal-article","created":{"date-parts":[[2019,7,26]],"date-time":"2019-07-26T19:54:42Z","timestamp":1564170882000},"page":"387-401","source":"Crossref","is-referenced-by-count":64,"title":["Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering"],"prefix":"10.1162","volume":"7","author":[{"given":"Eric","family":"Wallace","sequence":"first","affiliation":[{"name":"EE and UMIACS, University of Maryland."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pedro","family":"Rodriguez","sequence":"additional","affiliation":[{"name":"CS, UMIACS, University of Maryland."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shi","family":"Feng","sequence":"additional","affiliation":[{"name":"CS, UMIACS, University of Maryland."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ikuya","family":"Yamada","sequence":"additional","affiliation":[{"name":"Studio Ousia."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jordan","family":"Boyd-Graber","sequence":"additional","affiliation":[{"name":"CS, iSchool, UMIACS, LSC, University of Maryland."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"bib1","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Belinkov Yonatan","year":"2018"},{"key":"bib2","first-page":"49","volume-title":"Transactions of the Association for Computational Linguistics","volume":"7","author":"Belinkov Yonatan","year":"2019"},{"key":"bib3","volume-title":"Human-Computer Question Answering: The Case for Quizbowl","author":"Boyd-Graber Jordan","year":"2018"},{"key":"bib4","author":"Case Nicky","year":"2018","journal-title":"Journal of Design and Science"},{"key":"bib5","volume-title":"Proceedings of the Association for Computational Linguistics","author":"Chen Danqi","year":"2016"},{"key":"bib6","volume-title":"Proceedings of Empirical Methods in Natural Language Processing","author":"Cho Kyunghyun","year":"2014"},{"key":"bib7","volume-title":"Conference of the North American Chapter of the Association for Computational Linguistics","author":"Devlin Jacob","year":"2018"},{"key":"bib8","volume-title":"Conference of the North American Chapter of the Association for Computational Linguistics","author":"Dua Dheeru","year":"2019"},{"key":"bib9","volume-title":"Proceedings of the Association for Computational Linguistics","author":"Ebrahimi Javid","year":"2018"},{"key":"bib10","volume-title":"Proceedings of the First Workshop on Building Linguistically Generalizable NLP Systems","author":"Ettinger Allyson","year":"2017"},{"issue":"3","key":"bib11","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1609\/aimag.v31i3.2303","volume":"31","author":"Ferrucci David","year":"2010","journal-title":"AI Magazine"},{"key":"bib12","volume-title":"Proceedings of the Association for Computational Linguistics","author":"Glockner Max","year":"2018"},{"key":"bib13","volume-title":"Elasticsearch: The Definitive Guide","author":"Gormley Clinton","year":"2015"},{"key":"bib14","volume-title":"North American Association for Computational Linguistics","author":"Guha Anupam","year":"2015"},{"key":"bib15","volume-title":"Conference of the North American Chapter of the Association for Computational Linguistics","author":"Gururangan Suchin","year":"2018"},{"key":"bib16","first-page":"4","volume":"21","author":"Hardwick Elizabeth","year":"1967","journal-title":"The New York Review of Books"},{"key":"bib17","volume-title":"Prisoner of Trebekistan: A Decade in Jeopardy!","author":"Harris Bob","year":"2006"},{"key":"bib18","volume-title":"Conference of the North American Chapter of the Association for Computational Linguistics","author":"He He","year":"2016"},{"issue":"4","key":"bib19","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1017\/S1351324901002807","volume":"7","author":"Hirschman Lynette","year":"2001","journal-title":"Natural Language Engineering"},{"key":"bib20","volume-title":"Proceedings of the Association for Computational Linguistics","author":"Iyyer Mohit","year":"2015"},{"key":"bib21","volume-title":"Conference of the North American Chapter of the Association for Computational Linguistics","author":"Iyyer Mohit","year":"2018"},{"key":"bib22","volume-title":"Brainiac: Adventures in the Curious, Competitive, Compulsive World of Trivia Buffs","author":"Jennings Ken","year":"2006"},{"key":"bib23","volume-title":"Proceedings of Empirical Methods in Natural Language Processing","author":"Jia Robin","year":"2017"},{"key":"bib25","volume-title":"Proceedings of Empirical Methods in Natural Language Processing","author":"Kaushik Divyansh","year":"2018"},{"key":"bib26","first-page":"2019","volume-title":"Transactions of the Association for Computational Linguistics","volume":"7","author":"Kwiatkowski Tom","year":"2019"},{"key":"bib27","author":"Li Jiwei","year":"2016","journal-title":"arXiv preprint arXiv: 1612.08220"},{"key":"bib29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.dsp.2017.10.011","volume":"73","author":"Montavon Gr\u00e9goire","year":"2018","journal-title":"Digital Signal Processing"},{"key":"bib30","volume-title":"Proceedings of the Association for Computational Linguistics","author":"Mudrakarta Pramod Kaushik","year":"2018"},{"key":"bib31","volume-title":"Proceedings of International Conference on Computational Linguistics","author":"Naik Aakanksha","year":"2018"},{"key":"bib32","volume-title":"Conference on Computer Supported Cooperative Work and Social Computing","author":"Oeldorf-Hirsch Anne","year":"2014"},{"key":"bib33","volume-title":"Proceedings of the Association for Computational Linguistics","author":"Paperno Denis","year":"2016"},{"key":"bib34","volume-title":"Proceedings of Empirical Methods in Natural Language Processing","author":"Pennington Jeffrey","year":"2014"},{"key":"bib35","volume-title":"Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Ribeiro Marco Tulio","year":"2016"},{"key":"bib36","volume-title":"Proceedings of the Association for Computational Linguistics","author":"Ribeiro Marco Tulio","year":"2018"},{"key":"bib37","volume-title":"Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security","author":"Ruef Andrew","year":"2016"},{"key":"bib38","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Simonyan Karen","year":"2014"},{"key":"bib39","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Szegedy Christian","year":"2013"},{"key":"bib40","author":"Vinyals Oriol","year":"2017","journal-title":"arXiv preprint arXiv:1708.04782"},{"key":"bib41","volume-title":"EMNLP 2018 Workshop on Analyzing and Interpreting Neural Networks for NLP","author":"Wallace Eric","year":"2018"},{"key":"bib42","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Wang Alex","year":"2019"},{"issue":"2","key":"bib43","first-page":"59","volume":"28","author":"Watson Ritchie D.","year":"1996","journal-title":"The Southern Literary Journal"},{"key":"bib44","author":"Yamada Ikuya","year":"2018","journal-title":"arXiv preprint arXiv:1803.08652"},{"key":"bib45","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Adams Wei Yu","year":"2018"},{"key":"bib46","volume-title":"Proceedings of Empirical Methods in Natural Language Processing","author":"Zellers Rowan","year":"2018"},{"key":"bib47","author":"Zhang Sheng","year":"2018","journal-title":"arXiv preprint arXiv:1810.12885"},{"key":"bib48","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Zhao Zhengli","year":"2018"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00279","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:26Z","timestamp":1615585166000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/43493"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11]]},"references-count":46,"alternative-id":["10.1162\/tacl_a_00279"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00279","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11]]}}}