{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2023,8,17]],"date-time":"2023-08-17T05:05:17Z","timestamp":1692248717497},"reference-count":35,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,8,16]],"date-time":"2023-08-16T00:00:00Z","timestamp":1692144000000},"content-version":"vor","delay-in-days":227,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,8,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Contrast consistency, the ability of a model to make consistently correct predictions in the presence of perturbations, is an essential aspect in NLP. While studied in tasks such as sentiment analysis and reading comprehension, it remains unexplored in open-domain question answering (OpenQA) due to the difficulty of collecting perturbed questions that satisfy factuality requirements. In this work, we collect minimally edited questions as challenging contrast sets to evaluate OpenQA models. Our collection approach combines both human annotation and large language model generation. We find that the widely used dense passage retriever (DPR) performs poorly on our contrast sets, despite fitting the training set well and performing competitively on standard test sets. To address this issue, we introduce a simple and effective query-side contrastive loss with the aid of data augmentation to improve DPR training. Our experiments on the contrast sets demonstrate that DPR\u2019s contrast consistency is improved without sacrificing its accuracy on the standard test sets.1<\/jats:p>","DOI":"10.1162\/tacl_a_00591","type":"journal-article","created":{"date-parts":[[2023,8,16]],"date-time":"2023-08-16T19:47:20Z","timestamp":1692215240000},"page":"1082-1096","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions"],"prefix":"10.1162","volume":"11","author":[{"given":"Zhihan","family":"Zhang","sequence":"first","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN, USA. zzhang23@nd.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wenhao","family":"Yu","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN, USA. wyu1@nd.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zheng","family":"Ning","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN, USA. zning@nd.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mingxuan","family":"Ju","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN, USA. mju2@nd.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Meng","family":"Jiang","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN, USA. mjiang2@nd.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2023,8,15]]},"reference":[{"key":"2023081619465480900_bib1","article-title":"Language models are few-shot learners","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020","author":"Brown","year":"2020"},{"key":"2023081619465480900_bib2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1171","article-title":"Reading Wikipedia to answer open-domain questions","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017","author":"Chen","year":"2017"},{"key":"2023081619465480900_bib3","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019","author":"Devlin","year":"2019"},{"key":"2023081619465480900_bib4","doi-asserted-by":"publisher","first-page":"854","DOI":"10.18653\/v1\/2021.findings-emnlp.73","article-title":"R2-D2: A modular baseline for open-domain question answering","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021","author":"Fajcik","year":"2021"},{"key":"2023081619465480900_bib5","doi-asserted-by":"publisher","first-page":"981","DOI":"10.18653\/v1\/2021.emnlp-main.75","article-title":"Condenser: A pre-training architecture for dense retrieval","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021","author":"Gao","year":"2021"},{"key":"2023081619465480900_bib6","doi-asserted-by":"publisher","first-page":"1307","DOI":"10.18653\/v1\/2020.findings-emnlp.117","article-title":"Evaluating models\u2019 local decision boundaries via contrast sets","volume-title":"Findings of the Association for Computational Linguistics: EMNLP","author":"Gardner","year":"2020"},{"key":"2023081619465480900_bib7","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2112.09118","article-title":"Towards unsupervised dense information retrieval with contrastive learning","author":"Izacard","year":"2021","journal-title":"ArXiv preprint"},{"key":"2023081619465480900_bib8","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.74","article-title":"Leveraging passage retrieval with generative models for open domain question answering","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021","author":"Izacard","year":"2021"},{"key":"2023081619465480900_bib9","doi-asserted-by":"publisher","first-page":"6769","DOI":"10.18653\/v1\/2020.emnlp-main.550","article-title":"Dense passage retrieval for open-domain question answering","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020","author":"Karpukhin","year":"2020"},{"key":"2023081619465480900_bib10","article-title":"Learning the difference that makes a difference with counterfactually-augmented data","volume-title":"8th International Conference on Learning Representations, ICLR 2020","author":"Kaushik","year":"2020"},{"key":"2023081619465480900_bib11","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2022.emnlp-main.285","article-title":"Fie: Building a global probability space by leveraging early fusion in encoder for open-domain question answering","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022","author":"Kedia","year":"2022"},{"key":"2023081619465480900_bib12","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1145\/3397271.3401075","article-title":"Colbert: Efficient and effective passage search via contextualized late interaction over BERT","volume-title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020","author":"Khattab","year":"2020"},{"key":"2023081619465480900_bib13","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1162\/tacl_a_00276","article-title":"Natural questions: A benchmark for question answering research","volume":"7","author":"Kwiatkowski","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023081619465480900_bib14","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020","author":"Lewis","year":"2020"},{"key":"2023081619465480900_bib15","doi-asserted-by":"publisher","first-page":"1098","DOI":"10.1162\/tacl_a_00415","article-title":"PAQ: 65 million probably-asked questions and what you can do with them","volume":"9","author":"Lewis","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2023081619465480900_bib16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2211.05110","article-title":"Large language models with controllable working memory","author":"Li","year":"2022","journal-title":"ArXiv preprint"},{"key":"2023081619465480900_bib17","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1907.11692","article-title":"Roberta: A robustly optimized BERT pretraining approach","author":"Liu","year":"2019","journal-title":"ArXiv preprint"},{"key":"2023081619465480900_bib18","doi-asserted-by":"publisher","first-page":"7052","DOI":"10.18653\/v1\/2021.emnlp-main.565","article-title":"Entity-based knowledge conflicts in question answering","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021","author":"Longpre","year":"2021"},{"key":"2023081619465480900_bib19","doi-asserted-by":"publisher","first-page":"5783","DOI":"10.18653\/v1\/2020.emnlp-main.466","article-title":"Ambigqa: Answering ambiguous open-domain questions","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020","author":"Min","year":"2020"},{"key":"2023081619465480900_bib20","doi-asserted-by":"publisher","first-page":"314","DOI":"10.18653\/v1\/W19-5333","article-title":"Facebook fair\u2019s WMT19 news translation task submission","volume-title":"Proceedings of the Fourth Conference on Machine Translation, WMT 2019","author":"Ng","year":"2019"},{"key":"2023081619465480900_bib21","doi-asserted-by":"crossref","first-page":"9844","DOI":"10.18653\/v1\/2022.emnlp-main.669","article-title":"Large dual encoders are generalizable retrievers","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022","author":"Ni","year":"2022"},{"key":"2023081619465480900_bib22","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1807.03748","article-title":"Representation learning with contrastive predictive coding","author":"van den Oord","year":"2018","journal-title":"Arxiv preprint"},{"key":"2023081619465480900_bib23","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2203.02155","article-title":"Training language models to follow instructions with human feedback","author":"Ouyang","year":"2022","journal-title":"ArXiv preprint"},{"key":"2023081619465480900_bib24","doi-asserted-by":"publisher","first-page":"1670","DOI":"10.18653\/v1\/2022.acl-long.117","article-title":"Retrieval-guided counterfactual generation for QA","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022","author":"Paranjape","year":"2022"},{"key":"2023081619465480900_bib25","doi-asserted-by":"publisher","first-page":"3574","DOI":"10.18653\/v1\/2022.naacl-main.261","article-title":"Exposing the limits of video-text models through contrast sets","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022","author":"Park","year":"2022"},{"issue":"4","key":"2023081619465480900_bib26","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1561\/1500000019","article-title":"The probabilistic relevance framework: BM25 and beyond","volume":"3","author":"Robertson","year":"2009","journal-title":"Foundations and Trends\u00ae in Information Retrieval"},{"key":"2023081619465480900_bib27","doi-asserted-by":"publisher","first-page":"3194","DOI":"10.18653\/v1\/2022.acl-long.228","article-title":"Tailor: Generating and perturbing text with semantic controls","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022","author":"Ross","year":"2022"},{"key":"2023081619465480900_bib28","doi-asserted-by":"publisher","first-page":"815","DOI":"10.1109\/CVPR.2015.7298682","article-title":"Facenet: A unified embedding for face recognition and clustering","volume-title":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015","author":"Schroff","year":"2015"},{"key":"2023081619465480900_bib29","first-page":"479","article-title":"OPUS-MT - Building open translation services for the world","volume-title":"Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, EAMT 2020","author":"Tiedemann","year":"2020"},{"key":"2023081619465480900_bib30","doi-asserted-by":"publisher","first-page":"353","DOI":"10.18653\/v1\/W18-5446","article-title":"GLUE: A multi-task benchmark and analysis platform for natural language understanding","volume-title":"7th International Conference on Learning Representations, ICLR 2019","author":"Wang","year":"2019"},{"key":"2023081619465480900_bib31","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2203.11171","article-title":"Self-consistency improves chain of thought reasoning in language models","author":"Wang","year":"2022","journal-title":"ArXiv preprint"},{"key":"2023081619465480900_bib32","doi-asserted-by":"publisher","first-page":"6707","DOI":"10.18653\/v1\/2021.acl-long.523","article-title":"Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL\/IJCNLP 2021","author":"Tongshuang","year":"2021"},{"key":"2023081619465480900_bib33","doi-asserted-by":"publisher","first-page":"5496","DOI":"10.18653\/v1\/2021.emnlp-main.447","article-title":"Connecting attributions and QA model behavior on realistic counterfactuals","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021","author":"Xi","year":"2021"},{"key":"2023081619465480900_bib34","article-title":"Generate rather than retrieve: Large language models are strong context generators","volume-title":"11th International Conference on Learning Representations, ICLR 2023","author":"Wenhao","year":"2023"},{"key":"2023081619465480900_bib35","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2101.00774","article-title":"Retrieving and reading: A comprehensive survey on open-domain question answering","author":"Zhu","year":"2021","journal-title":"ArXiv preprint"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00591\/2154483\/tacl_a_00591.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00591\/2154483\/tacl_a_00591.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,16]],"date-time":"2023-08-16T19:47:31Z","timestamp":1692215251000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00591\/117218\/Exploring-Contrast-Consistency-of-Open-Domain"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":35,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00591","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}