{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T01:06:19Z","timestamp":1773795979869,"version":"3.50.1"},"reference-count":41,"publisher":"MIT Press - Journals","license":[{"start":{"date-parts":[[2021,9,23]],"date-time":"2021-09-23T00:00:00Z","timestamp":1632355200000},"content-version":"vor","delay-in-days":265,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,9,21]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Recent advancements in open-domain question answering (ODQA), that is, finding answers from large open-domain corpus like Wikipedia, have led to human-level performance on many datasets. However, progress in QA over book stories (Book QA) lags despite its similar task formulation to ODQA. This work provides a comprehensive and quantitative analysis about the difficulty of Book QA: (1) We benchmark the research on the NarrativeQA dataset with extensive experiments with cutting-edge ODQA techniques. This quantifies the challenges Book QA poses, as well as advances the published state-of-the-art with a \u223c7% absolute improvement on ROUGE-L. (2) We further analyze the detailed challenges in Book QA through human studies.1 Our findings indicate that the event-centric questions dominate this task, which exemplifies the inability of existing QA models to handle event-oriented scenarios.<\/jats:p>","DOI":"10.1162\/tacl_a_00411","type":"journal-article","created":{"date-parts":[[2021,9,24]],"date-time":"2021-09-24T03:51:19Z","timestamp":1632455479000},"page":"1032-1046","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":7,"title":["Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study"],"prefix":"10.1162","volume":"9","author":[{"given":"Xiangyang","family":"Mou","sequence":"first","affiliation":[{"name":"Rensselaer Polytechnic Institute & IBM, United States. moux4@rpi.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chenghao","family":"Yang","sequence":"additional","affiliation":[{"name":"Rensselaer Polytechnic Institute & IBM, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mo","family":"Yu","sequence":"additional","affiliation":[{"name":"Rensselaer Polytechnic Institute & IBM, United States. gflfof@gmail.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bingsheng","family":"Yao","sequence":"additional","affiliation":[{"name":"Rensselaer Polytechnic Institute & IBM, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaoxiao","family":"Guo","sequence":"additional","affiliation":[{"name":"Rensselaer Polytechnic Institute & IBM, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Saloni","family":"Potdar","sequence":"additional","affiliation":[{"name":"Rensselaer Polytechnic Institute & IBM, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hui","family":"Su","sequence":"additional","affiliation":[{"name":"Rensselaer Polytechnic Institute & IBM, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2021,9,21]]},"reference":[{"key":"2021092315280295800_bib1","first-page":"65","article-title":"Meteor: An automatic metric for MT evaluation with improved correlation with human judgments","volume-title":"Proceedings of the ACL 2005 Workshop","author":"Banerjee","year":"2005"},{"key":"2021092315280295800_bib2","doi-asserted-by":"publisher","first-page":"1870","DOI":"10.18653\/v1\/P17-1171","article-title":"Reading Wikipedia to answer open-domain questions","volume-title":"Proceedings of ACL 2017","author":"Chen","year":"2017"},{"key":"2021092315280295800_bib3","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.501","article-title":"Probabilistic assumptions matter: Improved models for distantly-supervised document-level question answering","author":"Cheng","year":"2020","journal-title":"arXiv preprint arXiv:2005.01898"},{"key":"2021092315280295800_bib4","first-page":"176","article-title":"Better hypothesis testing for statistical machine translation: Controlling for optimizer instability","volume-title":"Proceedings of ACL 2011","author":"Clark","year":"2011"},{"key":"2021092315280295800_bib5","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of NAACL-HLT 2019","author":"Devlin","year":"2019"},{"key":"2021092315280295800_bib6","article-title":"Quasar: Datasets for question answering by search and reading","author":"Dhingra","year":"2017","journal-title":"arXiv preprint arXiv: 1707.03904"},{"key":"2021092315280295800_bib7","doi-asserted-by":"publisher","first-page":"2185","DOI":"10.18653\/v1\/D19-1224","article-title":"Show your work: Improved reporting of experimental results","volume-title":"Proceedings of EMNLP-IJCNLP 2019","author":"Dodge","year":"2019"},{"key":"2021092315280295800_bib8","article-title":"SearchQA: A new q&a dataset augmented with context from a search engine","author":"Dunn","year":"2017","journal-title":"arXiv preprint arXiv:1704.05179"},{"issue":"5","key":"2021092315280295800_bib9","doi-asserted-by":"publisher","first-page":"378","DOI":"10.1037\/h0031619","article-title":"Measuring nominal scale agreement among many raters.","volume":"76","author":"Fleiss","year":"1971","journal-title":"Psychological Bulletin"},{"key":"2021092315280295800_bib10","doi-asserted-by":"publisher","first-page":"172","DOI":"10.18653\/v1\/D19-5823","article-title":"Extractive NarrativeQA with heuristic pre-training","volume-title":"Proceedings of the 2nd MRQA Workshop","author":"Frermann","year":"2019"},{"key":"2021092315280295800_bib11","article-title":"Realm: Retrieval-augmented language model pre- training","author":"Guu","year":"2020","journal-title":"arXiv preprint arXiv:2002.08909"},{"key":"2021092315280295800_bib12","article-title":"Reconsider: Re-ranking using span-focused cross-attention for open domain question answering","author":"Iyer","year":"2020","journal-title":"arXiv preprint arXiv:2010.10757"},{"key":"2021092315280295800_bib13","article-title":"Leveraging passage retrieval with generative models for open domain question answering","author":"Izacard","year":"2020","journal-title":"arXiv preprint arXiv:2007.01282"},{"key":"2021092315280295800_bib14","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.550","article-title":"Dense passage retrieval for open-domain question answering","volume-title":"Proceedings EMNLP 2020","author":"Karpukhin","year":"2020"},{"key":"2021092315280295800_bib15","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1162\/tacl_a_00023","article-title":"The NarrativeQA reading comprehension challenge","volume":"6","author":"Ko\u010disky\u0300","year":"2018","journal-title":"TACL"},{"key":"2021092315280295800_bib16","article-title":"Latent retrieval for weakly supervised open domain question answering","author":"Lee","year":"2019","journal-title":"Proceedings of ACL 2019"},{"key":"2021092315280295800_bib17","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.703","article-title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","author":"Lewis","year":"2019","journal-title":"arXiv preprint arXiv:1910.13461"},{"key":"2021092315280295800_bib18","first-page":"74","article-title":"ROUGE: A package for automatic evaluation of summaries","volume-title":"Text Summarization Branches Out","author":"Lin","year":"2004"},{"key":"2021092315280295800_bib19","first-page":"1736","article-title":"Denoising distantly supervised open-domain question answering","volume-title":"Proceedings of ACL 2018","author":"Lin","year":"2018"},{"key":"2021092315280295800_bib20","doi-asserted-by":"publisher","first-page":"2844","DOI":"10.18653\/v1\/D19-1284","article-title":"A discrete hard EM approach for weakly supervised question answering","volume-title":"Proceedings of EMNLP-IJCNLP 2019","author":"Min","year":"2019"},{"key":"2021092315280295800_bib21","article-title":"Complementary evidence identification in open-domain question answering","author":"Mou","year":"2021","journal-title":"arXiv preprint arXiv:2103 .11643"},{"key":"2021092315280295800_bib22","article-title":"Frustratingly hard evidence retrieval for qa over books","author":"Mou","year":"2020","journal-title":"ACL Nuse Workshop"},{"key":"2021092315280295800_bib23","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1220","article-title":"Multi-style generative reading comprehension","author":"Nishida","year":"2019","journal-title":"arXiv preprint arXiv:1901.02262"},{"key":"2021092315280295800_bib24","first-page":"311","article-title":"BLEU: A method for automatic evaluation of machine translation","volume-title":"Proceedings of ACL 2002","author":"Papineni","year":"2002"},{"key":"2021092315280295800_bib25","doi-asserted-by":"publisher","first-page":"2227","DOI":"10.18653\/v1\/N18-1202","article-title":"Deep contextualized word representations","volume-title":"Proceedings of NAACL 2018","author":"Peters","year":"2018"},{"issue":"8","key":"2021092315280295800_bib26","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2021092315280295800_bib27","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","author":"Raffel","year":"2019","journal-title":"arXiv preprint arXiv:1910.10683"},{"key":"2021092315280295800_bib28","doi-asserted-by":"publisher","first-page":"784","DOI":"10.18653\/v1\/P18-2124","article-title":"Know what you don\u2019t know: Unanswerable questions for squad","volume-title":"Proceedings of ACL 2018","author":"Rajpurkar","year":"2018"},{"key":"2021092315280295800_bib29","first-page":"109","article-title":"Okapi at trec-3","volume":"109","author":"Robertson","year":"1995","journal-title":"Nist Special Publication Sp"},{"key":"2021092315280295800_bib30","article-title":"Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation","author":"Sharma","year":"2017","journal-title":"arXiv preprint arXiv:1706.09799"},{"key":"2021092315280295800_bib31","doi-asserted-by":"publisher","first-page":"2633","DOI":"10.18653\/v1\/N19-1270","article-title":"Improving machine reading comprehension with general reading strategies","volume-title":"Proceedings of NAACL 2019","author":"Sun","year":"2019"},{"key":"2021092315280295800_bib32","doi-asserted-by":"publisher","first-page":"4922","DOI":"10.18653\/v1\/P19-1486","article-title":"Simple and effective curriculum pointer- generator networks for reading comprehension over long narratives","volume-title":"Proceedings of ACL 2019","author":"Yi","year":"2019"},{"key":"2021092315280295800_bib33","first-page":"45","article-title":"ACE 2005 multilingual training corpus","volume":"57","author":"Walker","year":"2006","journal-title":"Linguistic Data Consortium, Philadelphia"},{"key":"2021092315280295800_bib34","article-title":"R3: Reinforced ranker-reader for open-domain question answering","volume-title":"AAAI 2018","author":"Wang","year":"2018"},{"key":"2021092315280295800_bib35","article-title":"Evidence aggregation for answer re-ranking in open-domain question answering","volume-title":"ICLR 2018","author":"Wang","year":"2018"},{"key":"2021092315280295800_bib36","article-title":"Cluster-former: Clustering-based sparse transformer for long- range dependency encoding","author":"Wang","year":"2020","journal-title":"arXiv preprint arXiv:2009.06097"},{"key":"2021092315280295800_bib37","volume-title":"An Ideal Husband","author":"Wilde","year":"1916"},{"key":"2021092315280295800_bib38","article-title":"Pretrained encyclopedia: Weakly supervised knowledge- pretrained language model","volume-title":"International Conference on Learning Representations","author":"Xiong","year":"2019"},{"key":"2021092315280295800_bib39","doi-asserted-by":"publisher","first-page":"5020","DOI":"10.18653\/v1\/P19-1496","article-title":"TweetQA: A social media focused question answering dataset","volume-title":"Proceedings of ACL 2019","author":"Xiong","year":"2019"},{"key":"2021092315280295800_bib40","doi-asserted-by":"publisher","first-page":"2369","DOI":"10.18653\/v1\/D18-1259","article-title":"HotpotQA: A dataset for diverse, explainable multi-hop question answering","volume-title":"Proceedings of EMNLP 2018","author":"Yang","year":"2018"},{"key":"2021092315280295800_bib41","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.408","article-title":"Readtwice: Reading very large documents with memories","author":"Zemlyanskiy","year":"2021","journal-title":"arXiv preprint arXiv: 2105.04241"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00411\/1963997\/tacl_a_00411.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00411\/1963997\/tacl_a_00411.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,24]],"date-time":"2021-09-24T03:51:45Z","timestamp":1632455505000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00411\/107386\/Narrative-Question-Answering-with-Cutting-Edge"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":41,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00411","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}