{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T16:27:29Z","timestamp":1776184049301,"version":"3.50.1"},"reference-count":42,"publisher":"MIT Press","issue":"4","license":[{"start":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T00:00:00Z","timestamp":1687219200000},"content-version":"vor","delay-in-days":170,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Large neural models have brought a new challenge to natural language generation (NLG): It has become imperative to ensure the safety and reliability of the output of models that generate freely. To this end, we present an evaluation framework, Attributable to Identified Sources (AIS), stipulating that NLG output pertaining to the external world is to be verified against an independent, provided source. We define AIS and a two-stage annotation pipeline for allowing annotators to evaluate model output according to annotation guidelines. We successfully validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset). We provide full annotation guidelines in the appendices and publicly release the annotated data at https:\/\/github.com\/google-research-datasets\/AIS.<\/jats:p>","DOI":"10.1162\/coli_a_00486","type":"journal-article","created":{"date-parts":[[2023,6,20]],"date-time":"2023-06-20T20:19:46Z","timestamp":1687292386000},"page":"777-840","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":36,"title":["Measuring Attribution in Natural Language Generation Models"],"prefix":"10.1162","volume":"49","author":[{"given":"Hannah","family":"Rashkin","sequence":"first","affiliation":[{"name":"Google DeepMind. hrashkin@google.com"}]},{"given":"Vitaly","family":"Nikolaev","sequence":"additional","affiliation":[{"name":"Google DeepMind. vitalyn@google.com"}]},{"given":"Matthew","family":"Lamm","sequence":"additional","affiliation":[{"name":"Google DeepMind. mrlamm@google.com"}]},{"given":"Lora","family":"Aroyo","sequence":"additional","affiliation":[{"name":"Google Research. loraa@google.com"}]},{"given":"Michael","family":"Collins","sequence":"additional","affiliation":[{"name":"Google DeepMind. mjcollins@google.com"}]},{"given":"Dipanjan","family":"Das","sequence":"additional","affiliation":[{"name":"Google DeepMind. dipanjand@google.com"}]},{"given":"Slav","family":"Petrov","sequence":"additional","affiliation":[{"name":"Google DeepMind. slav@google.com"}]},{"given":"Gaurav Singh","family":"Tomar","sequence":"additional","affiliation":[{"name":"Google DeepMind. gtomar@google.com"}]},{"given":"Iulia","family":"Turc","sequence":"additional","affiliation":[{"name":"Storia AI. iulia@iuliaturc.com"}]},{"given":"David","family":"Reitter","sequence":"additional","affiliation":[{"name":"Google DeepMind. reitter@google.com"}]}],"member":"281","published-online":{"date-parts":[[2023,12,1]]},"reference":[{"key":"2024013020233352600_bib1","article-title":"Towards a human-like open-domain chatbot","author":"Adiwardana","year":"2020","journal-title":"arXiv preprint arXiv:2001.09977"},{"key":"2024013020233352600_bib2","doi-asserted-by":"publisher","first-page":"520","DOI":"10.18653\/v1\/2021.naacl-main.44","article-title":"Open-domain question answering goes conversational via question rewriting","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Anantha","year":"2021"},{"key":"2024013020233352600_bib3","doi-asserted-by":"crossref","first-page":"183","DOI":"10.18653\/v1\/2020.inlg-1.24","article-title":"Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing","volume-title":"Proceedings of the 13th International Conference on Natural Language Generation","author":"Belz","year":"2020"},{"key":"2024013020233352600_bib4","first-page":"155","article-title":"Implicature, explicature, and truth-theoretic semantics","volume-title":"Mental Representations: The Interface Between Language and Reality","author":"Carston","year":"1988"},{"key":"2024013020233352600_bib5","doi-asserted-by":"publisher","first-page":"2174","DOI":"10.18653\/v1\/D18-1241","article-title":"QuAC: Question answering in context","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Choi","year":"2018"},{"key":"2024013020233352600_bib6","doi-asserted-by":"publisher","first-page":"447","DOI":"10.1162\/tacl_a_00377","article-title":"Decontextualization: Making sentences stand-alone","volume":"9","author":"Choi","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024013020233352600_bib7","doi-asserted-by":"publisher","first-page":"1985","DOI":"10.1145\/3397271.3401206","article-title":"Cast-19: A dataset for conversational information seeking","author":"Dalton","year":"2020","journal-title":"Proceedings of SIGIR"},{"key":"2024013020233352600_bib8","article-title":"Wizard of Wikipedia: Knowledge-powered conversational agents","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR)","author":"Dinan","year":"2019"},{"key":"2024013020233352600_bib9","doi-asserted-by":"publisher","first-page":"5055","DOI":"10.18653\/v1\/2020.acl-main.454","article-title":"FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Durmus","year":"2020"},{"key":"2024013020233352600_bib10","doi-asserted-by":"publisher","first-page":"1066","DOI":"10.1162\/tacl_a_00506","article-title":"Evaluating attribution in dialogue systems: The BEGIN benchmark","volume":"10","author":"Dziri","year":"2022","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024013020233352600_bib11","doi-asserted-by":"publisher","first-page":"96","DOI":"10.18653\/v1\/2021.gem-1.10","article-title":"The GEM benchmark: Natural language generation, its evaluation and metrics","volume-title":"Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)","author":"Gehrmann","year":"2021"},{"issue":"1-2","key":"2024013020233352600_bib12","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1111\/j.1468-0017.1992.tb00202.x","article-title":"Why the child\u2019s theory of mind really is a theory","volume":"7","author":"Gopnik","year":"1992","journal-title":"Mind & Language"},{"key":"2024013020233352600_bib13","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1163\/9789004368811_003","article-title":"Logic and conversation","volume-title":"Speech Acts","author":"Grice","year":"1975"},{"key":"2024013020233352600_bib14","doi-asserted-by":"publisher","first-page":"3785","DOI":"10.18653\/v1\/2022.acl-long.263","article-title":"DialFact: A benchmark for fact-checking in dialogue","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Gupta","year":"2022"},{"key":"2024013020233352600_bib15","volume-title":"Harvey Friedman\u2019s Research on the Foundations of Mathematics","author":"Harrington","year":"1985"},{"key":"2024013020233352600_bib16","doi-asserted-by":"publisher","first-page":"7856","DOI":"10.18653\/v1\/2021.emnlp-main.619","article-title":"q2: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Honovich","year":"2021"},{"key":"2024013020233352600_bib17","doi-asserted-by":"crossref","first-page":"169","DOI":"10.18653\/v1\/2020.inlg-1.23","article-title":"Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions","volume-title":"Proceedings of the 13th International Conference on Natural Language Generation","author":"Howcroft","year":"2020"},{"key":"2024013020233352600_bib18","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1162\/tacl_a_00276","article-title":"Natural questions: A benchmark for question answering research","volume":"7","author":"Kwiatkowski","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2024013020233352600_bib19","doi-asserted-by":"publisher","first-page":"1410","DOI":"10.18653\/v1\/2022.acl-long.100","article-title":"Faithful or extractive? On mitigating the faithfulness-abstractiveness trade-off in abstractive summarization","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Ladhak","year":"2022"},{"key":"2024013020233352600_bib20","doi-asserted-by":"publisher","first-page":"1906","DOI":"10.18653\/v1\/2020.acl-main.173","article-title":"On faithfulness and factuality in abstractive summarization","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Maynez","year":"2020"},{"key":"2024013020233352600_bib21","doi-asserted-by":"publisher","first-page":"681","DOI":"10.18653\/v1\/2020.acl-main.64","article-title":"USR: An unsupervised and reference free evaluation metric for dialog generation","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Mehri","year":"2020"},{"key":"2024013020233352600_bib22","doi-asserted-by":"publisher","first-page":"280","DOI":"10.18653\/v1\/K16-1028","article-title":"Abstractive text summarization using sequence-to-sequence RNNs and beyond","volume-title":"Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning","author":"Nallapati","year":"2016"},{"key":"2024013020233352600_bib23","doi-asserted-by":"publisher","first-page":"6881","DOI":"10.18653\/v1\/2021.acl-long.536","article-title":"Improving factual consistency of abstractive summarization via question answering","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Nan","year":"2021"},{"issue":"01","key":"2024013020233352600_bib24","doi-asserted-by":"publisher","first-page":"6859","DOI":"10.1609\/aaai.v33i01.33016859","article-title":"Combining fact extraction and verification with neural semantic matching networks","volume":"33","author":"Nie","year":"2019","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2024013020233352600_bib25","doi-asserted-by":"publisher","first-page":"4812","DOI":"10.18653\/v1\/2021.naacl-main.383","article-title":"Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Pagnoni","year":"2021"},{"key":"2024013020233352600_bib26","doi-asserted-by":"publisher","first-page":"1173","DOI":"10.18653\/v1\/2020.emnlp-main.89","article-title":"ToTTo: A controlled table-to-text generation dataset","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Parikh","year":"2020"},{"key":"2024013020233352600_bib27","doi-asserted-by":"publisher","first-page":"677","DOI":"10.1162\/tacl_a_00293","article-title":"Inherent disagreements in human textual inferences","volume":"7","author":"Pavlick","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"issue":"1","key":"2024013020233352600_bib28","first-page":"5485","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2024013020233352600_bib29","doi-asserted-by":"publisher","first-page":"704","DOI":"10.18653\/v1\/2021.acl-long.58","article-title":"Increasing faithfulness in knowledge-grounded dialogue with controllable features","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Rashkin","year":"2021"},{"key":"2024013020233352600_bib30","article-title":"Model-agnostic interpretability of machine learning","author":"Ribeiro","year":"2016","journal-title":"arXiv preprint arXiv:1606.05386"},{"key":"2024013020233352600_bib31","article-title":"Rome was built in 1776: A case study on factual correctness in knowledge-grounded response generation","author":"Santhanam","year":"2021","journal-title":"arXiv preprint arXiv:2110.05456"},{"key":"2024013020233352600_bib32","doi-asserted-by":"publisher","first-page":"1073","DOI":"10.18653\/v1\/P17-1099","article-title":"Get to the point: Summarization with pointer-generator networks","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"See","year":"2017"},{"key":"2024013020233352600_bib33","doi-asserted-by":"publisher","first-page":"2453","DOI":"10.18653\/v1\/2020.acl-main.222","article-title":"The dialogue dodecathlon: Open-domain knowledge and image grounded conversational agents","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Shuster","year":"2020"},{"key":"2024013020233352600_bib34","article-title":"Evidence-based verification for real world information needs","author":"Thorne","year":"2021","journal-title":"arXiv preprint arXiv:2104.00640"},{"key":"2024013020233352600_bib35","first-page":"3346","article-title":"Automated fact checking: Task formulations, methods and future directions","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Thorne","year":"2018"},{"key":"2024013020233352600_bib36","doi-asserted-by":"publisher","first-page":"809","DOI":"10.18653\/v1\/N18-1074","article-title":"FEVER: A large-scale dataset for fact extraction and VERification","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Thorne","year":"2018"},{"key":"2024013020233352600_bib37","doi-asserted-by":"publisher","first-page":"5008","DOI":"10.18653\/v1\/2020.acl-main.450","article-title":"Asking and answering questions to evaluate the factual consistency of summaries","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Wang","year":"2020"},{"key":"2024013020233352600_bib38","doi-asserted-by":"publisher","first-page":"3731","DOI":"10.18653\/v1\/P19-1363","article-title":"Dialogue natural language inference","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Welleck","year":"2019"},{"key":"2024013020233352600_bib39","first-page":"607","article-title":"Relevance theory","volume-title":"The Handbook of Pragmatics","author":"Wilson","year":"2004"},{"key":"2024013020233352600_bib40","doi-asserted-by":"publisher","first-page":"2253","DOI":"10.18653\/v1\/D17-1239","article-title":"Challenges in data-to-document generation","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Wiseman","year":"2017"},{"key":"2024013020233352600_bib41","first-page":"17283","article-title":"Big Bird: Transformers for longer sequences","volume-title":"Advances in Neural Information Processing Systems","author":"Zaheer","year":"2020"},{"key":"2024013020233352600_bib42","doi-asserted-by":"publisher","first-page":"6197","DOI":"10.18653\/v1\/2020.acl-main.552","article-title":"Extractive summarization as text matching","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Zhong","year":"2020"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/49\/4\/777\/2269661\/coli_a_00486.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/coli\/article-pdf\/49\/4\/777\/2269661\/coli_a_00486.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,30]],"date-time":"2024-01-30T20:24:42Z","timestamp":1706646282000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/49\/4\/777\/116438\/Measuring-Attribution-in-Natural-Language"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":42,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,12,1]]},"published-print":{"date-parts":[[2023,12,1]]}},"URL":"https:\/\/doi.org\/10.1162\/coli_a_00486","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}