{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T16:25:40Z","timestamp":1791217540651,"version":"4.1.0"},"reference-count":50,"publisher":"MIT Press - Journals","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2014,12]]},"abstract":"<jats:p> Most previous work on trainable language generation has focused on two paradigms: (a) using a generation decisions of an existing generator. Both approaches rely on the existence of a handcrafted generation component, which is likely to limit their scalability to new domains. The first contribution of this article is to present Bagel, a fully data-driven generation method that treats the language generation task as a search for the most likely sequence of semantic concepts and realization phrases, according to Factored Language Models (FLMs). As domain utterances are not readily available for most natural language generation tasks, a large creative effort is required to produce the data necessary to represent human linguistic variation for nontrivial domains. This article is based on the assumption that learning to produce paraphrases can be facilitated by collecting data from a large sample of untrained annotators using crowdsourcing\u2014rather than a few domain experts\u2014by relying on a coarse meaning representation. A second contribution of this article is to use crowdsourced data to show how dialogue naturalness can be improved by learning to vary the output utterances generated for a given semantic input. Two data-driven methods for generating paraphrases in dialogue are presented: (a) by sampling from the n-best list of realizations produced by Bagel's FLM reranker; and (b) by learning a structured perceptron predicting whether candidate realizations are valid paraphrases. We train Bagel on a set of 1,956 utterances produced by 137 annotators, which covers 10 types of dialogue acts and 128 semantic concepts in a tourist information system for Cambridge. An automated evaluation shows that Bagel outperforms utterance class LM baselines on this domain. A human evaluation of 600 resynthesized dialogue extracts shows that Bagel's FLM output produces utterances comparable to a handcrafted baseline, whereas the perceptron classifier performs worse. Interestingly, human judges find the system sampling from the n-best list to be more natural than a system always returning the first-best utterance. The judges are also more willing to interact with the n-best system in the future. These results suggest that capturing the large variation found in human language using data-driven methods is beneficial for dialogue interaction. <\/jats:p>","DOI":"10.1162\/coli_a_00199","type":"journal-article","created":{"date-parts":[[2014,4,14]],"date-time":"2014-04-14T18:38:40Z","timestamp":1397500720000},"page":"763-799","source":"Crossref","is-referenced-by-count":33,"title":["Stochastic Language Generation in Dialogue using Factored Language Models"],"prefix":"10.1162","volume":"40","author":[{"given":"Fran\u00e7ois","family":"Mairesse","sequence":"first","affiliation":[{"name":"University of Cambridge"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steve","family":"Young","sequence":"additional","affiliation":[{"name":"University of Cambridge"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"R1","unstructured":"Angeli, Gabor, Percy Liang, and Dan Klein. 2010. A simple domain-independent probabilistic approach to generation. In Proceedings of EMNLP, pages 502\u2013512, Cambridge, MA."},{"key":"R3","doi-asserted-by":"crossref","unstructured":"Bannard, Colin and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 597\u2013604, Ann Arbor, MI.","DOI":"10.3115\/1219840.1219914"},{"key":"R4","doi-asserted-by":"crossref","unstructured":"Barzilay, Regina and Kathleen McKeown. 2001. Extracting paraphrases from a parallel corpus. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics (ACL), pages 50\u201357, Toulouse.","DOI":"10.3115\/1073012.1073020"},{"key":"R5","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324907004664"},{"key":"R6","doi-asserted-by":"crossref","unstructured":"Bilmes, Jeff and Katrin Kirchhoff. 2003. Factored language models and generalized parallel backoff. In Proceedings of HLT-NAACL, Short Papers, pages 4\u20136, Edmonton.","DOI":"10.3115\/1073483.1073485"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/S0885-2308(02)00023-2"},{"key":"R8","doi-asserted-by":"crossref","unstructured":"Cahill, Aoife and Josef van Genabith. 2006. Robust PCFG-based generation using automatically acquired LFG approximations. In Proceedings of the 44th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1,033\u20131,044, Sydney.","DOI":"10.3115\/1220175.1220305"},{"key":"R9","unstructured":"Chambers, Nathanael and James Allen. 2004. Stochastic language generation in a dialogue system: Toward a domain independent generator. In Proceedings 5th SIGdial Workshop on Discourse and Dialogue, pages 9\u201318, Cambridge, MA."},{"key":"R10","doi-asserted-by":"crossref","unstructured":"Collins, Michael. 2002a. Discriminative training methods for hidden Markov models: Theory and experiments with perceptron algorithm. In Proceedings of EMNLP, pages 1\u20138, Philadelphia, PA.","DOI":"10.3115\/1118693.1118694"},{"key":"R11","doi-asserted-by":"crossref","unstructured":"Collins, Michael. 2002b. Ranking algorithms for named-entity extraction: Boosting and the voted perceptron. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 489\u2013496, Philadelphia, PA.","DOI":"10.3115\/1073083.1073165"},{"key":"R12","doi-asserted-by":"crossref","unstructured":"Collins, Michael and Brian Roark. 2004. Incremental parsing with the perceptron algorithm. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 111\u2013118, Barcelona.","DOI":"10.3115\/1218955.1218970"},{"key":"R13","unstructured":"Espinosa, Dominic, Michael White, and Dennis Mehay. 2008. Hypertagging: Supertagging for surface realization with CCG. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL), pages 183\u2013191, Columbus, OH."},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Foster, Mary Ellen and Michael White. 2005. Assessing the impact of adaptive generation in the COMIC multimodal dialogue system. In Proceedings of the IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems, pages 24\u201331, Edinburgh.","DOI":"10.3115\/1225753.1225765"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2004.03.001"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Isard, Amy, Carsten Brockmann, and Jon Oberlander. 2006. Individuality and alignment in generated dialogues. In Proceedings of INLG, pages 22\u201329, Sydney.","DOI":"10.3115\/1706269.1706277"},{"key":"R17","unstructured":"Kondadadi, Ravi, Blake Howald, and Frank Schilder. 2013. A statistical NLG framework for aggregated planning and realization. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL), pages 1,406\u20131,415, Sofia."},{"key":"R18","unstructured":"Konstas, Ioannis and Mirella Lapata. 2012. Concept-to-text generation via discriminative reranking. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL), pages 369\u2013378, Jeju Island."},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.3115\/980845.980963"},{"key":"R20","unstructured":"Langkilde-Geary, Irene. 2002. An empirical verification of coverage and correctness for a general-purpose sentence generator. In Proceedings of the International Conference on Natural Language Generation, pages 17\u201324, Harriman, NY."},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.3115\/974557.974596"},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Lef\u00e8vre, Fabrice. 2006. A DBN-based multi-level stochastic spoken language understanding system. In Proceedings of SLT, pages 78\u201381, Palm Beach, Aruba.","DOI":"10.1109\/SLT.2006.326822"},{"key":"R23","doi-asserted-by":"crossref","unstructured":"Lin, Dekang and Patrick Pantel. 2001. DIRT\u2014discovery of inference rules from text. In Proceedings of ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 323\u2013328, San Francisco, CA.","DOI":"10.1145\/502512.502559"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.3115\/1699510.1699563"},{"key":"R25","unstructured":"Mairesse, Fran\u00e7ois, Milica Ga\u0161i\u0107, Filip Jur\u010d\u00ed\u010dek, Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. Phrase-based statistical language generation using graphical models and active learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1,552\u20131,561, Uppsala."},{"key":"R26","unstructured":"Mairesse, Fran\u00e7ois and Marilyn A. Walker. 2008. Trainable generation of Big-Five personality styles through data-driven parameter estimation. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL), pages 165\u2013173, Columbus, OH."},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00063"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.1515\/text.1.1988.8.3.243"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.3115\/1654494.1654504"},{"key":"R30","doi-asserted-by":"crossref","unstructured":"Nakatsu, Crystal and Michael White. 2006. Learning to say it well: Reranking realizations by predicted synthesis quality. In Proceedings of the 44th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1,113\u20131,120, Sydney.","DOI":"10.3115\/1220175.1220315"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.1162\/089120103321337421"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.1016\/S0885-2308(02)00012-8"},{"key":"R33","doi-asserted-by":"crossref","unstructured":"Paiva, Daniel S. and Roger Evans. 2005. Empirically-based control of natural language generation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 58\u201365, Ann Arbor, MI.","DOI":"10.3115\/1219840.1219848"},{"key":"R34","doi-asserted-by":"crossref","unstructured":"Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311\u2013318, Philadelphia, PA.","DOI":"10.3115\/1073083.1073135"},{"key":"R35","unstructured":"Pon-Barry, Heather, Karl Schultz, Elizabeth Owen Bratt, Brady Clark, and Stanley Peters. 2006. Responding to student uncertainty in spoken tutorial dialogue systems. International Journal of Artificial Intelligence in Education, 16:171\u2013194."},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.1109\/5.18626"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.1016\/S0885-2308(02)00025-6"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2009.35.4.35405"},{"key":"R40","doi-asserted-by":"crossref","unstructured":"Schatzmann, Jost, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a POMDP dialogue system. In Proceedings of HLT-NAACL, Short Papers, pages 149\u2013152, Rochester, NY.","DOI":"10.3115\/1614108.1614146"},{"key":"R41","doi-asserted-by":"crossref","unstructured":"Stent, Amanda and Martin Molina. 2009. Evaluating automatic extraction of rules for sentence plan construction. In Proceedings of the SIGdial Conference on Discourse and Dialogue, pages 290\u2013297, London.","DOI":"10.3115\/1708376.1708417"},{"key":"R42","doi-asserted-by":"crossref","unstructured":"Stent, Amanda, Rashmi Prasad, and Marilyn A. Walker. 2004. Trainable sentence planning for complex information presentation in spoken dialog systems. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 79\u201386, Barcelona.","DOI":"10.3115\/1218955.1218966"},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.1145\/1015706.1015753"},{"key":"R44","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2009.07.003"},{"key":"R45","doi-asserted-by":"crossref","unstructured":"Thomson, Blaise, Kai Yu, Simon Keizer, Milica Ga\u0161i\u0107, Filip Jur\u010d\u00ed\u010dek, Fran\u00e7ois Mairesse, and Steve Young. 2010. Bayesian dialogue system for the Let's Go spoken dialogue challenge. In Proceedings of SLT, Special Session: The Spoken Dialogue Challenge, pages 460\u2013465, Berkeley, CA.","DOI":"10.1109\/SLT.2010.5700896"},{"key":"R46","unstructured":"Tokuda, Keiichi, Takayoshi Yoshimura, Takashi Masuko, Takao Kobayashi, and Tadashi Kitamura. 2000. Speech parameter generation algorithms for HMM-based speech synthesis. In Proceedings of ICASSP, pages 1,315\u20131,318, Istanbul."},{"key":"R47","unstructured":"Torrance, Nancy, Elizabeth Lee, and David R. Olson. 1992. The development of the distinction between paraphrase and exact wording in the recognition of utterances. In Proceedings of the Annual Meeting of the American Educational Research Association, pages 1\u201311, San Francisco, CA."},{"key":"R49","doi-asserted-by":"crossref","unstructured":"Varges, Sebastian and Chris Mellish. 2001. Instance-based natural language generation. In Proceedings of the Annual Meeting of the North American Chapter of the ACL (NAACL), pages 1\u20138, Pittsburgh, PA.","DOI":"10.3115\/1073336.1073337"},{"key":"R50","doi-asserted-by":"publisher","DOI":"10.1016\/S0885-2308(02)00027-X"},{"key":"R51","unstructured":"White, Michael, Rajakrishnan Rajkumar, and Scott Martin. 2007. Towards broad coverage surface realization with CCG. In Proceedings of the Workshop on Using Corpora for NLG: Language Generation and Machine Translation, pages 22\u201330, Copenhagen."},{"key":"R52","unstructured":"Wong, Yuk Wah and Raymond J. Mooney. 2007. Generation by inverting a semantic parser that uses statistical machine translation. In Proceedings of HLT-NAACL, pages 172\u2013179, Rochester, NY."},{"key":"R53","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2009.04.001"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/COLI_a_00199","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:27:38Z","timestamp":1615584458000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/40\/4\/763-799\/1490"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2014,12]]},"references-count":50,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2014,12]]}},"alternative-id":["10.1162\/COLI_a_00199"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00199","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2014,12]]}}}