{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T06:55:37Z","timestamp":1771484137830,"version":"3.50.1"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA2","license":[{"start":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T00:00:00Z","timestamp":1667174400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CCF-1811865,CCF-1712067,CCF-1762299,CCF-1918889"],"award-info":[{"award-number":["CCF-1811865,CCF-1712067,CCF-1762299,CCF-1918889"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2022,10,31]]},"abstract":"<jats:p>We propose a new technique based on program synthesis for automatically generating visualizations from natural language queries. Our method parses the natural language query into a refinement type specification using the intents-and-slots paradigm and leverages type-directed synthesis to generate a set of visualization programs that are most likely to meet the user's intent. Our refinement type system captures useful hints present in the natural language query and allows the synthesis algorithm to reject visualizations that violate well-established design guidelines for the input data set. We have implemented our ideas in a tool called Graphy and evaluated it on NLVCorpus, which consists of 3 popular datasets and over 700 real-world natural language queries. Our experiments show that Graphy significantly outperforms state-of-the-art natural language based visualization tools, including transformer and rule-based ones.<\/jats:p>","DOI":"10.1145\/3563307","type":"journal-article","created":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T20:23:35Z","timestamp":1667247815000},"page":"532-559","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Type-directed synthesis of visualizations from natural language queries"],"prefix":"10.1145","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4680-5157","authenticated-orcid":false,"given":"Qiaochu","family":"Chen","sequence":"first","affiliation":[{"name":"University of Texas at Austin, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9253-9585","authenticated-orcid":false,"given":"Shankara","family":"Pailoor","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7688-6133","authenticated-orcid":false,"given":"Celeste","family":"Barnaby","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2596-1831","authenticated-orcid":false,"given":"Abby","family":"Criswell","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5933-6620","authenticated-orcid":false,"given":"Chenglong","family":"Wang","sequence":"additional","affiliation":[{"name":"Microsoft Research, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7061-7298","authenticated-orcid":false,"given":"Greg","family":"Durrett","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8006-1230","authenticated-orcid":false,"given":"I\u015fil","family":"Dillig","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, USA"}]}],"member":"320","published-online":{"date-parts":[[2022,10,31]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"International Conference on Learning Representations. arxiv:1409","author":"Bahdanau Dzmitry","year":"2014","unstructured":"Dzmitry Bahdanau , Kyunghyun Cho , and Yoshua Bengio . 2014 . Neural Machine Translation by Jointly Learning to Align and Translate . In International Conference on Learning Representations. arxiv:1409 .0473 Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. In International Conference on Learning Representations. arxiv:1409.0473"},{"key":"e_1_2_1_2_1","volume-title":"Advances in Neural Information Processing Systems","author":"Brown Tom","year":"1877","unstructured":"Tom Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared D Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Daniel Ziegler , Jeffrey Wu , Clemens Winter , Chris Hesse , Mark Chen , Eric Sigler , Mateusz Litwin , Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford , Ilya Sutskever , and Dario Amodei . 2020. Language Models are Few-Shot Learners . In Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). 33, Curran Associates, Inc. , 1877 \u20131901. https:\/\/proceedings.neurips.cc\/paper\/2020\/file\/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.). 33, Curran Associates, Inc., 1877\u20131901. https:\/\/proceedings.neurips.cc\/paper\/2020\/file\/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf"},{"key":"#cr-split#-e_1_2_1_3_1.1","unstructured":"Qiaochu Chen Shankara Pailoor Celeste Barnaby Abby Criswell Chenglong Wang Greg Durrett and Isil Dillig. 2022. Type-Directed Synthesis of Visualizations from Natural Language Queries. https:\/\/doi.org\/10.48550\/ARXIV.2209.01081 10.48550\/ARXIV.2209.01081"},{"key":"#cr-split#-e_1_2_1_3_1.2","doi-asserted-by":"crossref","unstructured":"Qiaochu Chen Shankara Pailoor Celeste Barnaby Abby Criswell Chenglong Wang Greg Durrett and Isil Dillig. 2022. Type-Directed Synthesis of Visualizations from Natural Language Queries. https:\/\/doi.org\/10.48550\/ARXIV.2209.01081","DOI":"10.1145\/3563307"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-53291-8_30"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.2307\/2963593"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.3115\/1075812.1075823"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3296979.3192382"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2813885.2737977"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2837614.2837629"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2807442.2807478"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2612177"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2814270.2814295"},{"key":"e_1_2_1_14_1","first-page":"90","volume-title":"The ATIS Spoken Language Systems Pilot Corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27","author":"Hemphill Charles T.","year":"1990","unstructured":"Charles T. Hemphill , John J. Godfrey , and George R. Doddington . 1990 . The ATIS Spoken Language Systems Pilot Corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27 , 1990 . https:\/\/aclanthology.org\/H 90 - 1021 Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS Spoken Language Systems Pilot Corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990. https:\/\/aclanthology.org\/H90-1021"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.3115\/1273073.1273127"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3314221.3314602"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1481848.1481853"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.438"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC","author":"Lin Xi Victoria","year":"2018","unstructured":"Xi Victoria Lin , Chenglong Wang , Luke Zettlemoyer , and Michael D. Ernst . 2018. NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System . In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018 ). European Language Resources Association (ELRA), Miyazaki, Japan. https:\/\/aclanthology.org\/L18-1491 Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. 2018. NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan. https:\/\/aclanthology.org\/L18-1491"},{"key":"e_1_2_1_21_1","volume-title":"Decoupled Weight Decay Regularization. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=Bkg6RiCqY7","author":"Loshchilov Ilya","year":"2019","unstructured":"Ilya Loshchilov and Frank Hutter . 2019 . Decoupled Weight Decay Regularization. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=Bkg6RiCqY7 Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=Bkg6RiCqY7"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457261"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2021.3114848"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2007.70594"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1098\/rsta.1984.0073"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2865240"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/tvcg.2020.3030378"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331554.3342608"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2737924.2738007"},{"key":"e_1_2_1_30_1","volume-title":"Types and Programming Languages","author":"Pierce Benjamin C.","year":"2091","unstructured":"Benjamin C. Pierce . 2002. Types and Programming Languages ( 1 st ed.). The MIT Press . isbn:026216 2091 Benjamin C. Pierce. 2002. Types and Programming Languages (1st ed.). The MIT Press. isbn:0262162091","edition":"1"},{"key":"e_1_2_1_31_1","volume-title":"International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=KmtVD97J43e","author":"Poesia Gabriel","year":"2022","unstructured":"Gabriel Poesia , Alex Polozov , Vu Le , Ashish Tiwari , Gustavo Soares , Christopher Meek , and Sumit Gulwani . 2022 . Synchromesh: Reliable Code Generation from Pre-trained Language Models . In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=KmtVD97J43e Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable Code Generation from Pre-trained Language Models. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=KmtVD97J43e"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2980983.2908093"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.26599\/BDMA.2018.9020007"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1379022.1375602"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1099"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445400"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-13544-6_18"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/SLT.2010.5700816"},{"key":"e_1_2_1_39_1","volume-title":"\u0141 ukasz Kaiser, and Illia Polosukhin","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N Gomez , \u0141 ukasz Kaiser, and Illia Polosukhin . 2017 . Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). 30, Curran Associates, Inc .. https:\/\/proceedings.neurips.cc\/paper\/2017\/file\/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141 ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). 30, Curran Associates, Inc.. https:\/\/proceedings.neurips.cc\/paper\/2017\/file\/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.677"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3371117"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"e_1_2_1_43_1","volume-title":"Voyager: Exploratory analysis via faceted browsing of visualization recommendations","author":"Wongsuphasawat Kanit","year":"2015","unstructured":"Kanit Wongsuphasawat , Dominik Moritz , Anushka Anand , Jock Mackinlay , Bill Howe , and Jeffrey Heer . 2015 . Voyager: Exploratory analysis via faceted browsing of visualization recommendations . IEEE transactions on visualization and computer graphics, 22, 1 (2015), 649\u2013658. Kanit Wongsuphasawat, Dominik Moritz, Anushka Anand, Jock Mackinlay, Bill Howe, and Jeffrey Heer. 2015. Voyager: Exploratory analysis via faceted browsing of visualization recommendations. IEEE transactions on visualization and computer graphics, 22, 1 (2015), 649\u2013658."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133887"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.146"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/tvcg.2019.2934668"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2019.2934668"},{"key":"e_1_2_1_48_1","volume-title":"Mooney","author":"Zelle John M.","year":"1996","unstructured":"John M. Zelle and Raymond J . Mooney . 1996 . Learning to Parse Database Queries using Inductive Logic Programming. In AAAI\/IAAI. AAAI Press\/MIT Press , Portland, OR. 1050\u20131055. http:\/\/www.cs.utexas.edu\/users\/ai-lab?zelle:aaai96 John M. Zelle and Raymond J. Mooney. 1996. Learning to Parse Database Queries using Inductive Logic Programming. In AAAI\/IAAI. AAAI Press\/MIT Press, Portland, OR. 1050\u20131055. http:\/\/www.cs.utexas.edu\/users\/ai-lab?zelle:aaai96"},{"key":"e_1_2_1_49_1","volume-title":"Zettlemoyer and Michael Collins","author":"Luke","year":"2005","unstructured":"Luke S. Zettlemoyer and Michael Collins . 2005 . Learning to Map Sentences to Logical Form : Structured Classification with Probabilistic Categorial Grammars. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence (UAI\u201905). AUAI Press , Arlington, Virginia, USA. 658\u2013666. isbn:0974903914 Luke S. Zettlemoyer and Michael Collins. 2005. Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence (UAI\u201905). AUAI Press, Arlington, Virginia, USA. 658\u2013666. isbn:0974903914"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3563307","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3563307","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3563307","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:38:10Z","timestamp":1750178290000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3563307"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,31]]},"references-count":50,"journal-issue":{"issue":"OOPSLA2","published-print":{"date-parts":[[2022,10,31]]}},"alternative-id":["10.1145\/3563307"],"URL":"https:\/\/doi.org\/10.1145\/3563307","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,31]]},"assertion":[{"value":"2022-10-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}