{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,22]],"date-time":"2026-07-22T04:47:04Z","timestamp":1784695624746,"version":"3.55.0"},"reference-count":130,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2022,3,4]],"date-time":"2022-03-04T00:00:00Z","timestamp":1646352000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF","award":["1815287"],"award-info":[{"award-number":["1815287"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2022,4,30]]},"abstract":"<jats:p>\n            A great part of software development involves conceptualizing or communicating the underlying procedures and logic that needs to be expressed in programs. One major difficulty of programming is turning\n            <jats:italic>concept<\/jats:italic>\n            into\n            <jats:italic>code<\/jats:italic>\n            , especially when dealing with the APIs of unfamiliar libraries. Recently, there has been a proliferation of machine learning methods for code generation and retrieval from\n            <jats:italic>natural language queries<\/jats:italic>\n            , but these have primarily been evaluated purely based on retrieval accuracy or overlap of generated code with developer-written code, and the actual effect of these methods on the developer workflow is surprisingly unattested. In this article, we perform the first comprehensive investigation of the promise and challenges of using such technology inside the PyCharm IDE, asking, \u201cAt the current state of technology does it improve developer productivity or accuracy, how does it affect the developer experience, and what are the remaining gaps and challenges?\u201d To facilitate the study, we first develop a plugin for the PyCharm IDE that implements a hybrid of code generation and code retrieval functionality, and we orchestrate virtual environments to enable collection of many user events (e.g., web browsing, keystrokes, fine-grained code edits). We ask developers with various backgrounds to complete 7 varieties of 14 Python programming tasks ranging from basic file manipulation to machine learning or data visualization, with or without the help of the plugin. While qualitative surveys of developer experience are largely positive, quantitative results with regards to increased productivity, code quality, or program correctness are inconclusive. Further analysis identifies several pain points that could improve the effectiveness of future machine learning-based code generation\/retrieval developer assistants and demonstrates when developers prefer code generation over code retrieval and vice versa. We release all data and software to pave the road for future empirical studies on this topic, as well as development of better code generation models.\n          <\/jats:p>","DOI":"10.1145\/3487569","type":"journal-article","created":{"date-parts":[[2022,3,4]],"date-time":"2022-03-04T09:54:21Z","timestamp":1646387661000},"page":"1-47","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":121,"title":["In-IDE Code Generation from Natural Language: Promise and Challenges"],"prefix":"10.1145","volume":"31","author":[{"given":"Frank F.","family":"Xu","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bogdan","family":"Vasilescu","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Graham","family":"Neubig","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,3,4]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1546"},{"key":"e_1_3_3_3_2","first-page":"281","volume-title":"International Symposium on Foundations of Software Engineering (ESEC\/FSE)","author":"Allamanis Miltiadis","year":"2014","unstructured":"Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton. 2014. Learning natural coding conventions. In International Symposium on Foundations of Software Engineering (ESEC\/FSE). 281\u2013293."},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3212695"},{"key":"e_1_3_3_5_2","volume-title":"32nd International Conference on Machine Learning (ICML)","author":"Allamanis Miltiadis","year":"2015","unstructured":"Miltiadis Allamanis, Daniel Tarlow, A. Gordon, and Y. Wei. 2015. Bimodal modelling of source code and natural language. In 32nd International Conference on Machine Learning (ICML)."},{"key":"e_1_3_3_6_2","first-page":"1","article-title":"FeedBaG: An interaction tracker for Visual Studio","author":"Amann S.","year":"2016","unstructured":"S. Amann, Sebastian Proksch, and S. Nadi. 2016. FeedBaG: An interaction tracker for Visual Studio. In International Conference on Program Comprehension (ICPC). 1\u20133.","journal-title":"I"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2016.39"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00122124"},{"key":"e_1_3_3_9_2","first-page":"571","article-title":"Semantic parsing of ambiguous input through paraphrasing and verification","volume":"3","author":"Arthur Philip","year":"2015","unstructured":"Philip Arthur, Graham Neubig, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Semantic parsing of ambiguous input through paraphrasing and verification. Trans. Assoc. Comput. Ling. 3 (2015), 571\u2013584.","journal-title":"Trans. Assoc. Comput. Ling."},{"key":"e_1_3_3_10_2","first-page":"26","volume-title":"International Workshop on Recommendation Systems for Software Engineering (RSSE)","author":"Bacchelli Alberto","year":"2012","unstructured":"Alberto Bacchelli, Luca Ponzanelli, and Michele Lanza. 2012. Harnessing stack overflow for the IDE. In International Workshop on Recommendation Systems for Software Engineering (RSSE). IEEE, 26\u201330."},{"key":"e_1_3_3_11_2","article-title":"DeepCoder: Learning to write programs","author":"Balog Matej","year":"2017","unstructured":"Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to write programs. In 5th International Conference on Learning Representations (ICLR).","journal-title":"I"},{"key":"e_1_3_3_12_2","article-title":"Ringer: Web automation by demonstration","author":"Barman S.","year":"2016","unstructured":"S. Barman, Sarah E. Chasins, Rastislav Bod\u00edk, and Sumit Gulwani. 2016. Ringer: Web automation by demonstration. In ACM SIGPLAN International Conference on Object-oriented Programming, Systems, Languages, and Applications.","journal-title":"ACM SIGPLAN International Conference on Object-oriented Programming, Systems, Languages, and Applications"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-25951-0_2"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11135-018-0802-x"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3220134.3220135"},{"key":"e_1_3_3_16_2","first-page":"1533","volume-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Berant Jonathan","year":"2013","unstructured":"Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Conference on Empirical Methods in Natural Language Processing (EMNLP). 1533\u20131544."},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518944"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2017.56"},{"key":"e_1_3_3_19_2","article-title":"Application of the delphi method in computer science principles rubric creation","author":"Catet\u00e9 Veronica","year":"2017","unstructured":"Veronica Catet\u00e9 and T. Barnes. 2017. Application of the delphi method in computer science principles rubric creation. InACM Conference on Innovation and Technology in Computer Science Education.","journal-title":"I"},{"key":"e_1_3_3_20_2","article-title":"Browser record and replay as a building block for end-user web automation tools","author":"Chasins Sarah E.","year":"2015","unstructured":"Sarah E. Chasins, S. Barman, Rastislav Bod\u00edk, and Sumit Gulwani. 2015. Browser record and replay as a building block for end-user web automation tools. In 24th International Conference on World Wide Web (WWW).","journal-title":"I"},{"key":"e_1_3_3_21_2","article-title":"Rousillon: Scraping distributed hierarchical web data","author":"Chasins Sarah E.","year":"2018","unstructured":"Sarah E. Chasins, Maria Mueller, and Rastislav Bod\u00edk. 2018. Rousillon: Scraping distributed hierarchical web data. In 31st Annual ACM Symposium on User Interface Software and Technology (UIST).","journal-title":"31st Annual ACM Symposium on User Interface Software and Technology (UIST)"},{"key":"e_1_3_3_22_2","volume-title":"7th International Conference on Learning Representations (ICLR)","author":"Chen X.","year":"2019","unstructured":"X. Chen, C. Liu, and D. Song. 2019. Execution-guided neural program synthesis. In 7th International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_3_23_2","volume-title":"Applied Multiple Regression\/Correlation Analysis for the Behavioral Sciences","author":"Cohen J.","year":"2003","unstructured":"J. Cohen. 2003. Applied Multiple Regression\/Correlation Analysis for the Behavioral Sciences. Lawrence Erlbaum."},{"key":"e_1_3_3_24_2","volume-title":"Mathematical Methods of Statistics","author":"Cram\u00e9r Harald","year":"1999","unstructured":"Harald Cram\u00e9r. 1999. Mathematical Methods of Statistics. Vol. 43. Princeton University Press."},{"key":"e_1_3_3_25_2","unstructured":"A. Cypher Daniel C. Halbert D. Kurlander H. Lieberman D. Maulsby B. Myers and Alan Turransky. 1993. Watch what I do: Programming by demonstration."},{"key":"e_1_3_3_26_2","first-page":"551","article-title":"Rubric based assessment plan implementation for computer science program: A practical approach","author":"Dawood M.","year":"2013","unstructured":"M. Dawood, Khalid A. Buragga, Abdul Raouf Khan, and Noor Zaman. 2013. Rubric based assessment plan implementation for computer science program: A practical approach. In IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE). 551\u2013555.","journal-title":"I"},{"key":"e_1_3_3_27_2","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1007\/BFb0014656","volume-title":"Program Construction","author":"Dijkstra Edsger W.","year":"1979","unstructured":"Edsger W. Dijkstra. 1979. On the foolishness of \u201cnatural language programming.\u201d In Program Construction. Springer, 51\u201353."},{"key":"e_1_3_3_28_2","volume-title":"33rd Conference on Neural Information Processing Systems (NeurIPS)","author":"Ellis K.","year":"2019","unstructured":"K. Ellis, Maxwell Nye, Y. Pu, Felix Sosa, J. Tenenbaum, and Armando Solar-Lezama. 2019. Write, execute, assess: Program synthesis with a REPL. In 33rd Conference on Neural Information Processing Systems (NeurIPS)."},{"key":"e_1_3_3_29_2","article-title":"Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity","author":"Fedus William","year":"2021","unstructured":"William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 (2021).","journal-title":"arXiv preprint arXiv:2101.03961"},{"key":"e_1_3_3_30_2","article-title":"Program synthesis using conflict-driven learning","author":"Feng Y.","year":"2018","unstructured":"Y. Feng, R. Martins, Osbert Bastani, and Isil Dillig. 2018. Program synthesis using conflict-driven learning. In 39th ACM SIGPLAN Conference on Programming Language Design and Implementation.","journal-title":"I"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/2737924.2737977"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.228"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/2699688"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511790942"},{"key":"e_1_3_3_36_2","unstructured":"J. Ginsparg. 1978. Natural language processing in an automatic programming domain."},{"key":"e_1_3_3_37_2","article-title":"What we can learn about student learning from open-ended programming projects in middle school computer science","author":"Grover Shuchi","year":"2018","unstructured":"Shuchi Grover, S. Basu, and Patricia K. Schank. 2018. What we can learn about student learning from open-ended programming projects in middle school computer science. In 49th ACM Technical Symposium on Computer Science Education.","journal-title":"I"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180167"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/1925844.1926423"},{"key":"e_1_3_3_40_2","first-page":"842","article-title":"Automatic query reformulations for text retrieval in software engineering","author":"Haiduc Sonia","year":"2013","unstructured":"Sonia Haiduc, G. Bavota, A. Marcus, R. Oliveto, A. Lucia, and T. Menzies. 2013. Automatic query reformulations for text retrieval in software engineering. In 35th International Conference on Software Engineering (ICSE). 842\u2013851.","journal-title":"I"},{"key":"e_1_3_3_41_2","first-page":"10052","article-title":"A retrieve-and-edit framework for predicting structured outputs","author":"Hashimoto Tatsunori B.","year":"2018","unstructured":"Tatsunori B. Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S. Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Conference on Advances in Neural Information Processing Systems (NeurIPS). 10052\u201310062.","journal-title":"I"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1111"},{"key":"e_1_3_3_43_2","article-title":"Interactive extraction of examples from existing code","author":"Head Andrew","year":"2018","unstructured":"Andrew Head, Elena Leah Glassman, B. Hartmann, and Marti A. Hearst. 2018. Interactive extraction of examples from existing code. In CHI Conference on Human Factors in Computing Systems.","journal-title":"CHI Conference on Human Factors in Computing Systems"},{"key":"e_1_3_3_44_2","article-title":"Writing reusable code feedback at scale with mixed-initiative program synthesis","author":"Head Andrew","year":"2017","unstructured":"Andrew Head, Elena Leah Glassman, Gustavo Soares, R. Suzuki, Lucas Figueredo, L. D\u2019Antoni, and B. Hartmann. 2017. Writing reusable code feedback at scale with mixed-initiative program synthesis. In 4th ACM Conference on Learning @ Scale.","journal-title":"4th ACM Conference on Learning @ Scale"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1147\/rd.204.0302"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSMR-WCRE.2014.6747190"},{"key":"e_1_3_3_47_2","doi-asserted-by":"crossref","first-page":"598","DOI":"10.1214\/aoms\/1177704172","article-title":"Estimates of location based on rank tests","author":"Jr. Joseph L. Hodges","year":"1963","unstructured":"Joseph L. Hodges Jr. and Erich L. Lehmann. 1963. Estimates of location based on rank tests. Ann. Math. Statist. (1963), 598\u2013611.","journal-title":"Ann. Math. Statist."},{"key":"e_1_3_3_48_2","article-title":"CodeSearchNet challenge: Evaluating the state of semantic code search","author":"Husain Hamel","year":"2019","unstructured":"Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 (2019).","journal-title":"arXiv preprint arXiv:1909.09436"},{"key":"e_1_3_3_49_2","volume-title":"54th Annual Meeting of the Association for Computational Linguistics (ACL)","author":"Iyer Srini","year":"2016","unstructured":"Srini Iyer, Ioannis Konstas, A. Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In 54th Annual Meeting of the Association for Computational Linguistics (ACL)."},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1192"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1111\/2041-210X.12225"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.intexsempar-1.4"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568292"},{"key":"e_1_3_3_54_2","article-title":"Variolite: Supporting exploratory programming by data scientists","author":"Kery Mary Beth","year":"2017","unstructured":"Mary Beth Kery, Amber Horvath, and B. Myers. 2017. Variolite: Supporting exploratory programming by data scientists. CHI Conference on Human Factors in Computing Systems (CHI).","journal-title":"CHI Conference on Human Factors in Computing Systems (CHI)"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2017.8103446"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985712"},{"key":"e_1_3_3_57_2","first-page":"301","article-title":"Debugging reinvented","author":"Ko A.","year":"2008","unstructured":"A. Ko and B. Myers. 2008. Debugging reinvented. In ACM\/IEEE 30th International Conference on Software Engineering (ICSE). 301\u2013310.","journal-title":"ACM\/IEEE 30th International Conference on Software Engineering (ICSE)"},{"key":"e_1_3_3_58_2","first-page":"199","volume-title":"IEEE Symposium on Visual Languages and Human-centric Computing (VL\/HCC)","author":"Ko Amy","year":"2004","unstructured":"Amy Ko, Brad A. Myers, and Htet Htet Aung. 2004. Six learning barriers in end-user programming systems. In IEEE Symposium on Visual Languages and Human-centric Computing (VL\/HCC). IEEE, 199\u2013206."},{"issue":"7","key":"e_1_3_3_59_2","article-title":"Lateral collinearity and misleading results in variance-based SEM: An illustration and recommendations","volume":"13","author":"Kock Ned","year":"2012","unstructured":"Ned Kock and Gary Lynn. 2012. Lateral collinearity and misleading results in variance-based SEM: An illustration and recommendations. J. Assoc. Inf. Syst. 13, 7 (2012).","journal-title":"J. Assoc. Inf. Syst."},{"key":"e_1_3_3_60_2","volume-title":"33rd Conference on Neural Information Processing Systems (NeurIPS)","author":"Kulal S.","year":"2019","unstructured":"S. Kulal, Panupong Pasupat, K. Chandra, Mina Lee, Oded Padon, A. Aiken, and Percy Liang. 2019. SPoC: Search-based pseudocode to code. In 33rd Conference on Neural Information Processing Systems (NeurIPS)."},{"key":"e_1_3_3_61_2","volume-title":"Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL)","author":"Kushman Nate","year":"2013","unstructured":"Nate Kushman and R. Barzilay. 2013. Using semantic unification to generate regular expressions from natural language. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (HLT-NAACL)."},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1002\/smr.1760"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/2666356.2594333"},{"key":"e_1_3_3_64_2","volume-title":"51st Annual Meeting of the Association for Computational Linguistics (ACL)","author":"Lei Tao","year":"2013","unstructured":"Tao Lei, F. Long, R. Barzilay, and M. Rinard. 2013. From natural language specifications to program input parsers. In 51st Annual Meeting of the Association for Computational Linguistics (ACL)."},{"key":"e_1_3_3_65_2","article-title":"SUGILITE: Creating multimodal smartphone automation by demonstration","author":"Li Toby Jia-Jun","year":"2017","unstructured":"Toby Jia-Jun Li, Amos Azaria, and B. Myers. 2017. SUGILITE: Creating multimodal smartphone automation by demonstration. In CHI Conference on Human Factors in Computing Systems (CHI).","journal-title":"CHI Conference on Human Factors in Computing Systems (CHI)"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2018.8506506"},{"key":"e_1_3_3_67_2","article-title":"PUMICE: A multi-modal agent that learns concepts and conditionals from natural language and demonstrations","author":"Li Toby Jia-Jun","year":"2019","unstructured":"Toby Jia-Jun Li, Marissa Radensky, J. Jia, Kirielle Singarajah, Tom Michael Mitchell, and B. Myers. 2019. PUMICE: A multi-modal agent that learns concepts and conditionals from natural language and demonstrations. In 32nd Annual ACM Symposium on User Interface Software and Technology (UIST).","journal-title":"32nd Annual ACM Symposium on User Interface Software and Technology (UIST)"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1007\/1-4020-5386-X"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/p16-1057"},{"key":"e_1_3_3_70_2","article-title":"Opportunities and challenges in code search tools","volume":"2011","author":"Liu C.","year":"2020","unstructured":"C. Liu, Xin Xia, David Lo, Cuiyun Gao, Xiaohu Yang, and J. Grundy. 2020. Opportunities and challenges in code search tools. ArXiv abs\/2011.02297 (2020).","journal-title":"ArXiv"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC.2016.015"},{"key":"e_1_3_3_72_2","first-page":"545","volume-title":"International Conference on Software Analysis, Evolution, and Reengineering (SANER)","author":"Lu Meili","year":"2015","unstructured":"Meili Lu, Xiaobing Sun, S. Wang, D. Lo, and Yucong Duan. 2015. Query expansion via WordNet for effective code search. In International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 545\u2013549."},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/1868358.1868363"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.5555\/1394399"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v27i1.8695"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1976.233837"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1007\/11671299_34"},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-007-9040-x"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.98"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.2307\/1913646"},{"key":"e_1_3_3_81_2","first-page":"249","volume-title":"IEEE Symposium on Visual Languages and Human-centric Computing (VL\/HCC)","author":"Murphy Lauren","year":"2018","unstructured":"Lauren Murphy, Mary Beth Kery, Oluwatosin Alliyu, Andrew Macvean, and Brad A. Myers. 2018. API designers in the field: Design practices and challenges for creating usable APIs. In IEEE Symposium on Visual Languages and Human-centric Computing (VL\/HCC). IEEE, 249\u2013258."},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/1015864.1015888"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/2896587"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2016.200"},{"key":"e_1_3_3_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/2896587"},{"key":"e_1_3_3_86_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.2041-210x.2012.00261.x"},{"key":"e_1_3_3_87_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00063"},{"key":"e_1_3_3_88_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2015.32"},{"key":"e_1_3_3_89_2","doi-asserted-by":"publisher","DOI":"10.1177\/1609406917733847"},{"key":"e_1_3_3_90_2","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_3_3_91_2","article-title":"Neuro-symbolic program synthesis","author":"Parisotto Emilio","year":"2017","unstructured":"Emilio Parisotto, Abdel Rahman Mohamed, R. Singh, L. Li, Dengyong Zhou, and Pushmeet Kohli. 2017. Neuro-symbolic program synthesis. In 5th International Conference on Learning Representations (ICLR).","journal-title":"5th International Conference on Learning Representations (ICLR)"},{"key":"e_1_3_3_92_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2013.6606701"},{"key":"e_1_3_3_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/2597073.2597077"},{"key":"e_1_3_3_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/325737.325845"},{"key":"e_1_3_3_95_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196400"},{"key":"e_1_3_3_96_2","doi-asserted-by":"crossref","first-page":"34","DOI":"10.18653\/v1\/2020.intexsempar-1.5","volume-title":"1st Workshop on Interactive and Executable Semantic Parsing","author":"Radhakrishnan Karthik","year":"2020","unstructured":"Karthik Radhakrishnan, Arvind Srikantan, and Xi Victoria Lin. 2020. ColloQL: Robust Text-to-SQL over search queries. In 1st Workshop on Interactive and Executable Semantic Parsing. 34\u201345."},{"key":"e_1_3_3_97_2","first-page":"357","article-title":"SWIM: Synthesizing what I mean\u2014Code search and idiomatic snippet synthesis","author":"Raghothaman Mukund","year":"2016","unstructured":"Mukund Raghothaman, Y. Wei, and Y. Hamadi. 2016. SWIM: Synthesizing what I mean\u2014Code search and idiomatic snippet synthesis. In IEEE\/ACM 38th International Conference on Software Engineering (ICSE). 357\u2013367.","journal-title":"IEEE\/ACM 38th International Conference on Software Engineering (ICSE)"},{"key":"e_1_3_3_98_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2014.109"},{"key":"e_1_3_3_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSMR-WCRE.2014.6747170"},{"key":"e_1_3_3_100_2","doi-asserted-by":"publisher","DOI":"10.1145\/2594291.2594321"},{"key":"e_1_3_3_101_2","doi-asserted-by":"publisher","DOI":"10.5555\/2832249.2832359"},{"key":"e_1_3_3_102_2","doi-asserted-by":"publisher","DOI":"10.1090\/S0002-9947-1953-0053041-6"},{"key":"e_1_3_3_103_2","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242643"},{"key":"e_1_3_3_104_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416622"},{"key":"e_1_3_3_105_2","first-page":"191","volume-title":"10th Joint Meeting on Foundations of Software Engineering (ESEC\/FSE)","author":"Sadowski Caitlin","year":"2015","unstructured":"Caitlin Sadowski, Kathryn T. Stolee, and Sebastian Elbaum. 2015. How developers search for code: A case study. In 10th Joint Meeting on Foundations of Software Engineering (ESEC\/FSE). 191\u2013201."},{"key":"e_1_3_3_106_2","first-page":"171","article-title":"Supporting the understanding and comparison of low-code development platforms","author":"Sahay Apurvanand","year":"2020","unstructured":"Apurvanand Sahay, Arsene Indamutsa, D. D. Ruscio, and A. Pierantonio. 2020. Supporting the understanding and comparison of low-code development platforms. In 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). 171\u2013178.","journal-title":"I"},{"key":"e_1_3_3_107_2","article-title":"Program synthesis and semantic parsing with learned code idioms","author":"Shin Richard","year":"2019","unstructured":"Richard Shin, Miltiadis Allamanis, Marc Brockschmidt, and Oleksandr Polozov. 2019. Program synthesis and semantic parsing with learned code idioms. In 33rd Conference on Neural Information Processing Systems (NeurIPS).","journal-title":"I"},{"key":"e_1_3_3_108_2","doi-asserted-by":"publisher","DOI":"10.5555\/1324786"},{"key":"e_1_3_3_109_2","unstructured":"Armando Solar-Lezama. 2008. Program synthesis by sketching."},{"key":"e_1_3_3_110_2","article-title":"Live API documentation","author":"Subramanian Siddharth","year":"2014","unstructured":"Siddharth Subramanian, Laura Inozemtseva, and Reid Holmes. 2014. Live API documentation. In International Conference on Software Engineering (ICSE).","journal-title":"International Conference on Software Engineering (ICSE)"},{"key":"e_1_3_3_111_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2019.00034"},{"key":"e_1_3_3_112_2","first-page":"269","volume-title":"International Symposium on Foundations of Software Engineering (ESEC\/FSE)","author":"Tu Zhaopeng","year":"2014","unstructured":"Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. 2014. On the localness of software. In International Symposium on Foundations of Software Engineering (ESEC\/FSE). ACM, 269\u2013280."},{"key":"e_1_3_3_113_2","first-page":"191","volume-title":"Australasian Language Technology Workshop","author":"Vadas David","year":"2005","unstructured":"David Vadas and James R. Curran. 2005. Programming with unrestricted natural language. In Australasian Language Technology Workshop. 191\u2013199."},{"key":"e_1_3_3_114_2","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018691"},{"key":"e_1_3_3_115_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-1129"},{"key":"e_1_3_3_116_2","volume-title":"Building Bing Developer Assistant","author":"Wei Yi","year":"2015","unstructured":"Yi Wei, Nirupama Chandrasekaran, Sumit Gulwani, and Youssef Hamadi. 2015. Building Bing Developer Assistant. Technical Report. MSR-TR-2015-36, Microsoft Research."},{"key":"e_1_3_3_117_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2"},{"key":"e_1_3_3_118_2","first-page":"6045","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL)","author":"Xu Frank F.","year":"2020","unstructured":"Frank F. Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, and Graham Neubig. 2020. Incorporating external knowledge through pre-training for natural language to code generation. In Annual Meeting of the Association for Computational Linguistics (ACL). Association for Computational Linguistics, 6045\u20136052."},{"key":"e_1_3_3_119_2","first-page":"956","volume-title":"52nd Annual Meeting of the Association for Computational Linguistics (ACL)","author":"Yao Xuchen","year":"2014","unstructured":"Xuchen Yao and Benjamin Van Durme. 2014. Information extraction over structured data: Question answering with freebase. In 52nd Annual Meeting of the Association for Computational Linguistics (ACL). 956\u2013966."},{"key":"e_1_3_3_120_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33012547"},{"key":"e_1_3_3_121_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313632"},{"key":"e_1_3_3_122_2","article-title":"StaQC: A systematically mined question-code dataset from stack overflow","author":"Yao Ziyu","year":"2018","unstructured":"Ziyu Yao, Daniel S. Weld, W. Chen, and Huan Sun. 2018. StaQC: A systematically mined question-code dataset from stack overflow. In World Wide Web Conference (WWW).","journal-title":"World Wide Web Conference (WWW)"},{"key":"e_1_3_3_123_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196408"},{"key":"e_1_3_3_124_2","article-title":"A syntactic neural model for general-purpose code generation","author":"Yin Pengcheng","year":"2017","unstructured":"Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Annual Meeting of the Association for Computational Linguistics (ACL).","journal-title":"Annual Meeting of the Association for Computational Linguistics (ACL)"},{"key":"e_1_3_3_125_2","article-title":"TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation","author":"Yin Pengcheng","year":"2018","unstructured":"Pengcheng Yin and Graham Neubig. 2018. TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Demo Track.","journal-title":"Conference on Empirical Methods in Natural Language Processing (EMNLP), Demo Track"},{"key":"e_1_3_3_126_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1447"},{"key":"e_1_3_3_127_2","article-title":"NAPS: Natural program synthesis dataset","author":"Zavershynskyi Maksym","year":"2018","unstructured":"Maksym Zavershynskyi, Alex Skidanov, and Illia Polosukhin. 2018. NAPS: Natural program synthesis dataset. In 2nd Workshop on Neural Abstract Machines & Program Induction (NAMPI).","journal-title":"2nd Workshop on Neural Abstract Machines & Program Induction (NAMPI)"},{"key":"e_1_3_3_128_2","first-page":"1050","volume-title":"National Conference on Artificial Intelligence","author":"Zelle John M.","year":"1996","unstructured":"John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In National Conference on Artificial Intelligence. 1050\u20131055."},{"key":"e_1_3_3_129_2","first-page":"678","volume-title":"Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)","author":"Zettlemoyer Luke","year":"2007","unstructured":"Luke Zettlemoyer and Michael Collins. 2007. Online learning of relaxed CCG grammars for parsing to logical form. In Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). 678\u2013687."},{"key":"e_1_3_3_130_2","volume-title":"Meeting of the Association for Computational Linguistics","author":"Zhong Ruiqi","year":"2020","unstructured":"Ruiqi Zhong, Mitchell Stern, and D. Klein. 2020. Semantic scaffolds for pseudocode-to-code generation. In Meeting of the Association for Computational Linguistics."},{"key":"e_1_3_3_131_2","article-title":"Seq2SQL: Generating structured queries from natural language using reinforcement learning","author":"Zhong Victor","year":"2017","unstructured":"Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103 (2017).","journal-title":"arXiv preprint arXiv:1709.00103"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3487569","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3487569","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3487569","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:31:20Z","timestamp":1750188680000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3487569"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,4]]},"references-count":130,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,4,30]]}},"alternative-id":["10.1145\/3487569"],"URL":"https:\/\/doi.org\/10.1145\/3487569","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,4]]},"assertion":[{"value":"2021-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-03-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}