{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T17:16:21Z","timestamp":1783098981149,"version":"3.54.6"},"reference-count":86,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,19]]},"abstract":"<jats:p>Generative pre-trained models power intelligent software features used by millions of users controlled by developer-written natural language prompts. Despite the impact of prompt-powered software, little is known about its development process and its relationship to programming. In this work, we argue that some prompts are programs and that the development of prompts is a distinct phenomenon in programming known as \u201cprompt programming\u201d. We develop an understanding of prompt programming using Straussian grounded theory through interviews with 20 developers engaged in prompt development across a variety of contexts, models, domains, and prompt structures. We contribute 15 observations to form a preliminary understanding of current prompt programming practices. For example, rather than building mental models of code, prompt programmers develop mental models of the foundation model (FM)\u2019s behavior on the prompt by interacting with the FM. While prior research shows that experts have well-formed mental models, we find that prompt programmers who have developed dozens of prompts still struggle to develop reliable mental models. Our observations show that prompt programming differs from traditional software development, motivating the creation of prompt programming tools and providing implications for software engineering stakeholders.<\/jats:p>","DOI":"10.1145\/3729342","type":"journal-article","created":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T15:15:34Z","timestamp":1750346134000},"page":"1591-1614","source":"Crossref","is-referenced-by-count":17,"title":["Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6722-9959","authenticated-orcid":false,"given":"Jenny T.","family":"Liang","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-6795-8040","authenticated-orcid":false,"given":"Melissa","family":"Lin","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-5404-1534","authenticated-orcid":false,"given":"Nikitha","family":"Rao","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4769-0219","authenticated-orcid":false,"given":"Brad A.","family":"Myers","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,19]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Retrieved","year":"2024","unstructured":"2024. Bringing Microsoft Copilot to more customers worldwide. Retrieved August 29, 2024 from https:\/\/www.microsoft.com\/en-us\/microsoft-365\/blog\/2024\/03\/14\/bringing-copilot-to-more-customers-worldwide-across-life-and-work\/"},{"key":"e_1_2_1_2_1","unstructured":"2024. ChatGPT. Retrieved August 29 2024 from https:\/\/chatgpt.com\/"},{"key":"e_1_2_1_3_1","volume-title":"Retrieved","year":"2024","unstructured":"2024. Copilot for Microsoft 365 grounded in your data. Retrieved August 29, 2024 from https:\/\/copilot.cloud.microsoft\/en-US\/copilot-microsoft365-chat\/"},{"key":"e_1_2_1_4_1","volume-title":"Retrieved","author":"Google","year":"2024","unstructured":"2024. Google I\/O 2024: New generative AI experiences in search. Retrieved August 29, 2024 from https:\/\/blog.google\/products\/search\/generative-ai-google-search-may-2024\/"},{"key":"e_1_2_1_5_1","volume-title":"Retrieved","year":"2024","unstructured":"2024. Introducing the GPT Store | OpenAI. Retrieved August 29, 2024 from https:\/\/openai.com\/index\/introducing-the-gpt-store\/"},{"key":"e_1_2_1_6_1","unstructured":"2024. LangSmith Hub. Retrieved August 29 2024 from https:\/\/smith.langchain.com\/hub"},{"key":"e_1_2_1_7_1","volume-title":"Prompt engineering - OpenAI. Retrireved","year":"2024","unstructured":"2024. Prompt engineering - OpenAI. Retrireved August 29, 2024 from https:\/\/platform.openai.com\/docs\/guides\/prompt-engineering\/strategy-write-clear-instructions"},{"key":"e_1_2_1_8_1","volume-title":"Retrieved","year":"2024","unstructured":"2024. ShareGPT: Share your wildest ChatGPT conversations with one click.. Retrieved August 29, 2024 from https:\/\/sharegpt.com\/"},{"key":"e_1_2_1_9_1","volume-title":"Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat.","author":"Achiam Josh","year":"2023","unstructured":"Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774."},{"key":"e_1_2_1_10_1","volume-title":"ACM Symposium on User Interface Software and Technology (UIST). 1\u20133. https:\/\/doi.org\/10","author":"Arawjo Ian","year":"2023","unstructured":"Ian Arawjo, Priyan Vaithilingam, Martin Wattenberg, and Elena Glassman. 2023. ChainForge: An open-source visual programming environment for prompt engineering. In ACM Symposium on User Interface Software and Technology (UIST). 1\u20133. https:\/\/doi.org\/10.1145\/3586182.3616660 10.1145\/3586182.3616660"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i1.2564"},{"key":"e_1_2_1_12_1","volume-title":"Euromicro Conference on Software Engineering and Advanced Applications (SEAA). 50\u201359","author":"Arpteg Anders","year":"2018","unstructured":"Anders Arpteg, Bj\u00f6rn Brinne, Luka Crnkovic-Friis, and Jan Bosch. 2018. Software engineering challenges of deep learning. In Euromicro Conference on Software Engineering and Advanced Applications (SEAA). 50\u201359. https:\/\/doi.org\/10.1109\/SEAA.2018.00018 10.1109\/SEAA.2018.00018"},{"key":"e_1_2_1_13_1","volume-title":"ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE). 187\u2013200","author":"Baltes Sebastian","year":"2018","unstructured":"Sebastian Baltes and Stephan Diehl. 2018. Towards a theory of software development expertise. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE). 187\u2013200. https:\/\/doi.org\/10.1145\/3236024.3236061 10.1145\/3236024.3236061"},{"key":"e_1_2_1_14_1","volume-title":"ACM conference on Fairness, Accountability, and Transparency (FAccT). 610\u2013623","author":"Bender Emily M","year":"2021","unstructured":"Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In ACM conference on Fairness, Accountability, and Transparency (FAccT). 610\u2013623. https:\/\/doi.org\/10.1145\/3442188.3445922 10.1145\/3442188.3445922"},{"key":"e_1_2_1_15_1","volume-title":"Improving image generation with better captions. 2, 3","author":"Betker James","year":"2023","unstructured":"James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, and Yufei Guo. 2023. Improving image generation with better captions. 2, 3 (2023), 8. https:\/\/doi.org\/papers\/dall-e-3. pdf"},{"key":"e_1_2_1_16_1","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33 (2020), 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"e_1_2_1_17_1","volume-title":"Constructing grounded theory: A practical guide through qualitative analysis","author":"Charmaz Kathy","unstructured":"Kathy Charmaz. 2006. Constructing grounded theory: A practical guide through qualitative analysis. Sage."},{"key":"e_1_2_1_18_1","volume-title":"International Conference on Mining Software Repositories (MSR). 207\u2013211","author":"Chavan Omkar Sandip","year":"2024","unstructured":"Omkar Sandip Chavan, Divya Dilip Hinge, Soham Sanjay Deo, Yaxuan Wang, and Mohamed Wiem Mkaouer. 2024. Analyzing developer-ChatGPT conversations for software refactoring: An exploratory study. In International Conference on Mining Software Repositories (MSR). 207\u2013211. https:\/\/doi.org\/10.1145\/3643991.3645082 10.1145\/3643991.3645082"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1162\/99608f92.5317da47"},{"key":"e_1_2_1_20_1","volume-title":"Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman.","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374."},{"key":"e_1_2_1_21_1","volume-title":"ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI). 1\u201311","author":"Cole Tom","year":"2022","unstructured":"Tom Cole and Marco Gillies. 2022. More than a bit of coding:(un-) Grounded (non-) theory in HCI. In ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI). 1\u201311. https:\/\/doi.org\/10.1145\/3491101.3516392 10.1145\/3491101.3516392"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1177\/1094428108324514"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","unstructured":"Edsger W Dijkstra. 1979. On the foolishness of \"natural language programming\". 51\u201353. https:\/\/doi.org\/10.1007\/BFb0014656 10.1007\/BFb0014656","DOI":"10.1007\/BFb0014656"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","unstructured":"Edsger W Dijkstra. 1982. On the role of scientific thought. Selected writings on computing: A personal perspective 60\u201366. https:\/\/doi.org\/10.1007\/978-1-4612-5695-3_12 10.1007\/978-1-4612-5695-3_12","DOI":"10.1007\/978-1-4612-5695-3_12"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3338857"},{"key":"e_1_2_1_26_1","volume-title":"IEEE\/ACM International Conference on Software Engineering (ICSE). 1\u201313","author":"Dolata Mateusz","year":"2024","unstructured":"Mateusz Dolata, Norbert Lange, and Gerhard Schwabe. 2024. Development in times of hype: How freelancers explore generative AI? In IEEE\/ACM International Conference on Software Engineering (ICSE). 1\u201313. https:\/\/doi.org\/10.1145\/3597503.3639111 10.1145\/3597503.3639111"},{"key":"e_1_2_1_27_1","unstructured":"Michal Furmakiewicz Chang Liu Angus Taylor and Ilya Venger. 2024. Design and evaluation of AI copilots\u2013Case studies of retail copilot templates. arXiv preprint arXiv:2407.09512."},{"key":"e_1_2_1_28_1","volume-title":"International Conference on Machine Learning (ICML). 10764\u201310799","author":"Gao Luyu","year":"2023","unstructured":"Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning (ICML). 10764\u201310799."},{"key":"e_1_2_1_29_1","volume-title":"Sensemaking Interfaces for Human Evaluation of Language Model Outputs. In NeurIPS: Workshop on Human Evaluation of Generative Models.","author":"Gero Katy Ilonka","year":"2022","unstructured":"Katy Ilonka Gero, Jonathan K Kummerfeld, and Elena L Glassman. 2022. Sensemaking Interfaces for Human Evaluation of Language Model Outputs. In NeurIPS: Workshop on Human Evaluation of Generative Models."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.111031"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.4324\/9780203793206"},{"key":"e_1_2_1_32_1","unstructured":"Ebtesam Al Haque Chris Brown Thomas D LaToza and Brittany Johnson. 2024. Information seeking using AI assistants. arXiv preprint arXiv:2408.04032."},{"key":"e_1_2_1_33_1","volume-title":"Companion Proceedings ACM International Conference on the Foundations of Software Engineering. 294\u2013305","author":"Hassan Ahmed E","year":"2024","unstructured":"Ahmed E Hassan, Dayi Lin, Gopi Krishnan Rajbahadur, Keheliya Gallaba, Filipe Roseiro Cogo, Boyuan Chen, Haoxiang Zhang, Kishanthan Thangarajah, Gustavo Oliva, and Jiahuei Lin. 2024. Rethinking software engineering in the era of foundation models: A curated catalogue of challenges in the development of trustworthy FMware. In Companion Proceedings ACM International Conference on the Foundations of Software Engineering. 294\u2013305. https:\/\/doi.org\/10.1145\/3663529.3663849 10.1145\/3663529.3663849"},{"key":"e_1_2_1_34_1","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL), Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). 4198\u20134205","author":"Jacovi Alon","year":"2020","unstructured":"Alon Jacovi and Yoav Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Annual Meeting of the Association for Computational Linguistics (ACL), Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). 4198\u20134205. https:\/\/doi.org\/10.18653\/v1\/2020.acl-main.386 10.18653\/v1\/2020.acl-main.386"},{"key":"e_1_2_1_35_1","volume-title":"ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI). 1\u20138. https:\/\/doi.org\/10","author":"Jiang Ellen","year":"2022","unstructured":"Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022. Promptmaker: Prompt-based prototyping with large language models. In ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI). 1\u20138. https:\/\/doi.org\/10.1145\/3491101.3503564 10.1145\/3491101.3503564"},{"key":"e_1_2_1_36_1","volume-title":"ACM Symposium on User Interface Software and Technology (UIST). 1\u201320","author":"Jiang Peiling","year":"2023","unstructured":"Peiling Jiang, Jude Rayan, Steven P Dow, and Haijun Xia. 2023. Graphologue: Exploring large language model responses with interactive diagrams. In ACM Symposium on User Interface Software and Technology (UIST). 1\u201320. https:\/\/doi.org\/10.1145\/3586183.3606737 10.1145\/3586183.3606737"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00324"},{"key":"e_1_2_1_38_1","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL). 6282\u20136293","author":"Joshi Pratik","year":"2020","unstructured":"Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Annual Meeting of the Association for Computational Linguistics (ACL). 6282\u20136293. https:\/\/doi.org\/10.18653\/v1\/2020.acl-main.560 10.18653\/v1\/2020.acl-main.560"},{"key":"e_1_2_1_39_1","volume-title":"ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI). 10","author":"Kery Mary Beth","year":"2017","unstructured":"Mary Beth Kery, Amber Horvath, and Brad A Myers. 2017. Variolite: Supporting exploratory programming by data scientists.. In ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI). 10, 3025453\u20133025626. https:\/\/doi.org\/10.1145\/3025453.3025626 10.1145\/3025453.3025626"},{"key":"e_1_2_1_40_1","volume-title":"IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC). 25\u201329","author":"Kery Mary Beth","year":"2017","unstructured":"Mary Beth Kery and Brad A Myers. 2017. Exploring exploratory programming. In IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC). 25\u201329. https:\/\/doi.org\/10.1109\/VLHCC.2017.8103446 10.1109\/VLHCC.2017.8103446"},{"key":"e_1_2_1_41_1","volume-title":"International Conference on Software Engineering (ICSE). 96\u2013107","author":"Kim Miryung","year":"2016","unstructured":"Miryung Kim, Thomas Zimmermann, Robert DeLine, and Andrew Begel. 2016. The emerging role of data scientists on software development teams. In International Conference on Software Engineering (ICSE). 96\u2013107. https:\/\/doi.org\/10.1145\/2884781.2884783 10.1145\/2884781.2884783"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1922649.1922658"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9279-3"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","unstructured":"J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics 159\u2013174. https:\/\/doi.org\/10.2307\/2529310 10.2307\/2529310","DOI":"10.2307\/2529310"},{"key":"e_1_2_1_45_1","volume-title":"International Conference on Software Engineering (ICSE). 492\u2013501","author":"LaToza Thomas D","year":"2006","unstructured":"Thomas D LaToza, Gina Venolia, and Robert DeLine. 2006. Maintaining mental models: a study of developer work habits. In International Conference on Software Engineering (ICSE). 492\u2013501. https:\/\/doi.org\/10.1145\/1134285.1134355 10.1145\/1134285.1134355"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","unstructured":"Jenny T Liang Melissa Lin Nikitha Rao and Brad Myers. 2025. Supplemental Materials to \"Prompts are programs too! Understanding how developers build software containing prompts\". https:\/\/doi.org\/10.6084\/m9.figshare.24210468 10.6084\/m9.figshare.24210468","DOI":"10.6084\/m9.figshare.24210468"},{"key":"e_1_2_1_47_1","volume-title":"IEEE\/ACM International Conference on Software Engineering (ICSE). 1\u201313","author":"Liang Jenny T","year":"2024","unstructured":"Jenny T Liang, Chenyang Yang, and Brad A Myers. 2024. A large-scale survey on the usability of AI programming assistants: Successes and challenges. In IEEE\/ACM International Conference on Software Engineering (ICSE). 1\u201313. https:\/\/doi.org\/10.1145\/3597503.3608128 10.1145\/3597503.3608128"},{"key":"e_1_2_1_48_1","volume-title":"ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE). 170\u2013182","author":"Liang Jenny T","year":"2022","unstructured":"Jenny T Liang, Thomas Zimmermann, and Denae Ford. 2022. Understanding skills for OSS communities on GitHub. In ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE). 170\u2013182. https:\/\/doi.org\/10.1145\/3540250.3549082 10.1145\/3540250.3549082"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-024-01847-2"},{"key":"e_1_2_1_50_1","volume-title":"ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI). 1\u20139. https:\/\/doi.org\/10","author":"Liu Michael Xieyang","year":"2024","unstructured":"Michael Xieyang Liu, Frederick Liu, Alexander J Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J Cai. 2024. \"We need structured output\": Towards user-centered constraints on large language model output. In ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI). 1\u20139. https:\/\/doi.org\/10.1145\/3613905.3650756 10.1145\/3613905.3650756"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3560815"},{"key":"e_1_2_1_52_1","volume-title":"ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). 643\u2013653","author":"Luo Qingzhou","year":"2014","unstructured":"Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). 643\u2013653. https:\/\/doi.org\/10.1145\/2635868.2635920 10.1145\/2635868.2635920"},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the IEEE\/ACM 3rd International Conference on AI Engineering-Software Engineering for AI. 166\u2013171","author":"Ma Wanqin","year":"2024","unstructured":"Wanqin Ma, Chenyang Yang, and Christian K\u00e4stner. 2024. (Why) is my prompt getting worse? Rethinking regression testing for evolving LLM APIs. In Proceedings of the IEEE\/ACM 3rd International Conference on AI Engineering-Software Engineering for AI. 166\u2013171. https:\/\/doi.org\/10.1145\/3644815.3644950 10.1145\/3644815.3644950"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2622669"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2648.2007.04436.x"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2025.3535332"},{"key":"e_1_2_1_57_1","volume-title":"IEEE\/ACM International Conference on AI Engineering\u2013Software Engineering for AI (CAIN). 171\u2013183","author":"Nahar Nadia","year":"2023","unstructured":"Nadia Nahar, Haoran Zhang, Grace Lewis, Shurui Zhou, and Christian K\u00e4stner. 2023. A meta-summary of challenges in building products with ML components\u2013Collecting experiences from 4758+ practitioners. In IEEE\/ACM International Conference on AI Engineering\u2013Software Engineering for AI (CAIN). 171\u2013183. https:\/\/doi.org\/10.1109\/CAIN58948.2023.00034 10.1109\/CAIN58948.2023.00034"},{"key":"e_1_2_1_58_1","volume-title":"IEEE\/ACM International Conference on Software Engineering (ICSE). 413\u2013425","author":"Nahar Nadia","year":"2022","unstructured":"Nadia Nahar, Shurui Zhou, Grace Lewis, and Christian K\u00e4stner. 2022. Collaboration challenges in building ml-enabled systems: Communication, documentation, engineering, and process. In IEEE\/ACM International Conference on Software Engineering (ICSE). 413\u2013425. https:\/\/doi.org\/10.1145\/3510003.3510209 10.1145\/3510003.3510209"},{"key":"e_1_2_1_59_1","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, and Alex Ray. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35 (2022), 27730\u201327744.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_60_1","unstructured":"Chris Parnin Gustavo Soares Rahul Pandita Sumit Gulwani Jessica Rich and Austin Z Henley. 2023. Building your own product copilot: Challenges opportunities and needs. arXiv preprint arXiv:2312.14231."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2024.3440190"},{"key":"e_1_2_1_62_1","volume-title":"Linguistic Annotation Workshop (LAW-XVII). Association for Computational Linguistics, 252\u2013265","author":"Pei Jiaxin","year":"2023","unstructured":"Jiaxin Pei and David Jurgens. 2023. When do annotator demographics matter? Measuring the influence of annotator demographics with the POPQUORN dataset. In Linguistic Annotation Workshop (LAW-XVII). Association for Computational Linguistics, 252\u2013265. https:\/\/doi.org\/10.18653\/v1\/2023.law-1.25 10.18653\/v1\/2023.law-1.25"},{"key":"e_1_2_1_63_1","unstructured":"Sida Peng Eirini Kalliamvakou Peter Cihon and Mert Demirer. 2023. The impact of AI on developer productivity: Evidence from GitHub Copilot. arXiv preprint arXiv:2302.06590."},{"key":"e_1_2_1_64_1","volume-title":"ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI). 1\u20137. https:\/\/doi.org\/10","author":"Qian Crystal","year":"2024","unstructured":"Crystal Qian, Emily Reif, and Minsuk Kahng. 2024. Understanding the dataset practitioners behind large language models. In ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI). 1\u20137. https:\/\/doi.org\/10.1145\/3613905.3651007 10.1145\/3613905.3651007"},{"key":"e_1_2_1_65_1","volume-title":"Annual Meeting of the Association for Computational Linguistics (ACL). 9080\u20139102","author":"Sebastin","year":"2023","unstructured":"Sebastin Santy*, Jenny T Liang*, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023. NLPositionality: Characterizing design biases of datasets and models. In Annual Meeting of the Association for Computational Linguistics (ACL). 9080\u20139102. https:\/\/doi.org\/10.18653\/v1\/2023.acl-long.505 10.18653\/v1\/2023.acl-long.505"},{"key":"e_1_2_1_66_1","unstructured":"Morgan Klaus Scheuerman Katta Spiel Oliver L Haimson Foad Hamidi and Stacy M Branham. 2020. HCI guidelines for gender equity and inclusivity."},{"key":"e_1_2_1_67_1","volume-title":"Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems (NeurIPs), 36","author":"Schick Timo","year":"2024","unstructured":"Timo Schick, Jane Dwivedi-Yu, Roberto Dess\u00ec, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems (NeurIPs), 36 (2024)."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-934613-12-5.50048-3"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1006\/imms.1993.1028"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2022.3209479"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.3316\/QRJ1102063"},{"key":"e_1_2_1_72_1","volume-title":"Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, and Faisal Azhar. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2937083"},{"key":"e_1_2_1_74_1","volume-title":"Wordflow: Social prompt engineering for large language models. arXiv preprint arXiv:2401.14447.","author":"Wang Zijie J","year":"2024","unstructured":"Zijie J Wang, Aishwarya Chakravarthy, David Munechika, and Duen Horng Chau. 2024. Wordflow: Social prompt engineering for large language models. arXiv preprint arXiv:2401.14447."},{"key":"e_1_2_1_75_1","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35 (2022), 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_76_1","unstructured":"Jules White Quchen Fu Sam Hays Michael Sandborn Carlos Olea Henry Gilbert Ashraf Elnashar Jesse Spencer-Smith and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv preprint arXiv:2302.11382."},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-55642-5_4"},{"key":"e_1_2_1_78_1","volume-title":"IEEE\/ACM International Conference on Software Engineering Workshops. 86\u201392","author":"Wolf Christine T","year":"2020","unstructured":"Christine T Wolf and Drew Paine. 2020. Sensemaking practices in the everyday work of AI\/ML software engineering. In IEEE\/ACM International Conference on Software Engineering Workshops. 86\u201392. https:\/\/doi.org\/10.1109\/TSE.2019.2937083 10.1109\/TSE.2019.2937083"},{"key":"e_1_2_1_79_1","volume-title":"ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI). 1\u201310","author":"Wu Tongshuang","year":"2022","unstructured":"Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. Promptchainer: Chaining large language model prompts through visual programming. In ACM CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI). 1\u201310. https:\/\/doi.org\/10.1145\/3491101.3519729 10.1145\/3491101.3519729"},{"key":"e_1_2_1_80_1","volume-title":"ACM CHI Conference on Human Factors in Computing Systems (CHI). 1\u201322","author":"Wu Tongshuang","year":"2022","unstructured":"Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In ACM CHI Conference on Human Factors in Computing Systems (CHI). 1\u201322. https:\/\/doi.org\/10.1145\/3491102.3517582 10.1145\/3491102.3517582"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","unstructured":"Qinyuan Ye Mohamed Ahmed Reid Pryzant and Fereshte Khani. 2024. Prompt engineering a prompt engineer. In Findings of the Association for Computational Linguistics ACL (ACL) Lun-Wei Ku Andre Martins and Vivek Srikumar (Eds.). 355\u2013385. https:\/\/doi.org\/10.18653\/v1\/2024.findings-acl.21 10.18653\/v1\/2024.findings-acl.21","DOI":"10.18653\/v1"},{"key":"e_1_2_1_82_1","unstructured":"JD Zamfirescu-Pereira Bjoern Hartmann and Qian Yang. 2023. Conversation regression testing: A design technique for prototyping generalizable prompt strategies for pre-trained language models. arXiv preprint arXiv:2302.03154."},{"key":"e_1_2_1_83_1","volume-title":"ACM Designing Interactive Systems Conference (DIS). 2206\u20132220","author":"Zamfirescu-Pereira JD","year":"2023","unstructured":"JD Zamfirescu-Pereira, Heather Wei, Amy Xiao, Kitty Gu, Grace Jung, Matthew G Lee, Bjoern Hartmann, and Qian Yang. 2023. Herding AI cats: Lessons from designing a chatbot by prompting GPT-3. In ACM Designing Interactive Systems Conference (DIS). 2206\u20132220. https:\/\/doi.org\/10.1145\/3563657.3596138 10.1145\/3563657.3596138"},{"key":"e_1_2_1_84_1","volume-title":"ACM CHI Conference on Human Factors in Computing Systems (CHI). 1\u201321","author":"Zamfirescu-Pereira JD","year":"2023","unstructured":"JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can\u2019t prompt: How non-AI experts try (and fail) to design LLM prompts. In ACM CHI Conference on Human Factors in Computing Systems (CHI). 1\u201321. https:\/\/doi.org\/10.1145\/3544548.3581388 10.1145\/3544548.3581388"},{"key":"e_1_2_1_85_1","volume-title":"ACM SIGPLAN International Symposium on Machine Programming (MAPS). 21\u201329","author":"Ziegler Albert","year":"2022","unstructured":"Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productivity assessment of neural code completion. In ACM SIGPLAN International Symposium on Machine Programming (MAPS). 21\u201329. https:\/\/doi.org\/10.1145\/3520312.3534864 10.1145\/3520312.3534864"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/3633453"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3729342","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T15:30:16Z","timestamp":1750347016000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3729342"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,19]]},"references-count":86,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2025,6,19]]}},"alternative-id":["10.1145\/3729342"],"URL":"https:\/\/doi.org\/10.1145\/3729342","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,19]]}}}