{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T21:11:15Z","timestamp":1776114675074,"version":"3.50.1"},"reference-count":84,"publisher":"Association for Computing Machinery (ACM)","issue":"4","funder":[{"name":"National Science Foundation","award":["CNS-2213791, 2414915"],"award-info":[{"award-number":["CNS-2213791, 2414915"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2025,8,31]]},"abstract":"<jats:p>\n            Prompting LLMs for complex tasks (e.g., building a trip advisor chatbot) needs humans to clearly articulate customized requirements (e.g., \u201cstart the response with a tl;dr\u201d). However, existing prompt engineering instructions often lack focused training on requirement articulation and instead tend to emphasize increasingly automatable strategies (e.g., tricks like adding role-plays and \u201cthink step-by-step\u201d). To address the gap, we introduce Requirement-Oriented Prompt Engineering (\n            <jats:sc>ROPE<\/jats:sc>\n            ), a paradigm that focuses human attention on generating clear, complete requirements during prompting. We implement\n            <jats:sc>ROPE<\/jats:sc>\n            through an assessment and training suite that provides deliberate practice with LLM-generated feedback. In a randomized controlled experiment with 30 novices,\n            <jats:sc>ROPE<\/jats:sc>\n            significantly outperforms conventional prompt engineering training (20% vs. 1% gains), a gap that automatic prompt optimization cannot close. Furthermore, we demonstrate a direct correlation between the quality of input requirements and LLM outputs. Our work paves the way to empower more end-users to build complex LLM applications.\n          <\/jats:p>","DOI":"10.1145\/3731756","type":"journal-article","created":{"date-parts":[[2025,4,24]],"date-time":"2025-04-24T08:36:18Z","timestamp":1745483778000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-8634-130X","authenticated-orcid":false,"given":"Qianou","family":"Ma","sequence":"first","affiliation":[{"name":"Human Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-3417-2447","authenticated-orcid":false,"given":"Weirui","family":"Peng","sequence":"additional","affiliation":[{"name":"Electrical Engineering, Columbia University, New York, New York, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5016-7296","authenticated-orcid":false,"given":"Chenyang","family":"Yang","sequence":"additional","affiliation":[{"name":"Software and Societal Systems Department, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4928-525X","authenticated-orcid":false,"given":"Hua","family":"Shen","sequence":"additional","affiliation":[{"name":"Information School, University of Washington, Seattle, Washington, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5850-4768","authenticated-orcid":false,"given":"Ken","family":"Koedinger","sequence":"additional","affiliation":[{"name":"Human Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1630-0588","authenticated-orcid":false,"given":"Tongshuang","family":"Wu","sequence":"additional","affiliation":[{"name":"Human Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA"}]}],"member":"320","published-online":{"date-parts":[[2025,8,18]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aej.2014.06.001"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642016"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3689042"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.501"},{"key":"e_1_3_3_6_2","doi-asserted-by":"crossref","first-page":"93","DOI":"10.18653\/v1\/2022.acl-demo.9","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations","author":"Bach Stephen","year":"2022","unstructured":"Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M. Saiful Bari, Thibault Fevry, et al. 2022. PromptSource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics, Stroudsburg, PA, 93\u2013104. DOI: 10.18653\/v1\/2022.acl-demo.9"},{"key":"e_1_3_3_7_2","first-page":"1877","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of the Advances in Neural Information Processing Systems. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33, Curran Associates, Inc., 1877\u20131901. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2020\/file\/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf"},{"key":"e_1_3_3_8_2","unstructured":"Sondos Mahmoud Bsharat Aidar Myrzakhan and Zhiqiang Shen. 2023. Principled instructions are all you need for questioning llama-1\/2 gpt-3.5\/4. arXiv:2312.16171. Retrieved from https:\/\/arxiv.org\/abs\/2312.16171"},{"key":"e_1_3_3_9_2","unstructured":"Santiago Castro. 2017. Fast Krippendorff: Fast Computation of Krippendorff\u2019s Alpha Agreement Measure. Retrieved from https:\/\/github.com\/pln-fing-udelar\/fast-krippendorff"},{"key":"e_1_3_3_10_2","unstructured":"Harrison Chase. 2022. LangChain. Retrieved from https:\/\/github.com\/langchain-ai\/langchain"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642377"},{"key":"e_1_3_3_12_2","first-page":"30","volume-title":"Proceedings of the 41st International Conference on Machine Learning (ICML \u201924)","author":"Chiang Wei-Lin","year":"2024","unstructured":"Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph Gonzalez, et al. 2024. Chatbot Arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning (ICML \u201924). JMLR.org, 30. DOI: 10.48550\/arXiv.2403.04132"},{"key":"e_1_3_3_13_2","unstructured":"Bhavya Chopra Ananya Singha Anna Fariha Sumit Gulwani Chris Parnin Ashish Tiwari and Austin Z. Henley. 2023. Conversational challenges in AI-powered data science: Obstacles needs and design opportunities. arXiv:2310.16164. Retrieved from http:\/\/arxiv.org\/abs\/2310.16164"},{"key":"e_1_3_3_14_2","first-page":"4302","volume-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS \u201917)","author":"Christiano Paul F.","year":"2017","unstructured":"Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS \u201917). Curran Associates Inc., Red Hook, NY, 4302\u20134310."},{"key":"e_1_3_3_15_2","unstructured":"DeepSeek-AI Daya Guo Dejian Yang Haowei Zhang Jun-Mei Song Ruoyu Zhang R. Xu Qihao Zhu Shirong Ma Peiyi Wang et al. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948. Retrieved from http:\/\/arxiv.org\/abs\/2501.12948"},{"key":"e_1_3_3_16_2","first-page":"296","volume-title":"Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE \u201924)","author":"Denny Paul","year":"2024","unstructured":"Paul Denny, Juho Leinonen, James Prather, Andrew Luxton-Reilly, Thezyrie Amarouche, Brett A. Becker, and Brent N. Reeves. 2024. Prompt problems: A new programming exercise for the generative AI era. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE \u201924). ACM, New York, NY, 296\u2013302. DOI: 10.1145\/3626252.3630909"},{"key":"e_1_3_3_17_2","volume-title":"How Do You Design","author":"Dubberly H.","year":"2004","unstructured":"H. Dubberly. 2004. How Do You Design. Dubberly Design Office."},{"key":"e_1_3_3_18_2","unstructured":"Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Amy Yang Angela Fan et al. 2024. The Llama 3 herd of models. arXiv:2407.21783. Retrieved from https:\/\/arxiv.org\/abs\/2407.21783"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/1124706.1121468"},{"key":"e_1_3_3_20_2","first-page":"1","volume-title":"Proceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work (CHIWORK \u201924)","author":"Feldman Molly Q.","year":"2024","unstructured":"Molly Q. Feldman and Carolyn Jane Anderson. 2024. Non-expert programmers in the generative AI future. In Proceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work (CHIWORK \u201924). ACM, New York, NY, 1\u201319. DOI: 10.1145\/3663384.3663393"},{"key":"e_1_3_3_21_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM","author":"Feng Li","year":"2024","unstructured":"Li Feng, Ryan Yen, Yuzhe You, Mingming Fan, Jian Zhao, and Zhicong Lu. 2024. CoPrompt: Supporting prompt sharing and referring in collaborative natural language programming. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1\u201321. DOI: 10.1145\/3613904.3642212"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2020.106321"},{"key":"e_1_3_3_23_2","first-page":"19","volume-title":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI \u201922)","volume":"386","author":"Jiang Ellen","year":"2022","unstructured":"Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J. Cai, and Michael Terry. 2022. Discovering the syntax and strategies of natural language programming with generative language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI \u201922). ACM, New York, NY, Article 386, 19 pages. DOI: 10.1145\/3491102.3501870"},{"key":"e_1_3_3_24_2","doi-asserted-by":"crossref","first-page":"4667","DOI":"10.18653\/v1\/2024.acl-long.257","volume-title":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Jiang Yuxin","year":"2024","unstructured":"Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024. FollowBench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Stroudsburg, PA, 4667\u20134688. DOI: 10.18653\/v1\/2024.acl-long.257"},{"key":"e_1_3_3_25_2","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1162\/tacl_a_00324","article-title":"How can we know what language models know","volume":"8","author":"Jiang Zhengbao","year":"2020","unstructured":"Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Trans. Assoc. Comput. Linguist. 8 (2020), 423\u2013438.","journal-title":"Trans. Assoc. Comput. Linguist"},{"key":"e_1_3_3_26_2","unstructured":"Omar Khattab Arnav Singhvi Paridhi Maheshwari Zhiyuan Zhang Keshav Santhanam Sri Vardhamanan Saiful Haq Ashutosh Sharma Thomas T. Joshi Hanna Moazam et al. 2023. DSPy: Compiling declarative language model calls into self-improving pipelines. arXiv:2310.03714. Retrieved from https:\/\/arxiv.org\/abs\/2310.03714"},{"key":"e_1_3_3_27_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems","volume":"35","author":"Soo Kim Tae","year":"2024","unstructured":"Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2024. EvalLM: Interactive evaluation of large language model prompts on user-defined criteria. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Vol. 35. ACM, New York, NY, 1\u201321. DOI: 10.1145\/3613904.3642216"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/1922649.1922658"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1238056"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcm.2016.02.012"},{"key":"e_1_3_3_31_2","unstructured":"Shuvendu K. Lahiri Aaditya Naik Georgios Sakkas Piali Choudhury Curtis von Veh Madanlal Musuvathi Jeevana Priya Inala Chenglong Wang and Jianfeng Gao. 2022. Interactive code generation via test-driven user-intent formalization. arXiv:2208.05950. Retrieved from https:\/\/arxiv.org\/abs\/2208.05950"},{"key":"e_1_3_3_32_2","first-page":"2124","volume-title":"Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI \u201917)","author":"Lakkaraju Himabindu","year":"2017","unstructured":"Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Eric Horvitz. 2017. Identifying unknown unknowns in the open world: Representations and policies for guided exploration. In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI \u201917). AAAI Press, 2124\u20132132."},{"key":"e_1_3_3_33_2","volume-title":"Requirements Engineering: From System Goals to UML Models to Software Specifications","author":"van Lamsweerde A","year":"2009","unstructured":"A van Lamsweerde. 2009. Requirements Engineering: From System Goals to UML Models to Software Specifications. John Wiley & Sons, Ltd."},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3502087"},{"key":"e_1_3_3_35_2","first-page":"1","volume-title":"Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI \u201923)","author":"Liu Michael Xieyang","year":"2023","unstructured":"Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn, Jack Williams, Neil Toronto, and Andrew D. Gordon. 2023. \u201cWhat it wants me to say\u201d: Bridging the abstraction gap between end-user programmers and code-generating large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI \u201923). ACM, New York, NY, Article 598, 1\u201331. DOI: 10.1145\/3544548.3580817"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3560815"},{"key":"e_1_3_3_37_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI \u201922). ACM","author":"Liu Vivian","year":"2022","unstructured":"Vivian Liu and Lydia B. Chilton. 2022. Design guidelines for prompt engineering text-to-image generative models. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI \u201922). ACM, New York, NY, 1\u201323. DOI: 10.1145\/3491102.3501825"},{"key":"e_1_3_3_38_2","doi-asserted-by":"crossref","first-page":"10570","DOI":"10.18653\/v1\/2024.emnlp-main.591","volume-title":"Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing","author":"Louie Ryan","year":"2024","unstructured":"Ryan Louie, Ananjan Nandi, William Fang, Cheng Chang, Emma Brunskill, and Diyi Yang. 2024. Roleplay-doh: Enabling domain-experts to create LLM-simulated patients via eliciting and adhering to principles. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.), Association for Computational Linguistics, Miami, Florida, 10570\u201310603. DOI: 10.18653\/v1\/2024.emnlp-main.591"},{"key":"e_1_3_3_39_2","first-page":"265","volume-title":"Proceedings of the 25th International Conference on Artificial Intelligence in Education (AIED)Lecture Notes in Computer Science","author":"Ma Qianou","year":"2024","unstructured":"Qianou Ma, Hua Shen, Kenneth Koedinger, and Sherry Tongshuang Wu. 2024. How to teach programming in the AI era? Using LLMs as a teachable agent for debugging. In Proceedings of the 25th International Conference on Artificial Intelligence in Education (AIED). Lecture Notes in Computer Science, Springer Nature, Switzerland, Cham, 265\u2013279. DOI: 10.1007\/978-3-031-64302-6_19"},{"key":"e_1_3_3_40_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems","author":"Goloujeh Atefeh Mahdavi","year":"2024","unstructured":"Atefeh Mahdavi Goloujeh, Anne Sullivan, and Brian Magerko. 2024. Is it AI or is it me? Understanding users\u2019 prompt journey with text-to-image generative AI tools. In Proceedings of the CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1\u201313. DOI: 10.1145\/3613904.3642861"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-99-7962-2_30"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.destud.2014.01.003"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.2196\/50638"},{"key":"e_1_3_3_44_2","volume-title":"Software for Dependable Systems: Sufficient Evidence?","author":"Millett Lynette I.","year":"2007","unstructured":"Lynette I. Millett, Martyn Thomas, and Daniel Jackson. 2007. Software for Dependable Systems: Sufficient Evidence? National Academies Press, Washington, DC."},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00766-021-00367-z"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3660810"},{"key":"e_1_3_3_47_2","first-page":"1","volume-title":"Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering (ICSE \u201924)","author":"Nam Daye","year":"2024","unstructured":"Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an LLM to help with code understanding. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering (ICSE \u201924). ACM, New York, NY, Article 97, 1\u201313. DOI: 10.1145\/3597503.3639187"},{"key":"e_1_3_3_48_2","doi-asserted-by":"crossref","first-page":"1099","DOI":"10.1109\/ICSE.2013.6606660","volume-title":"Proceedings of the 2013 35th International Conference on Software Engineering (ICSE)","author":"Neto Vicente Lustosa","year":"2013","unstructured":"Vicente Lustosa Neto, Roberta Coelho, Larissa Leite, Dalton S. Guerrero, and Andrea P. Mendonca. 2013. POPT: A problem-oriented programming and testing approach for novice students. In Proceedings of the 2013 35th International Conference on Software Engineering (ICSE). IEEE, 1099\u20131108. DOI: 10.1109\/icse.2013.6606660"},{"key":"e_1_3_3_49_2","first-page":"1","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI \u201924)","author":"Nguyen Sydney","year":"2024","unstructured":"Sydney Nguyen, Hannah Mclean Babe, Yangtian Zi, Arjun Guha, Carolyn Jane Anderson, and Molly Q. Feldman. 2024. How beginning programmers and code LLMs (mis)read each other. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI \u201924). ACM, New York, NY, Article 651, 1\u201326. DOI: 10.1145\/3613904.3642706"},{"key":"e_1_3_3_50_2","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 27730\u201327744.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3697010"},{"key":"e_1_3_3_52_2","first-page":"853","volume-title":"Proceedings of the 29th International Conference on Intelligent User Interfaces (IUI \u201924)","author":"Petridis Savvas","year":"2024","unstructured":"Savvas Petridis, Benjamin D. Wedin, James Wexler, Mahima Pushkarna, Aaron Donsbach, Nitesh Goyal, Carrie J. Cai, and Michael Terry. 2024. ConstitutionMaker: Interactively critiquing large language models by converting feedback into principles. In Proceedings of the 29th International Conference on Intelligent User Interfaces (IUI \u201924). ACM, New York, NY, 853\u2013868. DOI: 10.1145\/3640543.3645144"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3617367"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-long.61"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.772"},{"key":"e_1_3_3_56_2","first-page":"291","volume-title":"Companion Proceedings of the 36th International Conference on Software Engineering (ICSE Companion \u201914","author":"Radermacher Alex","year":"2014","unstructured":"Alex Radermacher, Gursimran Walia, and Dean Knudson. 2014. Investigating the skill gap between graduating students and industry expectations. In Companion Proceedings of the 36th International Conference on Software Engineering (ICSE Companion \u201914). ACM, New York, NY, 291\u2013300. DOI: 10.1145\/2591062.2591159"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1080\/02602938.2018.1504277"},{"key":"e_1_3_3_58_2","volume-title":"Known and Unknown: A Memoir","author":"Rumsfeld Donald","year":"2011","unstructured":"Donald Rumsfeld. 2011. Known and Unknown: A Memoir. Penguin."},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613905.3636268"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3672359.3672364"},{"issue":"5","key":"e_1_3_3_61_2","doi-asserted-by":"crossref","first-page":"1763","DOI":"10.1213\/ANE.0000000000002864","article-title":"Correlation coefficients: Appropriate use and interpretation","volume":"126","author":"Schober Patrick","year":"2018","unstructured":"Patrick Schober, Christa Boer, and Lothar A. Schwarte. 2018. Correlation coefficients: Appropriate use and interpretation. Anesth. Analg. 126, 5 (2018), 1763\u20131768.","journal-title":"Anesth. Analg"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.14778\/3685800.3685835"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3654777.3676450"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.346"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3490099.3511119"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2009.01.004"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1080\/09511920802232902"},{"key":"e_1_3_3_68_2","first-page":"6","volume-title":"Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (CHI EA \u201924)","volume":"493","author":"Wang Qiaosi","year":"2024","unstructured":"Qiaosi Wang, Sarah Walsh, Mei Si, Jeffrey Kephart, Justin D. Weisz, and Ashok K. Goel. 2024. Theory of mind in human-AI interaction. In Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems (CHI EA \u201924). ACM, New York, NY, Article 493, 6 pages. DOI: 10.1145\/3613905.3636308"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642803"},{"key":"e_1_3_3_70_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 35, 24824\u201324837.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_71_2","volume-title":"Understanding by Design","author":"Wiggins Grant P.","year":"2005","unstructured":"Grant P. Wiggins and Jay McTighe. 2005. Understanding by Design. ASCD. Retrieved from https:\/\/play.google.com\/store\/books\/details?id=N2EfKlyUN4QC"},{"key":"e_1_3_3_72_2","unstructured":"Ben Wodecki. 2024. Devin: AI Software Engineer That Codes Entire Projects from Single Prompt. Retrieved September 2 2024 from https:\/\/aibusiness.com\/automation\/ai-software-engineer-devin-codes-entire-projects-from-single-prompt"},{"key":"e_1_3_3_73_2","first-page":"1","volume-title":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI \u201922)","author":"Wu Tongshuang","year":"2022","unstructured":"Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. AI chains: Transparent and controllable human-AI interaction by chaining large language model prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI \u201922). ACM, New York, NY, Article 385, 1\u201322. DOI: 10.1145\/3491102.3517582"},{"key":"e_1_3_3_74_2","first-page":"306","volume-title":"Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201924)","author":"Yang Chenyang","year":"2024","unstructured":"Chenyang Yang, Yining Hong, Grace Lewis, Tongshuang Wu, and Christian K\u00e4stner. 2024. What is wrong with my model? Identifying systematic problems with semantic data slicing. In Proceedings of the 39th IEEE\/ACM International Conference on Automated Software Engineering (ASE \u201924). ACM, New York, NY, 306\u2013318. DOI: 10.1145\/3691620.3695033"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-emnlp.901"},{"key":"e_1_3_3_76_2","first-page":"50528","article-title":"Swe-agent: Agent-computer interfaces enable automated software engineering","volume":"37","author":"Yang John","year":"2024","unstructured":"John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. In Proceedings of the Advances in Neural Information Processing Systems, Vol. 37, 50528\u201350652.","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_77_2","first-page":"51","volume-title":"Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS \u201923)","volume":"2433","author":"Yu Yue","year":"2023","unstructured":"Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023. Large language model as attributed training data generator: A tale of diversity and bias. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS \u201923). Curran Associates Inc., Red Hook, NY, Article 2433, 51 pages."},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.472"},{"key":"e_1_3_3_79_2","unstructured":"Mert Yuksekgonul Federico Bianchi Joseph Boen Sheng Liu Zhi Huang Carlos Guestrin and James Zou. 2024. TextGrad: Automatic \u201cdifferentiation\u201d via text. arXiv:2406.07496. Retrieved from http:\/\/arxiv.org\/abs\/2406.07496"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581388"},{"key":"e_1_3_3_81_2","unstructured":"Michael J. Q. Zhang and Eunsol Choi. 2023. Clarify when necessary: Resolving ambiguity through interaction with LMs. arXiv:2311.09469. Retrieved from http:\/\/arxiv.org\/abs\/2311.09469"},{"key":"e_1_3_3_82_2","unstructured":"Zejun Zhang Li Zhang Xin Yuan Anlan Zhang Mengwei Xu and Feng Qian. 2024. A first look at GPT apps: Landscape and vulnerability. arXiv:2402.15105. Retrieved from http:\/\/arxiv.org\/abs\/2402.15105"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/3444689"},{"key":"e_1_3_3_84_2","unstructured":"Denny Zhou Nathanael Sch\u00e4rli Le Hou Jason Wei Nathan Scales Xuezhi Wang Dale Schuurmans Claire Cui Olivier Bousquet Quoc Le et al. 2022. Least-to-most prompting enables complex reasoning in large language models. arXiv:2205.10625. Retrieved from http:\/\/arxiv.org\/abs\/2205.10625"},{"key":"e_1_3_3_85_2","unstructured":"Jeffrey Zhou Tianjian Lu Swaroop Mishra Siddhartha Brahma Sujoy Basu Yi Luan Denny Zhou and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv:2311.07911. Retrieved from https:\/\/arxiv.org\/abs\/2311.07911"}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3731756","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,18]],"date-time":"2025-08-18T18:02:23Z","timestamp":1755540143000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3731756"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,18]]},"references-count":84,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,8,31]]}},"alternative-id":["10.1145\/3731756"],"URL":"https:\/\/doi.org\/10.1145\/3731756","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,18]]},"assertion":[{"value":"2024-12-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-04-15","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}