{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T21:16:06Z","timestamp":1783804566797,"version":"3.55.0"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"8","license":[{"start":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T00:00:00Z","timestamp":1731974400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The Copenhagen Symposium on Human-Centered Software Engineering AI","award":["G-2023-21020 and CF23-0208"],"award-info":[{"award-number":["G-2023-21020 and CF23-0208"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,11,30]]},"abstract":"<jats:p>As researchers today, we are witnessing a fundamental change in our technologically-enabled world due to the advent and diffusion of highly disruptive technologies such as generative Artificial Intelligence (AI), Augmented Reality (AR) and Virtual Reality (VR). In particular, software engineering has been profoundly affected by the transformative power of disruptive innovations for decades, with a significant impact of technical advancements on social dynamics due to its socio-technical nature. In this article, we reflect on the importance of formulating and addressing research problems in software engineering through a socio-technical lens, thus ensuring a holistic understanding of the complex phenomena in this field. We propose a research playbook with the aim of providing a guide to formulate compelling and socially relevant research questions and to identify the appropriate research strategies for empirical investigations, with an eye on the long-term implications of technologies or their use. We showcase how to apply the research playbook. Firstly, we show how it can be used retrospectively to reflect on a prior disruptive technology, Stack Overflow, and its impact on software development. Secondly, we show how it can be used to question the impact of two current disruptive technologies: AI and AR\/VR. Finally, we introduce a specialized GPT model to support the researcher in framing future investigations. We conclude by discussing the broader implications of adopting the playbook for both researchers and practitioners in software engineering and beyond.<\/jats:p>","DOI":"10.1145\/3678172","type":"journal-article","created":{"date-parts":[[2024,7,15]],"date-time":"2024-07-15T14:25:39Z","timestamp":1721053539000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["A Disruptive Research Playbook for Studying Disruptive Innovations"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2278-2536","authenticated-orcid":false,"given":"Margaret-Anne","family":"Storey","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Victoria, BC, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7253-101X","authenticated-orcid":false,"given":"Daniel","family":"Russo","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Aalborg University, Copenhagen, Denmark"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1160-2608","authenticated-orcid":false,"given":"Nicole","family":"Novielli","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Bari, Bari, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2235-9992","authenticated-orcid":false,"given":"Takashi","family":"Kobayashi","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Tokyo Institute of Technology, Tokyo, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2004-0902","authenticated-orcid":false,"given":"Dong","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,11,19]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"GitHub Copilot. 2021. https:\/\/copilot.github.com"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2017.04.005"},{"key":"e_1_3_3_4_2","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1109\/SP.2016.25","volume-title":"Proceedings of the 2016 IEEE Symposium on Security and Privacy (SP)","author":"Acar Yasemin","year":"2016","unstructured":"Yasemin Acar, Michael Backes, Sascha Fahl, Doowon Kim, Michelle L. Mazurek, and Christian Stransky. 2016. You get where you\u2019re looking for: The impact of information sources on code security. In Proceedings of the 2016 IEEE Symposium on Security and Privacy (SP). 289\u2013305. DOI: 10.1109\/SP.2016.25"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488398"},{"key":"e_1_3_3_6_2","first-page":"97","volume-title":"Proceedings of the 10th International Working Conference on Mining Software Repositories","author":"Asaduzzaman Muhammad","year":"2013","unstructured":"Muhammad Asaduzzaman, Ahmed Shah Mashiyat, Chanchal K. Roy, and Kevin A. Schneider. 2013. Answering questions about unanswered questions of Stack Overflow. In Proceedings of the 10th International Working Conference on Mining Software Repositories. IEEE, 97\u2013100."},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/RSSE.2012.6233404"},{"issue":"1","key":"e_1_3_3_8_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s42467-020-00005-4","article-title":"Applications of AI in classical software engineering","volume":"2","author":"Barenkamp Marco","year":"2020","unstructured":"Marco Barenkamp, Jonas Rebstadt, and Oliver Thomas. 2020. Applications of AI in classical software engineering. AI Perspectives 2, 1 (2020), 1.","journal-title":"AI Perspectives"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-012-9231-y"},{"key":"e_1_3_3_10_2","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1007\/978-1-4614-6596-6_15","article-title":"Facilitating crowd sourced software engineering via stack overflow","author":"Barzilay Ohad","year":"2013","unstructured":"Ohad Barzilay, Christoph Treude, and Alexey Zagalsky. 2013. Facilitating crowd sourced software engineering via stack overflow. Finding Source Code on the Web for Remix and Reuse (2013), 289\u2013308.","journal-title":"Finding Source Code on the Web for Remix and Reuse"},{"key":"e_1_3_3_12_2","first-page":"81","volume-title":"Proceedings of the 2021 IEEE\/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE)","author":"Bhasin Trishala","year":"2021","unstructured":"Trishala Bhasin, Adam Murray, and Margaret-Anne Storey. 2021. Student experiences with github and stack overflow: An exploratory study. In Proceedings of the 2021 IEEE\/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). IEEE, 81\u201390."},{"key":"e_1_3_3_13_2","first-page":"89","volume-title":"Proceedings of the 10th International Working Conference on Mining Software Repositories","author":"Bosu Amiangshu","year":"2013","unstructured":"Amiangshu Bosu, Christopher S. Corley, Dustin Heaton, Debarshi Chatterji, Jeffrey C. Carver, and Nicholas A Kraft. 2013. Building reputation in StackOverflow: An empirical investigation. In Proceedings of the 10th International Working Conference on Mining Software Repositories. IEEE, 89\u201392."},{"issue":"1","key":"e_1_3_3_14_2","first-page":"43","article-title":"Disruptive technologies: Catching the wave","volume":"73","author":"Bower Joseph L.","year":"1995","unstructured":"Joseph L. Bower and Clayton M. Christensen. 1995. Disruptive technologies: Catching the wave. Harvard Business Review 73, 1 (1995), 43\u201353.","journal-title":"Harvard Business Review"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.4521754"},{"key":"e_1_3_3_16_2","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1109\/MSR.2015.56","volume-title":"Proceedings of the 2015 IEEE\/ACM 12th Working Conference on Mining Software Repositories","author":"Calefato Fabio","year":"2015","unstructured":"Fabio Calefato, Filippo Lanubile, Maria Concetta Marasciulo, and Nicole Novielli. 2015. Mining successful answers in stack overflow. In Proceedings of the 2015 IEEE\/ACM 12th Working Conference on Mining Software Repositories. 430\u2013433. DOI: 10.1109\/MSR.2015.56"},{"key":"e_1_3_3_17_2","first-page":"232","volume-title":"Proceedings of the 26th Annual International Conference on Computer Science and Software Engineering (CASCON \u201916)","author":"Campos Eduardo C.","year":"2016","unstructured":"Eduardo C. Campos, Martin Monperrus, and Marcelo A. Maia. 2016. Searching stack overflow for API-usage-related bug fixes using snippet-based queries. In Proceedings of the 26th Annual International Conference on Computer Science and Software Engineering (CASCON \u201916). IBM Corp., USA, 232\u2013242."},{"key":"e_1_3_3_18_2","first-page":"1273","volume-title":"Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE)","author":"Cao Kaibo","year":"2021","unstructured":"Kaibo Cao, Chunyang Chen, Sebastian Baltes, Christoph Treude, and Xiang Chen. 2021. Automated Query Reformulation for Efficient Search Based on Query Logs From Stack Overflow. In Proceedings of the 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE). 1273\u20131285. DOI: 10.1109\/ICSE43902.2021.00116"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2019.110454"},{"key":"e_1_3_3_20_2","unstructured":"Mark Chen Jakub Tworek Heewoo Jun Qijing Yuan Henryk P. de Oliveira Pinto Jared Kaplan Harrison Edwards Yuri Burda Niru Joseph Greg Brockman Alexander Ray Rishita Puri Gabriel Krueger Mike Petrov Hala Khlaaf Girish Sastry Pamela Mishkin Ben Chan Scott Gray Nick Ryder Maxim Pavlov Alex Power Lukasz Kaiser Maximilian Bavarian Carolyn Winter Philippe Tillet Felipe P. Such David Cummings Matthias Plappert Fotios Chantzis Eric Barnes Ariel Herbert-Voss William H. Guss Alex Nichol Andrew Paino Nick Tezak Jerry Tang Igor Babuschkin Sujith Balaji Shivendra Jain Will Saunders Christopher Hesse Andrew N. Carr Jan Leike Joshua Achiam Vedant Misra Eri Morikawa Alec Radford Melody Knight Miles Brundage Matej Murati Katja Mayer Peter Welinder Brian McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"issue":"2","key":"e_1_3_3_21_2","first-page":"66","article-title":"Meeting the challenge of disruptive change","volume":"78","author":"Christensen Clayton M.","year":"2000","unstructured":"Clayton M. Christensen and Michael Overdorf. 2000. Meeting the challenge of disruptive change. Harvard Business Review 78, 2 (2000), 66\u201376.","journal-title":"Harvard Business Review"},{"key":"e_1_3_3_22_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/BF01409793","article-title":"Collaborative virtual environments: An introductory review of issues and systems","volume":"3","author":"Churchill Elizabeth F.","year":"1998","unstructured":"Elizabeth F. Churchill and Dave Snowdon. 1998. Collaborative virtual environments: An introductory review of issues and systems. Virtual Reality 3 (1998), 3\u201315.","journal-title":"Virtual Reality"},{"key":"e_1_3_3_23_2","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1109\/SEAA.2019.00022","volume-title":"Proceedings of the 2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)","author":"Digkas Georgios","year":"2019","unstructured":"Georgios Digkas, Nikolaos Nikolaidis, Apostolos Ampatzoglou, and Alexander Chatzigeorgiou. 2019. Reusing code from StackOverflow: The effect on technical debt. In Proceedings of the 2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). 87\u201391. DOI: 10.1109\/SEAA.2019.00022"},{"key":"e_1_3_3_24_2","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1145\/3387940.3391534","volume-title":"Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering Workshops) (ICSEW \u201920)","author":"Dominic James","year":"2020","unstructured":"James Dominic, Jada Houser, Igor Steinmacher, Charles Ritter, and Paige Rodeghero. 2020. Conversational bot for newcomers onboarding to open source projects. In Proceedings of the IEEE\/ACM 42nd International Conference on Software Engineering Workshops) (ICSEW \u201920). ACM, New York, NY, 46\u201350. DOI: 10.1145\/3387940.3391534"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84800-044-5_11"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2023.3265877"},{"key":"e_1_3_3_27_2","doi-asserted-by":"crossref","first-page":"2630","DOI":"10.1007\/s10664-020-09818-7","article-title":"How software engineering research aligns with design science: a review","volume":"25","author":"Engstr\u00f6m Emelie","year":"2020","unstructured":"Emelie Engstr\u00f6m, Margaret-Anne Storey, Per Runeson, Martin H\u00f6st, and Maria Teresa Baldassarre. 2020. How software engineering research aligns with design science: a review. Empirical Software Engineering 25 (2020), 2630\u20132660.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_3_28_2","first-page":"49","volume-title":"Proceedings of the Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing (CSCW \u201921 Companion)","author":"Fang Jingchao","year":"2021","unstructured":"Jingchao Fang, Victoria Chang, Ge Gao, and Hao-Chuan Wang. 2021. Social interactions in virtual reality: What cues do people use most and how. In Proceedings of the Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing (CSCW \u201921 Companion). ACM, New York, NY, 49\u201352. DOI: 10.1145\/3462204.3481772"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1162\/pres_a_00371"},{"key":"e_1_3_3_30_2","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1109\/SP.2017.31","volume-title":"Proceedings of the 2017 IEEE Symposium on Security and Privacy (SP)","author":"Fischer Felix","year":"2017","unstructured":"Felix Fischer, Konstantin B\u00f6ttinger, Huang Xiao, Christian Stransky, Yasemin Acar, Michael Backes, and Sascha Fahl. 2017. Stack overflow considered harmful? The impact of copy & paste on android application security. In Proceedings of the 2017 IEEE Symposium on Security and Privacy (SP). 121\u2013136. DOI: 10.1109\/SP.2017.31"},{"key":"e_1_3_3_31_2","first-page":"1","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI \u201918)","author":"Ford Denae","year":"2018","unstructured":"Denae Ford, Kristina Lustig, Jeremy Banks, and Chris Parnin. 2018. \u201cWe don\u2019t do that here\u201d: How collaborative editing with mentors improves engagement in social Q & A communities. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI \u201918). ACM, New York, NY, 1\u201312. DOI: 10.1145\/3173574.3174182"},{"key":"e_1_3_3_32_2","doi-asserted-by":"crossref","first-page":"846","DOI":"10.1145\/2950290.2950331","volume-title":"Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering","author":"Ford Denae","year":"2016","unstructured":"Denae Ford, Justin Smith, Philip J Guo, and Chris Parnin. 2016. Paradise unplugged: Identifying barriers for female participation on stack overflow. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. 846\u2013857."},{"key":"e_1_3_3_33_2","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1109\/MSR.2013.6624007","volume-title":"Proceedings of the 2013 10th Working Conference on Mining Software Repositories (MSR)","author":"Grant Scott","year":"2013","unstructured":"Scott Grant and Buddy Betts. 2013. Encouraging user behaviour with achievements: An empirical study. In Proceedings of the 2013 10th Working Conference on Mining Software Repositories (MSR). 65\u201368."},{"key":"e_1_3_3_34_2","unstructured":"Catherine M. Hicks. 2024. Psychological Affordances Can Provide a Missing Explanatory Layer for Why Interventions to Improve Developer Experience Take Hold or Fail. Retrieved from https:\/\/files.osf.io\/v1\/resources\/qz43x\/providers\/osfstorage\/65b2f3ae4aa63c07d9df22ec?action=download&direct&version=5"},{"key":"e_1_3_3_35_2","volume-title":"Proceedings of the 39th IEEE International Conference on Software Maintenance and Evolution","author":"Hoff Adrian","year":"2023","unstructured":"Adrian Hoff, Christoph Seidl, Mircea F Lungu, and Michele Lanza. 2023. Preparing software re-engineering via freehand sketches in virtual reality. In Proceedings of the 39th IEEE International Conference on Software Maintenance and Evolution. IEEE."},{"key":"e_1_3_3_36_2","first-page":"22 pages","volume-title":"Proceedings of the ACM on Human-Computer Interaction 4, CSCW3","author":"Hoppe Adrian H.","year":"2021","unstructured":"Adrian H. Hoppe, Florian van de Camp, and Rainer Stiefelhagen. 2021. ShiSha: Enabling shared perspective with face-to-face collaboration using redirected avatars in virtual reality. Proceedings of the ACM on Human-Computer Interaction 4, CSCW3 (Jan 2021), Article 251, 22 pages. DOI: 10.1145\/3432950"},{"key":"e_1_3_3_37_2","doi-asserted-by":"crossref","unstructured":"Xinyi Hou Yanjie Zhao Yue Liu Zhou Yang Kailong Wang Li Li Xiapu Luo David Lo John Grundy and Haoyu Wang. 2023. Large language models for software engineering: A systematic literature review. arXiv:2308.10620. Retrieved from https:\/\/arxiv.org\/abs\/2308.10620","DOI":"10.1145\/3695988"},{"key":"e_1_3_3_38_2","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1145\/3510454.3522684","volume-title":"Proceedings of the ACM\/IEEE 44th International Conference on Software Engineering: Companion Proceedings (ICSE \u201922)","author":"Imai Saki","year":"2022","unstructured":"Saki Imai. 2022. Is GitHub copilot a substitute for human pair-programming? An empirical study. In Proceedings of the ACM\/IEEE 44th International Conference on Software Engineering: Companion Proceedings (ICSE \u201922). ACM, New York, NY, 319\u2013321. DOI: 10.1145\/3510454.3522684"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.displa.2022.102283"},{"key":"e_1_3_3_40_2","doi-asserted-by":"crossref","first-page":"107007","DOI":"10.1016\/j.infsof.2022.107007","article-title":"Collaborative program comprehension via software visualization in extended reality","volume":"151","author":"Krause-Glau Alexander","year":"2022","unstructured":"Alexander Krause-Glau, Malte Hansen, and Wilhelm Hasselbring. 2022. Collaborative program comprehension via software visualization in extended reality. Information and Software Technology 151 (2022), Article 107007.","journal-title":"Information and Software Technology"},{"key":"e_1_3_3_41_2","first-page":"1","volume-title":"Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems","author":"Krau\u00df Veronika","year":"2021","unstructured":"Veronika Krau\u00df, Alexander Boden, Leif Oppermann, and Ren\u00e9 Reiners. 2021. Current practices, challenges, and design implications for collaborative ar\/vr application development. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1\u201315."},{"key":"e_1_3_3_42_2","volume-title":"Proceedings of the 44th International Conference on Software Engineering (ICSE \u201922)","author":"Lorey Tobias","year":"2022","unstructured":"Tobias Lorey, Paul Ralph, and Michael Felderer. 2022. Social science theories in software engineering research. In Proceedings of the 44th International Conference on Software Engineering (ICSE \u201922). ACM, New York, NY, 1994\u20132005. DOI: 10.1145\/3510003.3510076"},{"key":"e_1_3_3_43_2","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1109\/MSR.2012.6224293","volume-title":"Proceedings of the 9th IEEE Working Conference of Mining Software Repositories (MSR \u201912)","author":"Lotufo Rafael","year":"2012","unstructured":"Rafael Lotufo, Leonardo Teixeira Passos, and Krzysztof Czarnecki. 2012. Towards improving bug tracking systems with game mechanisms. In Proceedings of the 9th IEEE Working Conference of Mining Software Repositories (MSR \u201912). Michele Lanza, Massimiliano Di Penta, and Tao Xie (Eds.), IEEE, 2\u201311. DOI: 10.1109\/MSR.2012.6224293"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/MAHC.2010.11"},{"key":"e_1_3_3_45_2","doi-asserted-by":"crossref","first-page":"2857","DOI":"10.1145\/1978942.1979366","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI \u201911)","author":"Mamykina Lena","year":"2011","unstructured":"Lena Mamykina, Bella Manoim, Manas Mittal, George Hripcsak, and Bj\u00f6rn Hartmann. 2011. Design lessons from the fastest Q & a site in the west. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI \u201911). ACM, New York, NY, 2857\u20132866. DOI: 10.1145\/1978942.1979366"},{"key":"e_1_3_3_46_2","first-page":"152","volume-title":"Readings in Human\u2013Computer Interaction","author":"McGrath Joseph E.","year":"1995","unstructured":"Joseph E. McGrath. 1995. Methodology matters: Doing research in the behavioral and social sciences. In Readings in Human\u2013Computer Interaction. Elsevier, 152\u2013169."},{"issue":"2","key":"e_1_3_3_47_2","first-page":"173","article-title":"Laws of the Media","volume":"34","author":"McLuhan Marshall","year":"1977","unstructured":"Marshall McLuhan. 1977. Laws of the Media. ETC: A Review of General Semantics 34, 2 (1977), 173\u2013179.","journal-title":"ETC: A Review of General Semantics"},{"key":"e_1_3_3_48_2","doi-asserted-by":"crossref","first-page":"390","DOI":"10.4324\/9781315080918-31","volume-title":"Communication Theory","author":"McLuhan Marshall","year":"2017","unstructured":"Marshall McLuhan. 2017. The medium is the message. In Communication Theory. Routledge, 390\u2013402."},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2020.102516"},{"key":"e_1_3_3_50_2","unstructured":"Sarah Meldrum Sherlock A. Licorish and Bastin Tony Roy Savarimuthu. 2020. Exploring research interest in stack overflow \u2013 A systematic mapping study and quality evaluation. arXiv:2010.12282. Retrieved from https:\/\/arxiv.org\/abs\/2010.12282"},{"key":"e_1_3_3_51_2","doi-asserted-by":"crossref","first-page":"3086","DOI":"10.1145\/3308558.3313583","volume-title":"Proceedings of the World Wide Web Conference","author":"Merchant Arpit","year":"2019","unstructured":"Arpit Merchant, Daksh Shah, Gurpreet Singh Bhatia, Anurag Ghosh, and Ponnurangam Kumaraguru. 2019. Signals matter: Understanding popularity and impact of users on stack overflow. In Proceedings of the World Wide Web Conference (WWW \u201919). ACM, New York, NY, 3086\u20133092. DOI: 10.1145\/3308558.3313583"},{"key":"e_1_3_3_52_2","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780195074772.001.0001","volume-title":"The Lever of Riches: Technological Creativity and Economic Progress","author":"Mokyr Joel","year":"1992","unstructured":"Joel Mokyr. 1992. The Lever of Riches: Technological Creativity and Economic Progress. Oxford University Press."},{"issue":"6","key":"e_1_3_3_53_2","doi-asserted-by":"crossref","first-page":"e0253010","DOI":"10.1371\/journal.pone.0253010","article-title":"Community evolution on stack overflow","volume":"16","author":"Moutidis Iraklis","year":"2021","unstructured":"Iraklis Moutidis and Hywel T. P. Williams. 2021. Community evolution on stack overflow. PLoS One 16, 6 (2021), Article e0253010.","journal-title":"PLoS One"},{"key":"e_1_3_3_54_2","first-page":"1272","volume-title":"Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA \u201916)","author":"Murgia Alessandro","year":"2016","unstructured":"Alessandro Murgia, Daan Janssens, Serge Demeyer, and Bogdan Vasilescu. 2016. Among the machines: Human-bot interaction on social Q & A websites. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA \u201916). ACM, New York, NY, 1272\u20131279. DOI: 10.1145\/2851581.2892311"},{"issue":"4","key":"e_1_3_3_55_2","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1109\/MS.2006.105","article-title":"How are java software developers using the eclipse ide?","volume":"23","author":"Murphy Gail","year":"2006","unstructured":"Gail Murphy, Mik Kersten, and Leah Findlater. 2006. How are java software developers using the eclipse ide? IEEE Software 23, 4 (2006), 76\u201383.","journal-title":"IEEE Software"},{"key":"e_1_3_3_56_2","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1145\/2661685.2661689","volume-title":"Proceedings of the 6th International Workshop on Social Software Engineering","author":"Novielli Nicole","year":"2014","unstructured":"Nicole Novielli, Fabio Calefato, and Filippo Lanubile. 2014. Towards discovering the role of emotions in stack overflow. In Proceedings of the 6th International Workshop on Social Software Engineering. 33\u201336."},{"key":"e_1_3_3_57_2","unstructured":"OpenAI. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_3_58_2","volume-title":"Proceedings of the International Symposium on Software Testing and Analysis","author":"Parnin Chris","year":"2011","unstructured":"Chris Parnin and Alessandro Orso. 2011. Are automated debugging techniques actually helping programmers?. In Proceedings of the International Symposium on Software Testing and Analysis. ACM."},{"key":"e_1_3_3_60_2","unstructured":"Sida Peng Eirini Kalliamvakou Peter Cihon and Mert Demirere. 2023. The impact of AI on developer productivity: Evidence from GitHub. arXiv:2302.06590. Retrieved from http:\/\/arxiv.org\/abs\/2302.06590"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSPEC.2016.7367470"},{"key":"e_1_3_3_62_2","doi-asserted-by":"crossref","first-page":"703","DOI":"10.1007\/978-3-030-68198-2_65","volume-title":"Educating Engineers for Future Industrial Revolutions","author":"P\u00f6ial Jaanus","year":"2021","unstructured":"Jaanus P\u00f6ial. 2021. Challenges of teaching programming in StackOverflow era. In Educating Engineers for Future Industrial Revolutions. Michael E. Auer and Tiia R\u00fc\u00fctmann (Eds.), Springer International Publishing, Cham, 703\u2013710."},{"key":"e_1_3_3_63_2","doi-asserted-by":"crossref","first-page":"1295","DOI":"10.1109\/ICSE.2013.6606701","volume-title":"Proceedings of the 2013 35th International Conference on Software Engineering (ICSE)","author":"Ponzanelli Luca","year":"2013","unstructured":"Luca Ponzanelli, Alberto Bacchelli, and Michele Lanza. 2013. Seahawk: Stack overflow in the ide. In Proceedings of the 2013 35th International Conference on Software Engineering (ICSE). IEEE, 1295\u20131298."},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2900307"},{"key":"e_1_3_3_65_2","unstructured":"P. Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard Nicolau de Fran\u00e7a Carlo Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Mart\u00ednez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarim\u00e4ki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2020. ACM SIGSOFT empirical standards. arXiv:2010.03525. Retrieved from https:\/\/arxiv.org\/abs\/2010.03525"},{"key":"e_1_3_3_66_2","unstructured":"Paul Ralph Nauman bin Ali Sebastian Baltes Domenico Bianculli Jessica Diaz Yvonne Dittrich Neil Ernst Michael Felderer Robert Feldt Antonio Filieri Breno Bernard Nicolau de Fran\u00e7a Carlo Alberto Furia Greg Gay Nicolas Gold Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara Kitchenham Valentina Lenarduzzi Jorge Mart\u00ednez Jorge Melegati Daniel Mendez Tim Menzies Jefferson Molleri Dietmar Pfahl Romain Robbes Daniel Russo Nyyti Saarim\u00e4ki Federica Sarro Davide Taibi Janet Siegmund Diomidis Spinellis Miroslaw Staron Klaas Stol Margaret-Anne Storey Davide Taibi Damian Tamburri Marco Torchiano Christoph Treude Burak Turhan Xiaofeng Wang and Sira Vegas. 2020. Empirical standards for software engineering research. arXiv:2010.03525. Retrieved from https:\/\/arxiv.org\/abs\/2010.03525"},{"key":"e_1_3_3_67_2","doi-asserted-by":"crossref","first-page":"491","DOI":"10.1145\/3581641.3584037","volume-title":"Proceedings of the 28th International Conference on Intelligent User Interfaces","author":"Ross Steven I.","year":"2023","unstructured":"Steven I. Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D. Weisz. 2023. The programmer\u2019s assistant: Conversational interaction with a large language model for software development. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 491\u2013514."},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.111063"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84800-044-5_12"},{"key":"e_1_3_3_70_2","first-page":"219","volume-title":"Proceedings of the 37th International Conference on Software Engineering (ICSE \u201915)","volume":"2","author":"Squire Megan","year":"2015","unstructured":"Megan Squire. 2015. Should we move to stack overflow? Measuring the utility of social media for developer support. In Proceedings of the 37th International Conference on Software Engineering (ICSE \u201915), Vol. 2. IEEE, 219\u2013228."},{"key":"e_1_3_3_71_2","doi-asserted-by":"crossref","first-page":"1425","DOI":"10.1109\/HICSS.2014.185","volume-title":"Proceedings of the 2014 47th Hawaii International Conference on System Sciences","author":"Squire Megan","year":"2014","unstructured":"Megan Squire and Christian Funkhouser. 2014. \u201cA bit of code\u201d: How the stack overflow community creates quality postings. In Proceedings of the 2014 47th Hawaii International Conference on System Sciences. 1425\u20131434. DOI: 10.1109\/HICSS.2014.185"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/GTSE.2013.6613863"},{"key":"e_1_3_3_73_2","doi-asserted-by":"crossref","first-page":"4097","DOI":"10.1007\/s10664-020-09858-z","article-title":"The who, what, how of software engineering research: A socio-technical framework","volume":"25","author":"Storey Margaret-Anne","year":"2020","unstructured":"Margaret-Anne Storey, Neil A Ernst, Courtney Williams, and Eirini Kalliamvakou. 2020. The who, what, how of software engineering research: A socio-technical framework. Empirical Software Engineering 25 (2020), 4097\u20134129.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_3_74_2","first-page":"85","volume-title":"Proceedings of the 10th International Working Conference on Mining Software Repositories","author":"Subramanian Siddharth","year":"2013","unstructured":"Siddharth Subramanian and Reid Holmes. 2013. Making sense of online code snippets. In Proceedings of the 10th International Working Conference on Mining Software Repositories. IEEE, 85\u201388."},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-09954-8"},{"key":"e_1_3_3_77_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_3_78_2","first-page":"804","volume-title":"Proceedings of the 33rd International Conference on Software Engineering (ICSE \u201911)","author":"Treude Christoph","year":"2011","unstructured":"Christoph Treude, Ohad Barzilay, and Margaret-Anne D. Storey. 2011. How do programmers ask and answer questions on the web?. In Proceedings of the 33rd International Conference on Software Engineering (ICSE \u201911). Richard N. Taylor, Harald C. Gall, and Nenad Medvidovic (Eds.), ACM, New York, NY, 804\u2013807. DOI: 10.1145\/1985793.1985907"},{"key":"e_1_3_3_79_2","doi-asserted-by":"crossref","first-page":"392","DOI":"10.1145\/2884781.2884800","volume-title":"Proceedings of the 38th International Conference on Software Engineering","author":"Treude Christoph","year":"2016","unstructured":"Christoph Treude and Martin P Robillard. 2016. Augmenting API documentation with insights from stack overflow. In Proceedings of the 38th International Conference on Software Engineering. 392\u2013403."},{"key":"e_1_3_3_80_2","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1109\/ICSME.2017.24","volume-title":"Proceedings of the 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)","author":"Treude Christoph","year":"2017","unstructured":"Christoph Treude and Martin P. Robillard. 2017. Understanding stack overflow code fragments. In Proceedings of the 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). 509\u2013513. DOI: 10.1109\/ICSME.2017.24"},{"key":"e_1_3_3_81_2","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1109\/SocialCom.2013.35","volume-title":"Proceedings of the 2013 ASE\/IEEE International Conference on Social Computing","author":"Vasilescu Bogdan","year":"2013","unstructured":"Bogdan Vasilescu, Vladimir Filkov, and Alexander Serebrenik. 2013. StackOverflow and GitHub: Associations between software development and crowdsourced knowledge. In Proceedings of the 2013 ASE\/IEEE International Conference on Social Computing. IEEE, 188\u2013195."},{"key":"e_1_3_3_82_2","unstructured":"Stefan Wagner and G\u00fcnter Ruhe. 2018. A systematic review of productivity factors in software development. arXiv:1801.06475. Retrieved from https:\/\/arxiv.org\/abs\/1801.06475"},{"key":"e_1_3_3_83_2","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-29044-2","volume-title":"Experimentation in Software Engineering","author":"Wohlin C.","year":"2012","unstructured":"C. Wohlin, P. Runeson, M. H\u00f6st, M. Ohlsson, B. Regnell, and A. Wessl\u00e9n. 2012. Experimentation in Software Engineering. Springer Science & Business Media."},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-018-9634-5"},{"issue":"9","key":"e_1_3_3_85_2","first-page":"e2531","article-title":"Examining the training and education potential of the metaverse: Results from an empirical study of next generation SAFe training","volume":"35","author":"Yilmaz Murat","year":"2023","unstructured":"Murat Yilmaz, Emer O\u2019farrell, and Paul Clarke. 2023. Examining the training and education potential of the metaverse: Results from an empirical study of next generation SAFe training. Journal of Software: Evolution and Process 35, 9 (2023), Article e2531.","journal-title":"Journal of Software: Evolution and Process"},{"key":"e_1_3_3_86_2","doi-asserted-by":"crossref","first-page":"886","DOI":"10.1145\/3180155.3180260","volume-title":"Proceedings of the 40th International Conference on Software Engineering (ICSE \u201918)","author":"Zhang Tianyi","year":"2018","unstructured":"Tianyi Zhang, Ganesha Upadhyaya, Anastasia Reinhardt, Hridesh Rajan, and Miryung Kim. 2018. Are code examples on an online Q & A forum reliable? A Study of API misuse on stack overflow. In Proceedings of the 40th International Conference on Software Engineering (ICSE \u201918). ACM, New York, NY, 886\u2013896. DOI: 10.1145\/3180155.3180260"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678172","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3678172","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:54:08Z","timestamp":1750287248000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678172"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,19]]},"references-count":82,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2024,11,30]]}},"alternative-id":["10.1145\/3678172"],"URL":"https:\/\/doi.org\/10.1145\/3678172","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,19]]},"assertion":[{"value":"2024-02-21","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-08","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-19","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}