{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T14:32:27Z","timestamp":1754145147720,"version":"3.41.2"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"ISSTA","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,22]]},"abstract":"<jats:p>Gamification is an emerging technique to enhance motivation and performance in traditionally unengaging tasks like software testing. Previous studies have indicated that gamified systems have the potential to improve software testing processes by providing testers with achievements and feedback. However, further evidence of these benefits across different environments, programming languages, and participant groups is required. This paper aims to replicate and validate the effects of IntelliGame, a gamification plugin for IntelliJ IDEA to engage developers in writing and executing tests. The objective is to generalize the benefits observed in earlier studies to new contexts, i.e., the TypeScript programming language and a larger participant pool. The replicability study consists of a controlled experiment with 174 participants, divided into two groups: one using IntelliGame and one with no gamification plugin. The study employed a two-group experimental design to compare testing behavior, coverage, mutation scores, and participant feedback between the groups. Data was collected through test metrics and participant surveys, and statistical analysis was performed to determine the statistical significance. Participants using IntelliGame showed higher engagement and productivity in testing practices than the control group, evidenced by the creation of more tests, increased frequency of executions, and enhanced utilization of testing tools. This ultimately led to better code implementations, highlighting the effectiveness of gamification in improving functional outcomes and motivating users in their testing endeavors. The replication study confirms that gamification, through IntelliGame, positively impacts software testing behavior and developer engagement in coding tasks. These findings suggest that integrating game elements into the testing environment can be an effective strategy to improve software testing practices.<\/jats:p>","DOI":"10.1145\/3728983","type":"journal-article","created":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T10:52:56Z","timestamp":1750589576000},"page":"2407-2429","source":"Crossref","is-referenced-by-count":0,"title":["Gamifying Testing in IntelliJ: A Replicability Study"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9265-5789","authenticated-orcid":false,"given":"Philipp","family":"Straubinger","sequence":"first","affiliation":[{"name":"University of Passau, Passau, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8765-6501","authenticated-orcid":false,"given":"Tommaso","family":"Fulcini","sequence":"additional","affiliation":[{"name":"Politecnico di Torino, Torino, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7254-9578","authenticated-orcid":false,"given":"Giacomo","family":"Garaccione","sequence":"additional","affiliation":[{"name":"Politecnico di Torino, Torino, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0501-7886","authenticated-orcid":false,"given":"Luca","family":"Ardito","sequence":"additional","affiliation":[{"name":"Politecnico di Torino, Torino, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4364-6595","authenticated-orcid":false,"given":"Gordon","family":"Fraser","sequence":"additional","affiliation":[{"name":"University of Passau, Passau, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,6,22]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/CHASE52884.2021.00020"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2024645.2024657"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111647"},{"key":"e_1_2_1_4_1","first-page":"35","article-title":"When to use Fisher\u2019s exact test. In American Society for Quality","volume":"2","author":"Bower Keith M","year":"2003","unstructured":"Keith M Bower. 2003. When to use Fisher\u2019s exact test. In American Society for Quality, Six Sigma Forum Magazine. 2, 35\u201337.","journal-title":"Six Sigma Forum Magazine."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW52544.2021.00032"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3068324"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3348036"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.5220\/0007711603070314"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3266003.3266007"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3364641.3364661"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110870"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2181037.2181040"},{"key":"e_1_2_1_13_1","unstructured":"Association for Computing Machinery. 24-08-2020. Artifact Review and Badging Version 1.1. https:\/\/www.acm.org\/publications\/policies\/artifact-review-and-badging-current [Accessed 28-10-2024]."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW55395.2022.00045"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3582273"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3643796.3648459"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3548771.3561408"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISESE.2005.1541818"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-25231-0_2"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472673.3473960"},{"key":"e_1_2_1_21_1","volume-title":"Storm: A model for sustainably onboarding software testers. arXiv preprint arXiv:2206.01020.","author":"Lorey Tobias","year":"2022","unstructured":"Tobias Lorey, Stefan Mohacsi, Armin Beer, and Michael Felderer. 2022. Storm: A model for sustainably onboarding software testers. arXiv preprint arXiv:2206.01020."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177730491"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/GAS59301.2023.00008"},{"volume-title":"Innovations in Computing Sciences and Software Engineering","author":"Paiva Edgy","key":"e_1_2_1_24_1","unstructured":"Edgy Paiva, Danielly Barbosa, Roberto Lima, and Adriano Albuquerque. 2010. Factors that Influence the Productivity of Software Developers in a Developer View. In Innovations in Computing Sciences and Software Engineering, Tarek Sobh and Khaled Elleithy (Eds.). Springer Netherlands, Dordrecht. 99\u2013104. isbn:978-90-481-9112-3"},{"volume-title":"Gamification-Based Cyber-Enabled Learning Environment of Software Testing. In 2016 ASEE Annual Conference & Exposition. ASEE Conferences","author":"Clarke Yujian","key":"e_1_2_1_25_1","unstructured":"Yujian Fu P.E. and Peter J. Clarke. 2016. Gamification-Based Cyber-Enabled Learning Environment of Software Testing. In 2016 ASEE Annual Conference & Exposition. ASEE Conferences, New Orleans, Louisiana. https:\/\/peer.asee.org\/27000."},{"key":"e_1_2_1_26_1","volume-title":"proceedings of the royal society of London, 58","author":"Pearson Karl","year":"1895","unstructured":"Karl Pearson. 1895. VII. Note on regression and inheritance in the case of two parents. proceedings of the royal society of London, 58, 347-352 (1895), 240\u2013242."},{"key":"e_1_2_1_27_1","unstructured":"Paul Ralph Sebastian Baltes Domenico Bianculli Yvonne Dittrich Michael Felderer Robert Feldt Antonio Filieri Carlo Alberto Furia Daniel Graziotin Pinjia He Rashina Hoda Natalia Juristo Barbara A. Kitchenham Romain Robbes Daniel M\u00e9ndez Jefferson Molleri Diomidis Spinellis Miroslaw Staron Klaas-Jan Stol Damian A. Tamburri Marco Torchiano Christoph Treude Burak Turhan and Sira Vegas. 2020. ACM SIGSOFT Empirical Standards. CoRR abs\/2010.03525 (2020) arxiv:2010.03525. arxiv:2010.03525"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/EURMIC.2003.1231572"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2016.43"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.01.006"},{"key":"e_1_2_1_31_1","first-page":"91","article-title":"A gameful approach to teaching software design and software testing","volume":"9","author":"Sheth Swapneel","year":"2015","unstructured":"Swapneel Sheth, Jonathan Bell, and Gail Kaiser. 2015. A gameful approach to teaching software design and software testing. Computer Games and Software Engineering, 9 (2015), 91.","journal-title":"Computer Games and Software Engineering"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10062-w"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE59848.2023.00075"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3678869.3685683"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3639474.3640054"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3623339"},{"key":"e_1_2_1_37_1","first-page":"251","article-title":"A research study on importance of testing and quality assurance in software development life cycle (SDLC) models","volume":"2","author":"Tuteja Maneela","year":"2012","unstructured":"Maneela Tuteja and Gaurav Dubey. 2012. A research study on importance of testing and quality assurance in software development life cycle (SDLC) models. International Journal of Soft Computing and Engineering (IJSCE), 2, 3 (2012), 251\u2013257.","journal-title":"International Journal of Soft Computing and Engineering (IJSCE)"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 2nd international conference on extreme programming and flexible processes in software engineering (XP2001)","author":"Deursen Arie Van","year":"2001","unstructured":"Arie Van Deursen, Leon Moonen, Alex Van Den Bergh, and Gerard Kok. 2001. Refactoring test code. In Proceedings of the 2nd international conference on extreme programming and flexible processes in software engineering (XP2001). 92\u201395."},{"volume-title":"Experimentation in software engineering. 236","author":"Wohlin Claes","key":"e_1_2_1_39_1","unstructured":"Claes Wohlin, Per Runeson, Martin H\u00f6st, Magnus C Ohlsson, Bj\u00f6rn Regnell, and Anders Wessl\u00e9n. 2012. Experimentation in software engineering. 236, Springer."},{"volume-title":"Educational Innovations and Gamification for Fostering Training and Testing in Software Implementation Projects","author":"Yordanova Zornitsa","key":"e_1_2_1_40_1","unstructured":"Zornitsa Yordanova. 2019. Educational Innovations and Gamification for Fostering Training and Testing in Software Implementation Projects. In Software Business, Sami Hyrynsalmi, Mari Suoranta, Anh Nguyen-Duc, Pasi Tyrv\u00e4inen, and Pekka Abrahamsson (Eds.). Springer International Publishing, Cham. 293\u2013305. isbn:978-3-030-33742-1"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728983","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,16]],"date-time":"2025-07-16T16:56:03Z","timestamp":1752684963000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728983"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,22]]},"references-count":40,"journal-issue":{"issue":"ISSTA","published-print":{"date-parts":[[2025,6,22]]}},"alternative-id":["10.1145\/3728983"],"URL":"https:\/\/doi.org\/10.1145\/3728983","relation":{},"ISSN":["2994-970X"],"issn-type":[{"type":"electronic","value":"2994-970X"}],"subject":[],"published":{"date-parts":[[2025,6,22]]}}}