{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T02:33:36Z","timestamp":1784342016519,"version":"3.55.0"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,1,18]],"date-time":"2025-01-18T00:00:00Z","timestamp":1737158400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,2,28]]},"abstract":"<jats:p>Modern video games are extremely complex software systems and, as such, they might suffer from several types of post-release issues. A particularly insidious issue is constituted by drops in the frame rate (i.e., stuttering events), which might have a negative impact on the user experience. Stuttering events are frequently documented in the million of hours of gameplay videos shared by players on platforms such as Twitch or YouTube. From the developers\u2019 perspective, these videos represent a free source of documented \u201ctesting activities.\u201d However, especially for popular games, the quantity and length of these videos make impractical their manual inspection. We introduce HASTE, an approach for the automatic detection of stuttering events in gameplay videos that can be exploited to generate candidate bug reports. HASTE firstly splits a given video into visually coherent slices, with the goal of filtering-out those that not representing actual gameplay (e.g., navigating the game settings). Then, it identifies the subset of pixels in the video frames which actually show the game in action excluding additional elements on screen such as the logo of the YouTube channel, on-screen chats, and so forth. In this way, HASTE can exploit state-of-the-art image similarity metrics to identify candidate stuttering events, namely subsequent frames being almost identical in the pixels depicting the game. We evaluate the different steps behind HASTE on a total of 105 videos showing that it can correctly extract video slices with a 76% precision, and can correctly identify the slices related to gameplay with a recall and precision higher than 77%. Overall, HASTE achieves 71% recall and 89% precision for the identification of stuttering events in gameplay videos.<\/jats:p>","DOI":"10.1145\/3695992","type":"journal-article","created":{"date-parts":[[2024,9,18]],"date-time":"2024-09-18T15:06:48Z","timestamp":1726672008000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Automatic Identification of Game Stuttering via Gameplay Videos Analysis"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5443-1303","authenticated-orcid":false,"given":"Emanuela","family":"Guglielmi","sequence":"first","affiliation":[{"name":"DEVISER, University of Molise, Pesche, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2216-3148","authenticated-orcid":false,"given":"Gabriele","family":"Bavota","sequence":"additional","affiliation":[{"name":"SEART, Universit\u00e0 della Svizzera italiana, Lugano, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7995-8582","authenticated-orcid":false,"given":"Rocco","family":"Oliveto","sequence":"additional","affiliation":[{"name":"DEVISER, University of Molise, Pesche, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1764-9685","authenticated-orcid":false,"given":"Simone","family":"Scalabrino","sequence":"additional","affiliation":[{"name":"DEVISER, University of Molise, Pesche, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,1,18]]},"reference":[{"key":"e_1_3_1_2_2","first-page":"1","volume-title":"Proceedings of the IEEE Games, Multimedia, Animation and Multiple Realities Conference (GMAX \u201920)","author":"Ahumada Tom\u00e1s","year":"2020","unstructured":"Tom\u00e1s Ahumada and Alexandre Bergel. 2020. Reproducing bugs in video games using genetic algorithms. In Proceedings of the IEEE Games, Multimedia, Animation and Multiple Realities Conference (GMAX \u201920), IEEE, 1\u20136."},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","unstructured":"Emanuela Guglielmi Gabriele Bavota Rocco Oliveto and Simone Scalabrino. 2024. Replication Package of \u201cAutomatic Identification of Game Stuttering via Gameplay Videos Analysis\u201d. Retrieved from https:\/\/doi.org\/10.6084\/m9.figshare.20714566","DOI":"10.6084\/m9.figshare.20714566"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/TG.2019.2947597"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-015-9417-1"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380328"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780203771587"},{"key":"e_1_3_1_10_2","first-page":"957","volume-title":"Proceedings of the IEEE\/ACM 43rd International Conference on Software Engineering (ICSE \u201921)","author":"Cooper Nathan","year":"2021","unstructured":"Nathan Cooper, Carlos Bernal-C\u00e1rdenas, Oscar Chaparro, Kevin Moran, and Denys Poshyvanyk. 2021. It takes two to tango: Combining visual and textual information for detecting duplicate video-based bug reports. In Proceedings of the IEEE\/ACM 43rd International Conference on Software Engineering (ICSE \u201921). IEEE, 957\u2013969."},{"key":"e_1_3_1_11_2","doi-asserted-by":"crossref","first-page":"1045","DOI":"10.1145\/3510003.3510048","volume-title":"Proceedings of the 44th International Conference on Software Engineering","author":"Feng Sidong","year":"2022","unstructured":"Sidong Feng and Chunyang Chen. 2022. Gifdroid: Automated replay of visual bug reports for android apps. In Proceedings of the 44th International Conference on Software Engineering, 1045\u20131057."},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00197"},{"key":"e_1_3_1_13_2","unstructured":"Google Forms. 2023. Retrieved from https:\/\/www.google.com\/forms\/about\/"},{"issue":"6","key":"e_1_3_1_14_2","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1007\/s10664-023-10365-0","article-title":"Using gameplay videos for detecting issues in video games","volume":"28","author":"Guglielmi Emanuela","year":"2023","unstructured":"Emanuela Guglielmi, Simone Scalabrino, Gabriele Bavota, and Rocco Oliveto. 2023. Using gameplay videos for detecting issues in video games. Empirical Software Engineering 28, 6 (2023), 136.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236055"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.5555\/3351736.3351787"},{"key":"e_1_3_1_17_2","first-page":"1","volume-title":"Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services","author":"Krieter Philipp","year":"2018","unstructured":"Philipp Krieter and Andreas Breiter. 2018. Analyzing mobile application usage: generating log files from mobile screen recordings. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services, 1\u201310."},{"key":"e_1_3_1_18_2","doi-asserted-by":"crossref","first-page":"511","DOI":"10.1145\/3524610.3527882","volume-title":"Proceedings of the 30th IEEE\/ACM International Conference on Program Comprehension","author":"Kuramoto Hiroki","year":"2022","unstructured":"Hiroki Kuramoto, Masanari Kondo, Yutaro Kashiwa, Yuta Ishimoto, Kaze Shindo, Yasutaka Kamei, and Naoyasu Ubayashi. 2022. Do visual issue reports help developers fix bugs? A preliminary study of using videos and images to report issues on github. In Proceedings of the 30th IEEE\/ACM International Conference on Program Comprehension, 511\u2013515."},{"key":"e_1_3_1_19_2","first-page":"108","volume-title":"Proceedings of the 5h International Conference on the Foundations of Digital Games","author":"Lewis Chris","year":"2010","unstructured":"Chris Lewis, Jim Whitehead, and Noah Wardrip-Fruin. 2010. What went wrong: A taxonomy of video game bugs. In Proceedings of the 5h International Conference on the Foundations of Digital Games, 108\u2013115."},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09733-6"},{"key":"e_1_3_1_21_2","first-page":"104","volume-title":"Proceedings of the IEEE 23rd International Conference on Program Comprehension","author":"MacLeod Laura","year":"2015","unstructured":"Laura MacLeod, Margaret-Anne Storey, and Andreas Bergen. 2015. Code, camera, action: How software developers document and share program knowledge using YouTube. In Proceedings of the IEEE 23rd International Conference on Program Comprehension. IEEE, 104\u2013114."},{"key":"e_1_3_1_22_2","unstructured":"Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. Retrieved from https:\/\/people.engr.tamu.edu\/guni\/csce642\/files\/dqn.pdf"},{"key":"e_1_3_1_23_2","first-page":"33","volume-title":"Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST \u201916)","author":"Moran Kevin","year":"2016","unstructured":"Kevin Moran, Mario Linares-V\u00e1squez, Carlos Bernal-C\u00e1rdenas, Christopher Vendome, and Denys Poshyvanyk. 2016. Automatically discovering, reporting and reproducing android application crashes. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST \u201916). IEEE, 33\u201344."},{"key":"e_1_3_1_24_2","unstructured":"OpenCV. 2023. Retrieved https:\/\/opencv.org"},{"key":"e_1_3_1_25_2","volume-title":"Automated Game Testing with ICARUS: Intelligent Completion of Adventure Riddles via Unsupervised Solving","author":"Pfau Johannes","year":"2017","unstructured":"Johannes Pfau, Jan David Smeddinck, and Rainer Malaka. 2017. Automated Game Testing with ICARUS: Intelligent Completion of Adventure Riddles via Unsupervised Solving. Association for Computing Machinery, New York, NY."},{"key":"e_1_3_1_26_2","first-page":"90","volume-title":"Proceedings of the IEEE\/ACM International Conference on Automation of Software Test (AST \u201921)","author":"Politowski Cristiano","year":"2021","unstructured":"Cristiano Politowski, Fabio Petrillo, and Yann-Ga\u00ebl Gu\u00e9h\u00e9neuc. 2021. A survey of video game testing. In Proceedings of the IEEE\/ACM International Conference on Automation of Software Test (AST \u201921). IEEE, 90\u201399."},{"key":"e_1_3_1_27_2","first-page":"645","volume-title":"Proceedings of the IEEE\/ACM 38th International Conference on Software Engineering Companion (ICSE-C \u201916)","author":"Ponzanelli Luca","year":"2016","unstructured":"Luca Ponzanelli, Gabriele Bavota, Andrea Mocci, Massimiliano Di Penta, Rocco Oliveto, Barbara Russo, Sonia Haiduc, and Michele Lanza. 2016. Codetube: extracting relevant fragments from software development video tutorials. In Proceedings of the IEEE\/ACM 38th International Conference on Software Engineering Companion (ICSE-C \u201916). IEEE, 645\u2013648."},{"key":"e_1_3_1_28_2","unstructured":"Prolific. 2023. Retrieved from https:\/\/www.prolific.co\/"},{"key":"e_1_3_1_29_2","unstructured":"PyTube. 2023. Retrieved from https:\/\/github.com\/pytube\/pytube"},{"key":"e_1_3_1_30_2","volume-title":"Fundamentals of Biostatistics","author":"Rosner B.","year":"2011","unstructured":"B. Rosner. 2011. Fundamentals of Biostatistics (7th. ed.). Brooks\/Cole, Boston, MA.","edition":"7"},{"key":"e_1_3_1_31_2","unstructured":"Satista. 2022. Retrieved from https:\/\/www.statista.com\/topics\/868\/video-games"},{"key":"e_1_3_1_32_2","unstructured":"Scikit-image. 2023. Retrieved from https:\/\/scikit-image.org"},{"key":"e_1_3_1_33_2","volume-title":"Integrated Spatial and Feature Image Systems: Retrieval, Analysis and Compression","author":"Smith John Richard","year":"1997","unstructured":"John Richard Smith. 1997. Integrated Spatial and Feature Image Systems: Retrieval, Analysis and Compression. Columbia University."},{"key":"e_1_3_1_34_2","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1145\/3524842.3528438","volume-title":"Proceedings of the 19th International Conference on Mining Software Repositories","author":"Taesiri Mohammad Reza","year":"2022","unstructured":"Mohammad Reza Taesiri, Finlay Macklon, and Cor-Paul Bezemer. 2022. CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning. In Proceedings of the 19th International Conference on Mining Software Repositories, 270\u2013281."},{"key":"e_1_3_1_35_2","unstructured":"The Last of Us. 2015. Retrieved from https:\/\/youtu.be\/yH5MgEbBOps?t=3494"},{"key":"e_1_3_1_36_2","first-page":"736","volume-title":"Proceedings of the IEEE\/ACM 43rd International Conference on Software Engineering (ICSE \u201921)","author":"Truelove Andrew","year":"2021","unstructured":"Andrew Truelove, Eduardo Santana de Almeida, and Iftekhar Ahmed. 2021. We\u2019ll fix it in post: What do bug fixes in video game update notes tell us? In Proceedings of the IEEE\/ACM 43rd International Conference on Software Engineering (ICSE \u201921). IEEE, 736\u2013747."},{"key":"e_1_3_1_37_2","unstructured":"Andrew Truelove Shiyue Rong Eduardo Santana de Almeida and Iftekhar Ahmed. 2023. Finding the needle in a haystack: Detecting bug occurrences in gameplay videos. arXiv:2311.10926. Retrieved from https:\/\/arxiv.org\/pdf\/ 2311.10926"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510625"},{"key":"e_1_3_1_39_2","unstructured":"Twitch. 2011. Retrieved from https:\/\/www.twitch.tv\/"},{"key":"e_1_3_1_40_2","unstructured":"Twitch-dl. 2020. Retrieved from https:\/\/github.com\/ihabunek\/twitch-dl\/"},{"key":"e_1_3_1_41_2","unstructured":"Twitch Stream Time. 2022. Retrieved from https:\/\/twitchtracker.com\/statistics\/stream-time"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2003.819861"},{"key":"e_1_3_1_43_2","unstructured":"WEKA [n.d.]. Retrieved from http:\/\/www.cs.waikato.ac.nz\/ml\/weka\/"},{"key":"e_1_3_1_44_2","unstructured":"What is Microstuttering and How Do I Fix It. 2022. Retrieved from https:\/\/www.pcgamer.com\/what-is-microstutter-and-how-do-i-fix-it\/"},{"key":"e_1_3_1_45_2","first-page":"456","volume-title":"Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement","author":"Wu Huijuan","year":"2009","unstructured":"Huijuan Wu, Yuepu Guo, and Carolyn B. Seaman. 2009. Analyzing video data: A study of programming behavior under two software engineering paradigms. In Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement. IEEE, 456\u2013459."},{"key":"e_1_3_1_46_2","first-page":"692","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME \u201920)","author":"Wu Yuechen","year":"2020","unstructured":"Yuechen Wu, Yingfeng Chen, Xiaofei Xie, Bing Yu, Changjie Fan, and Lei Ma. 2020. Regression testing of massively multiplayer online role-playing games. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME \u201920). IEEE, 692\u2013696."},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639163"},{"key":"e_1_3_1_48_2","unstructured":"YouTube. 2024. Retrieved from https:\/\/www.youtube.com"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","unstructured":"Yan Zheng Xiaofei Xie Ting Su Lei Ma Jianye Hao Zhaopeng Meng Yang Liu Ruimin Shen Yingfeng Chen and Changjie Fan. 2019. Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. DOI: 10.1109\/ASE.2019.00077","DOI":"10.1109\/ASE.2019.00077"},{"key":"e_1_3_1_50_2","doi-asserted-by":"crossref","unstructured":"Georg Zoeller. 2013. Game development telemetry in production. In Game Analytics: Maximizing the Value of Player Data. M. Seif El-Nasr A. Drachen and A. Canossa (Eds.) Springer 111\u2013135.","DOI":"10.1007\/978-1-4471-4769-5_7"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3695992","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3695992","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:30Z","timestamp":1750291470000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3695992"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,18]]},"references-count":49,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,2,28]]}},"alternative-id":["10.1145\/3695992"],"URL":"https:\/\/doi.org\/10.1145\/3695992","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,18]]},"assertion":[{"value":"2023-12-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-18","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}