{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,12]],"date-time":"2026-11-12T13:42:31Z","timestamp":1794490951960,"version":"4.9.1"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"CHI PLAY","license":[{"start":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T00:00:00Z","timestamp":1695945600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2023,9,29]]},"abstract":"<jats:p>Heuristics present a cheap and effective way of evaluating usability. However, in video games, evaluating unique player experiences that are dependent on individual preferences and abilities presents a challenge that goes beyond usability. Video games are more than just functional software, so games heuristics have been adapted to help examine functionality and experience. This paper reports on how papers published in the ACM Digital Library between 2012 and 2022 develop and apply heuristics in video games research. We found that heuristics are often used outside their intended purpose of being used in an expert evaluation. Instead, they are used as survey instruments, interview guides, codes for thematic analysis, and as design guidelines. This research contributes to HCI and video games research by distinguishing the terms design guidelines and design principles from heuristics. We make recommendations for researchers around developing heuristics and conducting video game heuristic evaluations. We propose a method for operationalising heuristics and make recommendations for the implementation of heuristics to improve the quality of video game heuristic reviews.<\/jats:p>","DOI":"10.1145\/3611035","type":"journal-article","created":{"date-parts":[[2023,10,4]],"date-time":"2023-10-04T15:28:28Z","timestamp":1696433308000},"page":"402-424","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["A Scoping Review of Heuristics in Videos Games Research: Definitions, Development, Application, and Operationalisation"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3905-288X","authenticated-orcid":false,"given":"Anne","family":"Ozdowska","sequence":"first","affiliation":[{"name":"Australian National University, Canberra, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6543-557X","authenticated-orcid":false,"given":"Penny","family":"Sweetser","sequence":"additional","affiliation":[{"name":"Australian National University, Canberra, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1955-2062","authenticated-orcid":false,"given":"Mahsuum","family":"Daiiani","sequence":"additional","affiliation":[{"name":"Australian National University, Belconnen, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,10,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3170427.3188580"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2535597.2535613"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2593968.2593976"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3335595.3335626"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3136907.3136940"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2890602.2890626"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645791.2645843"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/985921.986102"},{"key":"e_1_2_1_9_1","first-page":"557","volume-title":"July 2009 ( 2009 )","author":"Desurvire Heather","unstructured":"Heather Desurvire and Charlotte Wiberg . 2009. Game usability heuristics (PLAY) for evaluating and designing better games: The next iteration. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5621 LNCS , July 2009 ( 2009 ) , 557 - 566 . https:\/\/doi.org\/10.1007\/978-3-642-02774-1{_} 60 10.1007\/978-3-642-02774-1 Heather Desurvire and Charlotte Wiberg. 2009. Game usability heuristics (PLAY) for evaluating and designing better games: The next iteration. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5621 LNCS, July 2009 ( 2009 ), 557-566. https:\/\/doi.org\/10.1007\/978-3-642-02774-1{_} 60"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84882-963-3"},{"key":"#cr-split#-e_1_2_1_11_1.1","doi-asserted-by":"crossref","unstructured":"Heather Desurvire and Charlotte Wiberg. 2015. User Experience Design for Inexperienced Gamers: GAP-Game Approachability Principles. ( 2015 ) 169-186. https:\/\/doi.org\/10.1007\/978-3-319-15985-0{_} 8 10.1007\/978-3-319-15985-0","DOI":"10.1007\/978-3-319-15985-0_8"},{"key":"#cr-split#-e_1_2_1_11_1.2","doi-asserted-by":"crossref","unstructured":"Heather Desurvire and Charlotte Wiberg. 2015. User Experience Design for Inexperienced Gamers: GAP-Game Approachability Principles. ( 2015 ) 169-186. https:\/\/doi.org\/10.1007\/978-3-319-15985-0{_} 8","DOI":"10.1007\/978-3-319-15985-0_8"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/CISE.2009.5363526"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.4324\/9781003011293-1-3-4"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2008.07.004"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3364138.3364172"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2793107.2810309"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2910896.2910897"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851613.2851909"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1002\/pra2.2016.14505301107"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1201\/9781003109389-3"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474697"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474683"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3325480.3326574"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1236224.1236238"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3429360.3468204"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2468356.2468548"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1152215.1152218"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2513506.2513530"},{"key":"e_1_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Rolf Molich and Jakob Nielsen. 1990. Improving a Human-Computer Dialogue. Commun. ACM 33 3 ( 1990 ) 338-348. \t\t\t\t  Rolf Molich and Jakob Nielsen. 1990. Improving a Human-Computer Dialogue. Commun. ACM 33 3 ( 1990 ) 338-348.","DOI":"10.1145\/77481.77486"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3469595.3469617"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/142750.142834"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/191666.191729"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/259963.260531"},{"key":"e_1_2_1_34_1","volume-title":"Heuristic Evaluation Of User Interfaces. In CHI 90 Proceedings. 240-256","author":"Nielsen Jukob","year":"1990","unstructured":"Jukob Nielsen and Rolf Molich . 1990 . Heuristic Evaluation Of User Interfaces. In CHI 90 Proceedings. 240-256 . Jukob Nielsen and Rolf Molich. 1990. Heuristic Evaluation Of User Interfaces. In CHI 90 Proceedings. 240-256."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3337722.3337753"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the International Academic Conference on the Future of Game Design and Technology ( 2010 ), 56-65","author":"Paavilainen Janne","year":"2010","unstructured":"Janne Paavilainen . 2010 . Critical review on video game evaluation heuristics . Proceedings of the International Academic Conference on the Future of Game Design and Technology ( 2010 ), 56-65 . http:\/\/dl.acm.org\/citation.cfm?id= 1920787 % 5Cnhttp:\/\/portal.acm.org\/citation.cfm?doid= 1920778. 1920787 Janne Paavilainen. 2010. Critical review on video game evaluation heuristics. Proceedings of the International Academic Conference on the Future of Game Design and Technology ( 2010 ), 56-65. http:\/\/dl.acm.org\/citation.cfm?id= 1920787 % 5Cnhttp:\/\/portal.acm.org\/citation.cfm?doid= 1920778. 1920787"},{"key":"e_1_2_1_37_1","volume-title":"SoPlay Heuristics for Design and Evaluation of Social Network Games. CHI 2013 Games User Research Workshop","author":"Paavilainen Janne","year":"2013","unstructured":"Janne Paavilainen . 2013 . SoPlay Heuristics for Design and Evaluation of Social Network Games. CHI 2013 Games User Research Workshop January 2013 ( 2013 ), 6. Janne Paavilainen. 2013. SoPlay Heuristics for Design and Evaluation of Social Network Games. CHI 2013 Games User Research Workshop January 2013 ( 2013 ), 6."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.n71"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3136907.3136934"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472538.3472573"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3315002.3332440"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357282"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1531674.1531700"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1515\/9781400828678-039"},{"key":"e_1_2_1_45_1","volume-title":"Conference: Proceedings of the International Conference on Advances in Computing, Communications and Informatics ( 2013 ), 244-247","author":"Ponnada Aditya","year":"2013","unstructured":"Aditya Ponnada and Ajaykumar Kannan . 2013 . Evaluation of mobile games using playability heuristics . Conference: Proceedings of the International Conference on Advances in Computing, Communications and Informatics ( 2013 ), 244-247 . https:\/\/doi.org\/10.1007\/978-3-319-02958-0{_} 25 10.1007\/978-3-319-02958-0 Aditya Ponnada and Ajaykumar Kannan. 2013. Evaluation of mobile games using playability heuristics. Conference: Proceedings of the International Conference on Advances in Computing, Communications and Informatics ( 2013 ), 244-247. https:\/\/doi.org\/10.1007\/978-3-319-02958-0{_} 25"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csi.2017.03.009"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csi.2018.03.002"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2534903.2534915"},{"key":"e_1_2_1_49_1","volume-title":"ACHI 2011 : The Fourth International Conference on Advances in Computer-Human Interactions ( 2011 ), 59-62","author":"Rusu Cristian","year":"2011","unstructured":"Cristian Rusu , Silvana Roncagliolo , Virginica Rusu , and Cesar Collazos . 2011 . A Methodology to Establish Usability Heuristics . ACHI 2011 : The Fourth International Conference on Advances in Computer-Human Interactions ( 2011 ), 59-62 . Cristian Rusu, Silvana Roncagliolo, Virginica Rusu, and Cesar Collazos. 2011. A Methodology to Establish Usability Heuristics. ACHI 2011 : The Fourth International Conference on Advances in Computer-Human Interactions ( 2011 ), 59-62."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/WSC.2015.7408233"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238249"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2307096.2307153"},{"key":"e_1_2_1_53_1","first-page":"204","volume-title":"PART 1 ( 2009 )","author":"Sim Gavin","unstructured":"Gavin Sim , Janet C. Read , and Gilbert Cockton . 2009. Evidence based design of heuristics for computer assisted assessment. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5726 LNCS , PART 1 ( 2009 ) , 204 - 216 . https:\/\/doi.org\/10.1007\/978-3-642-03655-2{_} 25 10.1007\/978-3-642-03655-2 Gavin Sim, Janet C. Read, and Gilbert Cockton. 2009. Evidence based design of heuristics for computer assisted assessment. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5726 LNCS, PART 1 ( 2009 ), 204-216. https:\/\/doi.org\/10.1007\/978-3-642-03655-2{_} 25"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2513002.2513005"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3441000.3441048"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3369457.3369486"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3373017.3373068"},{"key":"e_1_2_1_58_1","article-title":"Revisiting the GameFlow Model with Detailed Heuristics","volume":"2012","author":"Sweetser Penelope","year":"2012","unstructured":"Penelope Sweetser , Daniel Johnson , and Peta Wyeth . 2012 . Revisiting the GameFlow Model with Detailed Heuristics . Journal of Creative Technologies 2012 , 3 ( 2012 ), 1-16. http:\/\/journal.colab.org.nz\/article\/21 Penelope Sweetser, Daniel Johnson, and Peta Wyeth. 2012. Revisiting the GameFlow Model with Detailed Heuristics. Journal of Creative Technologies 2012, 3 ( 2012 ), 1-16. http:\/\/journal.colab.org.nz\/article\/21","journal-title":"Journal of Creative Technologies"},{"key":"e_1_2_1_59_1","volume-title":"The 8th Australasian Conference on Interactive Entertainment. The 8th Australasian Conference on Interactive Entertainment, Auckland, NZ.","author":"Sweetser Penny","unstructured":"Penny Sweetser , Daniel Johnson , Peta Wyeth , and Anne Ozdowska . 2012. GameFlow Heuristics for Designing and Evaluating Real-Time Strategy Games . In The 8th Australasian Conference on Interactive Entertainment. The 8th Australasian Conference on Interactive Entertainment, Auckland, NZ. Penny Sweetser, Daniel Johnson, Peta Wyeth, and Anne Ozdowska. 2012. GameFlow Heuristics for Designing and Evaluating Real-Time Strategy Games. In The 8th Australasian Conference on Interactive Entertainment. The 8th Australasian Conference on Interactive Entertainment, Auckland, NZ."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3034780"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3441000.3441050"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359996.3364800"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/1077246.1077253"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2367616.2367626"},{"key":"e_1_2_1_65_1","first-page":"363","volume-title":"E-Learn 2010-World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education","author":"Tan Jean Lee","year":"2010","unstructured":"Jean Lee Tan , Dion Hoe-Lian Goh , Rebecca P Ang , and Vivien S Huan . 2010 . Usability and Playability Heuristics for Evaluation of an Instructional Game . E-Learn 2010-World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education September 2016 ( 2010 ), 363 - 373 . http:\/\/www.learntechlib.org\/p\/35570 Jean Lee Tan, Dion Hoe-Lian Goh, Rebecca P Ang, and Vivien S Huan. 2010. Usability and Playability Heuristics for Evaluation of an Instructional Game. E-Learn 2010-World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education September 2016 ( 2010 ), 363-373. http:\/\/www.learntechlib.org\/p\/35570"},{"key":"e_1_2_1_66_1","first-page":"224","article-title":"Gameful Design Heuristics","author":"Tondello Gustavo F","year":"2019","unstructured":"Gustavo F Tondello , Dennis Kappen , Marim Ganaba , and Lennart E Nacke . 2019 . Gameful Design Heuristics : A Gamification Inspection Tool. In Proceedings of HCI International. 224 - 244 . Gustavo F Tondello, Dennis Kappen, Marim Ganaba, and Lennart E Nacke. 2019. Gameful Design Heuristics: A Gamification Inspection Tool. In Proceedings of HCI International. 224-244.","journal-title":"A Gamification Inspection Tool. In Proceedings of HCI International."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2968120.2987729"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.14236\/ewic\/hci2014.57"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3611035","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3611035","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:50:52Z","timestamp":1750287052000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3611035"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,29]]},"references-count":69,"journal-issue":{"issue":"CHI PLAY","published-print":{"date-parts":[[2023,9,29]]}},"alternative-id":["10.1145\/3611035"],"URL":"https:\/\/doi.org\/10.1145\/3611035","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,29]]},"assertion":[{"value":"2023-10-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}