{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,28]],"date-time":"2026-05-28T00:48:13Z","timestamp":1779929293641,"version":"3.53.1"},"reference-count":86,"publisher":"Association for Computing Machinery (ACM)","issue":"CHI PLAY","license":[{"start":{"date-parts":[[2024,10,14]],"date-time":"2024-10-14T00:00:00Z","timestamp":1728864000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2024,10,14]]},"abstract":"<jats:p>In gamification users show increased motivation and engagement towards tasks. So far, this effect has mostly been empirically tested based on the visual depiction of game elements, while research on the use and addition of auditory aspects is sparse. In this work we investigate the effect of different modalities of game elements (auditory, visual, audiovisual) on user experience, perception and performance in a gamified image-tagging task. We approached this via an online validation survey to find suitable sound effects (N=50), a main quantitative study (N=109) and a qualitative semi-structured interview (N=9). Our results show that while visual gamification increases performance, auditory and audiovisual gamification had no effect. However, they were shown to have an influence on the user's flow state. Our qualitative follow-up study shed light on underlying reasons and revealed each modality has its own drawbacks and advantages and that combining both visual and auditory aspects was preferred by participants.<\/jats:p>","DOI":"10.1145\/3677059","type":"journal-article","created":{"date-parts":[[2024,10,15]],"date-time":"2024-10-15T12:14:13Z","timestamp":1728994453000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Auditory, Visual, or Both? Comparing Visual and Auditory Representations of Game Elements in a Gamified Image-Tagging Task"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8132-2594","authenticated-orcid":false,"given":"Marc","family":"Schubhan","sequence":"first","affiliation":[{"name":"German Research Center for Artificial Intelligence (DFKI), Saarland Informatics Campus, Saarbr\u00fccken, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0279-5276","authenticated-orcid":false,"given":"Maximilian","family":"Altmeyer","sequence":"additional","affiliation":[{"name":"Saarland University of Applied Sciences (htw saar), Saarbr\u00fccken, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5958-3576","authenticated-orcid":false,"given":"Katja","family":"Rogers","sequence":"additional","affiliation":[{"name":"University of Amsterdam, Amsterdam, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1029-931X","authenticated-orcid":false,"given":"Donald","family":"Degraen","sequence":"additional","affiliation":[{"name":"University of Duisburg-Essen, Essen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5117-6928","authenticated-orcid":false,"given":"Pascal","family":"Lessel","sequence":"additional","affiliation":[{"name":"German Research Center for Artificial Intelligence (DFKI), Saarland Informatics Campus, Saarbr\u00fccken, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8055-8367","authenticated-orcid":false,"given":"Antonio","family":"Kr\u00fcger","sequence":"additional","affiliation":[{"name":"German Research Center for Artificial Intelligence (DFKI), Saarland Informatics Campus, Saarbr\u00fccken, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,10,15]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3520495.3520500"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517581"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3549491"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2016.08.049"},{"key":"e_1_2_2_5_1","volume-title":"Persuasive Messages: The Process of Influence","author":"Benoit William L","year":"2008","unstructured":"William L Benoit and Pamela J Benoit. 2008. Persuasive Messages: The Process of Influence.Blackwell Publishing."},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-89454-4_19"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327051hci0401_1"},{"key":"e_1_2_2_8_1","volume-title":"Memory for Auditory Icons and Earcons with Localization Cues","author":"Bonebright Terry L","unstructured":"Terry L Bonebright and Michael A Nees. 2007. Memory for Auditory Icons and Earcons with Localization Cues. Georgia Institute of Technology. https:\/\/www.researchgate.net\/profile\/Michael-Nees\/publication\/228817420_Memory_for_auditory_icons_and_earcons_with_localization_cues\/links\/0fcfd5100a69c95b41000000\/Memory-for-auditory-icons-and-earcons-with-localization-cues.pdf"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/0005--7916(94)90063--9"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_2_2_11_1","volume-title":"A detailed investigation into the effectiveness of earcons","author":"Brewster Stephen A","unstructured":"Stephen A Brewster, Peter C Wright, and Alastair DN Edwards. 1994. A detailed investigation into the effectiveness of earcons. In Santa Fe Institute Studies In The Sciences Of Complexity, Vol. 18. Addison-Wesley Publishing CO, 471--471."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2022.107362"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11135-021-01182-y"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1177\/102986490901300207"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467--9450.2010.00830.x"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3586182.3615787"},{"key":"e_1_2_2_17_1","unstructured":"Mihaly Csikszentmihalyi. 1975. Beyond Boredom and Anxiety. Jossey-Bass."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3464327.3464362"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2181037.2181040"},{"key":"e_1_2_2_20_1","first-page":"75","article-title":"Gamification in education: A systematic mapping study","volume":"18","author":"Dicheva Darina","year":"2015","unstructured":"Darina Dicheva, Christo Dichev, Gennady Agre, and Galia Angelova. 2015. Gamification in education: A systematic mapping study. Journal of educational technology & society, Vol. 18, 3 (2015), 75--88.","journal-title":"Journal of educational technology & society"},{"key":"e_1_2_2_21_1","unstructured":"Tilman Dingler Jeffrey Lindsay and Bruce N Walker. 2008. Learnabiltiy of sound cues for environmental features: Auditory icons earcons spearcons and speech. International Community for Auditory Display."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2769493.2769546"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/PacificVis.2018.00036"},{"key":"e_1_2_2_24_1","unstructured":"Inger Ekman. 2013. On the desire to not kill your players: Rethinking sound in pervasive and mixed reality games. In Foundations of Digital Games (FDG). 142--149."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1177\/160940690600500107"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/28189.1044809"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/169059.169184"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2014.377"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/cig.2019.8848105"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581087"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1460096.1460104"},{"key":"e_1_2_2_32_1","volume-title":"William Forde Thompson, and E Glenn Schellenberg","author":"Husain Gabriela","year":"2002","unstructured":"Gabriela Husain, William Forde Thompson, and E Glenn Schellenberg. 2002. Effects of musical tempo and mode on arousal, mood, and spatial abilities. Music perception, Vol. 20, 2 (2002), 151--171."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858515"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.invent.2016.10.002"},{"key":"e_1_2_2_35_1","volume-title":"Attention and effort","author":"Kahneman Daniel","unstructured":"Daniel Kahneman. 1973. Attention and effort. Vol. 1063. Citeseer."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2020.102495"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2018.10.013"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3316782.3316783"},{"key":"e_1_2_2_39_1","unstructured":"Gregory Kramer Bruce Walker Terri L. Bonebright Perry R. Cook John H. Flowers Nadine E. Miner John G. Neuhoff Robin Bargar Stephen Barrass Jonathan Berger Grigori E. Evreinov W. Tecumseh Fitch Matti T. Gr\u00f6hn Steve Handel Hans G. Kaper Haim Levkowitz Suresh K. Lodha Barbara G. Shinn-Cunningham Mary Simoni and Sever Tipei. 1999. Sonification Report: Status of the Field and Research Agenda Prepared for the National Science Foundation by members of the International Community for Auditory Display. https:\/\/api.semanticscholar.org\/CorpusID:2927531"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2015.08.008"},{"key":"e_1_2_2_41_1","volume-title":"Flow and its affective, cognitive, and performance-related consequences. Advances in flow research","author":"Landh\u00e4u\u00dfer Anne","year":"2012","unstructured":"Anne Landh\u00e4u\u00dfer and Johannes Keller. 2012. Flow and its affective, cognitive, and performance-related consequences. Advances in flow research (2012), 65--85."},{"key":"e_1_2_2_42_1","first-page":"13","article-title":"Making fictions sound real-On film sound, perceptual realism and genre. MedieKultur","volume":"26","author":"Langkj\u00e6r Birger","year":"2009","unstructured":"Birger Langkj\u00e6r. 2009. Making fictions sound real-On film sound, perceptual realism and genre. MedieKultur: Journal of media and communication research, Vol. 26, 48 (2009), 13--p.","journal-title":"Journal of media and communication research"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858463"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300380"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2793107.2810329"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1177\/1071181319631258"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380131"},{"key":"e_1_2_2_48_1","unstructured":"Mats Liljedahl. 2010. Sound for Fantasy and Freedom."},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/0030--5073(68)90004--4"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.57.9.705"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1080\/02701367.1989.10607413"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2583008.2583017"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2015.08.048"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-013-0357--7"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2016.11.062"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.4018\/978--1--61692--828--5.ch013"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.21785\/icad2017.008"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1037\/t06855-000"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0022359"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13104-021-05685--7"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1017\/s0954579405050340"},{"key":"e_1_2_2_62_1","volume-title":"CHI'13 Extended Abstracts on Human Factors in Computing Systems","author":"Robinson David","year":"2013","unstructured":"David Robinson and Victoria Belotti. 2013. A Preliminary Taxonomy of Gamification Elements for Varying Anticipated Commitment. CHI'13 Extended Abstracts on Human Factors in Computing Systems (2013)."},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3311350.3347158"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3356590.3356599"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432402"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022--3514.43.3.450"},{"key":"e_1_2_2_67_1","volume-title":"Ryan and Edward L Deci","author":"Richard","year":"2017","unstructured":"Richard M. Ryan and Edward L Deci. 2017. Self-Determination Theory: Basic Psychological Needs in Motivation, Development, and Wellness. The Guilford Press, New York."},{"key":"e_1_2_2_68_1","volume-title":"Predictions and the brain: How musical sounds become rewarding. Trends in cognitive sciences","author":"Salimpoor Valorie N","year":"2015","unstructured":"Valorie N Salimpoor, David H Zald, Robert J Zatorre, Alain Dagher, and Anthony Randal McIntosh. 2015. Predictions and the brain: How musical sounds become rewarding. Trends in cognitive sciences, Vol. 19, 2 (2015), 86--91."},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376360"},{"key":"e_1_2_2_70_1","volume-title":"realism, and the other in recent video games. Space and culture","author":"Schwartz Leigh","year":"2006","unstructured":"Leigh Schwartz. 2006. Fantasy, realism, and the other in recent video games. Space and culture, Vol. 9, 3 (2006), 313--325."},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2014.09.006"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376843"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03193770"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.4018\/jgcms.2010070101"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.4018\/jgcms.2010070101"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1007\/978--3--319--97934--2"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/2967934.2968082"},{"key":"e_1_2_2_78_1","unstructured":"John W Tukey et al. 1977. Exploratory data analysis. Vol. 2. Reading MA."},{"key":"e_1_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411170.3411250"},{"key":"e_1_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1007\/978--3--642--12439--6_6"},{"key":"e_1_2_2_81_1","volume-title":"Spearcons: Speech-based earcons improve navigation performance in auditory menus","author":"Walker Bruce N","year":"2006","unstructured":"Bruce N Walker, Amanda Nance, and Jeffrey Lindsay. 2006. Spearcons: Speech-based earcons improve navigation performance in auditory menus. Georgia Institute of Technology."},{"key":"e_1_2_2_82_1","volume-title":"European conference on games based learning","volume":"20","author":"Wang Alf Inge","year":"2016","unstructured":"Alf Inge Wang and Andreas Lieberoth. 2016. The effect of points and audio on concentration, engagement, enjoyment, learning, motivation, and classroom dynamics using Kahoot. In European conference on games based learning, Vol. 20. Academic Conferences International Limited."},{"key":"e_1_2_2_83_1","volume-title":"The structural characteristics of video games: A psycho-structural analysis. CyberPsychology & behavior","author":"Wood Richard TA","year":"2004","unstructured":"Richard TA Wood, Mark D Griffiths, Darren Chappell, and Mark NO Davies. 2004. The structural characteristics of video games: A psycho-structural analysis. CyberPsychology & behavior, Vol. 7, 1 (2004), 1--10."},{"key":"e_1_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2018.12.002"},{"key":"e_1_2_2_85_1","first-page":"65","article-title":"The effect of music on the performance and impression in a video racing game","volume":"7","author":"Yamada Masashi","year":"2001","unstructured":"Masashi Yamada. 2001. The effect of music on the performance and impression in a video racing game. Journal of Music Perception and Cognition, Vol. 7 (2001), 65--76.","journal-title":"Journal of Music Perception and Cognition"},{"key":"e_1_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41550-022-01721-zshowDOI"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3677059","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3677059","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T11:36:17Z","timestamp":1755776177000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3677059"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,14]]},"references-count":86,"journal-issue":{"issue":"CHI PLAY","published-print":{"date-parts":[[2024,10,14]]}},"alternative-id":["10.1145\/3677059"],"URL":"https:\/\/doi.org\/10.1145\/3677059","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,14]]},"assertion":[{"value":"2024-10-15","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}