{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T15:56:19Z","timestamp":1769010979094,"version":"3.49.0"},"publisher-location":"Cham","reference-count":39,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031768200","type":"print"},{"value":"9783031768217","type":"electronic"}],"license":[{"start":{"date-parts":[[2024,12,17]],"date-time":"2024-12-17T00:00:00Z","timestamp":1734393600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,12,17]],"date-time":"2024-12-17T00:00:00Z","timestamp":1734393600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Serious games have emerged as a powerful tool for achieving targeted outcomes beyond entertainment, such as learning or raising awareness about a topic. Considering their pervasiveness and wide adoption in the educational domain, traditional assessment approaches of these games have predominantly focused on their entertainment value and achievement of learning objectives. This paper proposes a comprehensive evaluation framework that goes beyond traditional dimensions to include aspects relevant to empathy raising and attitude change. The proposed framework has been validated through three user studies, assessing entertainment, historical awareness, empathy raising, and attitude change for three games, involving in total 98 high school students. Results from the studies are presented, as well as implications and lessons learned regarding the overall methodological approach, the evaluation instruments used, and the procedures followed.<\/jats:p>","DOI":"10.1007\/978-3-031-76821-7_11","type":"book-chapter","created":{"date-parts":[[2024,12,16]],"date-time":"2024-12-16T10:25:21Z","timestamp":1734344721000},"page":"141-164","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Serious Games Beyond Entertainment and Learning: An Evaluation Methodology for Assessing Awareness Raising, Empathy, and Social Change"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6270-8333","authenticated-orcid":false,"given":"Stavroula","family":"Ntoa","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-5364-3919","authenticated-orcid":false,"given":"Anastasia","family":"Ntagianta","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fernanda","family":"Flores","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2269-992X","authenticated-orcid":false,"given":"Luk\u00e1\u0161","family":"Kolek","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexandra","family":"Petrova","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4609-4079","authenticated-orcid":false,"given":"Konstantinos C.","family":"Apostolakis","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1144-8850","authenticated-orcid":false,"given":"Stefania","family":"Stamou","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9101-6301","authenticated-orcid":false,"given":"George","family":"Margetis","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3687-4220","authenticated-orcid":false,"given":"Constantine","family":"Stephanidis","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2024,12,17]]},"reference":[{"key":"11_CR1","unstructured":"Susi, T., Johannesson, M., Backlund, P.: Serious Games: An Overview. Institutionen f\u00f6r kommunikation och information (2007)"},{"key":"11_CR2","doi-asserted-by":"publisher","first-page":"661","DOI":"10.1016\/j.compedu.2012.03.004","volume":"59","author":"TM Connolly","year":"2012","unstructured":"Connolly, T.M., Boyle, E.A., MacArthur, E., Hainey, T., Boyle, J.M.: A systematic literature review of empirical evidence on computer games and serious games. Comput. Educ. 59, 661\u2013686 (2012). https:\/\/doi.org\/10.1016\/j.compedu.2012.03.004","journal-title":"Comput. Educ."},{"key":"11_CR3","doi-asserted-by":"publisher","first-page":"1789","DOI":"10.3390\/su12051789","volume":"12","author":"T-S Neset","year":"2020","unstructured":"Neset, T.-S., Andersson, L., Uhrqvist, O., Navarra, C.: Serious gaming for climate adaptation\u2014assessing the potential and challenges of a digital serious game for urban climate adaptation. Sustainability 12, 1789 (2020). https:\/\/doi.org\/10.3390\/su12051789","journal-title":"Sustainability"},{"key":"11_CR4","unstructured":"Tegos, S., Kioumousidou, M., Vavami, M., Kioutsiouki, D., Gkoumas, D., Tsiatsos, T.: HistoryLand: a serious game for history learning in primary education. In: European Conference on Games Based Learning, p. 838. Academic Conferences International Limited (2014)"},{"key":"11_CR5","doi-asserted-by":"crossref","unstructured":"Selekos, P., et al.: A video game about gulag archaeology and the memoirs of women prisoners. In: 2024 IEEE Gaming, Entertainment, and Media Conference (GEM), Turin, Italy, pp. 1\u20136. IEEE (2024)","DOI":"10.1109\/GEM61861.2024.10585492"},{"key":"11_CR6","doi-asserted-by":"publisher","unstructured":"Papadaki, E., Ntoa, S., Adami, I., Stephanidis, C.: Let\u2019s cook: an augmented reality system towards developing cooking skills for children with cognitive impairments. In: Guidi, B., Ricci, L., Calafate, C., Gaggi, O., Marquez-Barja, J. (eds.) GOODTECHS 2017. LNICST, vol. 233, pp. 237\u2013247. Springer, Cham (2018). https:\/\/doi.org\/10.1007\/978-3-319-76111-4_24","DOI":"10.1007\/978-3-319-76111-4_24"},{"key":"11_CR7","doi-asserted-by":"publisher","DOI":"10.1007\/s40692-019-00134-8","author":"K Stefanidis","year":"2019","unstructured":"Stefanidis, K., Psaltis, A., Apostolakis, K.C., Dimitropoulos, K., Daras, P.: Learning prosocial skills through multiadaptive games: a case study. J. Comput. Educ. (2019). https:\/\/doi.org\/10.1007\/s40692-019-00134-8","journal-title":"J. Comput. Educ."},{"key":"11_CR8","doi-asserted-by":"crossref","unstructured":"Xhako, A., et al.: Gamified experiences using 360\u00b0 photography: a methodology for creating gamified learning experiences in 360\u00b0 virtual environments. In: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, pp. 53\u201361. Association for Computing Machinery, New York (2023)","DOI":"10.1145\/3594806.3594858"},{"key":"11_CR9","doi-asserted-by":"publisher","unstructured":"Partarakis, N., et al.: Enhancing the educational value of tangible and intangible dimensions of traditional crafts through role-play gaming. In: Brooks, A., Brooks, E.I., Jonathan, D. (eds.) ArtsIT 2020. LNICST, vol. 367, pp. 243\u2013254. Springer, Cham (2021). https:\/\/doi.org\/10.1007\/978-3-030-73426-8_14","DOI":"10.1007\/978-3-030-73426-8_14"},{"key":"11_CR10","doi-asserted-by":"publisher","unstructured":"Stamou, S., Apostolakis, K.C., Ntoa, S., Margetis, G., Stephanidis, C.: Museum-inspired video games as a symbolic transitional justice policy: overview, concepts and research directions. ACM Games (2024). https:\/\/doi.org\/10.1145\/3651279","DOI":"10.1145\/3651279"},{"key":"11_CR11","doi-asserted-by":"publisher","unstructured":"Petri, G., Gresse von Wangenheim, C.: How games for computing education are evaluated? A systematic literature review. Comput. Educ. 107, 68\u201390 (2017). https:\/\/doi.org\/10.1016\/j.compedu.2017.01.004","DOI":"10.1016\/j.compedu.2017.01.004"},{"key":"11_CR12","unstructured":"Let players evaluate serious games. Design and validation of the Serious Games Evaluation Scale - IOS Press. https:\/\/content.iospress.com\/articles\/icga-journal\/icg190111"},{"key":"11_CR13","doi-asserted-by":"publisher","unstructured":"Ntoa, S.: Usability and user experience evaluation in intelligent environments: a review and reappraisal. Int. J. Hum. Comput. Interact., 1\u201330 (2024). https:\/\/doi.org\/10.1080\/10447318.2024.2394724","DOI":"10.1080\/10447318.2024.2394724"},{"key":"11_CR14","doi-asserted-by":"publisher","DOI":"10.1155\/2013\/136864","volume":"2013","author":"F Bellotti","year":"2013","unstructured":"Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., Berta, R.: Assessment in and of serious games: an overview. Adv. Hum. Comput. Interact. 2013, e136864 (2013). https:\/\/doi.org\/10.1155\/2013\/136864","journal-title":"Adv. Hum. Comput. Interact."},{"key":"11_CR15","doi-asserted-by":"publisher","first-page":"502","DOI":"10.1111\/bjet.12067","volume":"45","author":"I Mayer","year":"2014","unstructured":"Mayer, I., et al.: The research and evaluation of serious games: toward a comprehensive methodology. Br. J. Educ. Technol. 45, 502\u2013527 (2014). https:\/\/doi.org\/10.1111\/bjet.12067","journal-title":"Br. J. Educ. Technol."},{"key":"11_CR16","doi-asserted-by":"publisher","first-page":"396","DOI":"10.1016\/j.compedu.2015.07.011","volume":"87","author":"A Calder\u00f3n","year":"2015","unstructured":"Calder\u00f3n, A., Ruiz, M.: A systematic literature review on serious games evaluation: an application to software project management. Comput. Educ. 87, 396\u2013422 (2015). https:\/\/doi.org\/10.1016\/j.compedu.2015.07.011","journal-title":"Comput. Educ."},{"key":"11_CR17","unstructured":"Design and evaluation of a Serious Game for immersive cultural training | IEEE Conference Publication | IEEE Xplore. https:\/\/ieeexplore.ieee.org\/abstract\/document\/5665978"},{"key":"11_CR18","doi-asserted-by":"publisher","first-page":"502","DOI":"10.1111\/bjet.12067","volume":"45","author":"I Mayer","year":"2014","unstructured":"Mayer, I., et al.: The research and evaluation of serious games: toward a comprehensive methodology. Br. J. Edu. Technol. 45, 502\u2013527 (2014). https:\/\/doi.org\/10.1111\/bjet.12067","journal-title":"Br. J. Edu. Technol."},{"key":"11_CR19","unstructured":"Steiner, C., et al.: Evaluation of serious games: a holistic approach: ICERI2015. In: ICERI 2015 Proceedings, pp. 4334\u20134342 (2015)"},{"key":"11_CR20","doi-asserted-by":"publisher","first-page":"5755","DOI":"10.1007\/s11042-016-3845-9","volume":"76","author":"R Y\u00e1\u00f1ez-G\u00f3mez","year":"2017","unstructured":"Y\u00e1\u00f1ez-G\u00f3mez, R., Cascado-Caballero, D., Sevillano, J.-L.: Academic methods for usability evaluation of serious games: a systematic review. Multimedia Tools Appl. 76, 5755\u20135784 (2017). https:\/\/doi.org\/10.1007\/s11042-016-3845-9","journal-title":"Multimedia Tools Appl."},{"key":"11_CR21","doi-asserted-by":"crossref","unstructured":"Wilson, D.W., et al.: Serious games: an evaluation framework and case study. In: 2016 49th Hawaii International Conference on System Sciences (HICSS), pp. 638\u2013647 (2016)","DOI":"10.1109\/HICSS.2016.85"},{"key":"11_CR22","unstructured":"Serious games: Quality characteristics evaluation framework and case study | IEEE Conference Publication | IEEE Xplore. https:\/\/ieeexplore.ieee.org\/abstract\/document\/8340460"},{"key":"11_CR23","doi-asserted-by":"crossref","unstructured":"Mitgutsch, K., Alvarado, N.: Purposeful by design? A serious game design assessment framework. In: Proceedings of the International Conference on the Foundations of Digital Games, pp. 121\u2013128. Association for Computing Machinery, New York (2012)","DOI":"10.1145\/2282338.2282364"},{"key":"11_CR24","doi-asserted-by":"publisher","unstructured":"Emmerich, K., Bockholt, M.: Serious games evaluation: processes, models, and concepts. In: D\u00f6rner, R., G\u00f6bel, S., Kickmeier-Rust, M., Masuch, M., Zweig, K. (eds.) Entertainment Computing and Serious Games. LNCS, vol. 9970, pp. 265\u2013283. Springer, Cham (2016). https:\/\/doi.org\/10.1007\/978-3-319-46152-6_11","DOI":"10.1007\/978-3-319-46152-6_11"},{"key":"11_CR25","unstructured":"Guidelines for the Process of Cross-Cultural Adaptation of S...\u202f: Spine. https:\/\/journals.lww.com\/spinejournal\/fulltext\/2000\/12150\/Guidelines_for_the_Process_of_Cros_Cultural.14.aspx"},{"key":"11_CR26","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1016\/j.ijhcs.2005.06.002","volume":"64","author":"K Hornb\u00e6k","year":"2006","unstructured":"Hornb\u00e6k, K.: Current practice in measuring usability: challenges to usability studies and research. Int. J. Hum. Comput. Stud. 64, 79\u2013102 (2006). https:\/\/doi.org\/10.1016\/j.ijhcs.2005.06.002","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"11_CR27","unstructured":"Systematic review and validation of the game experience questionnaire (GEQ) - implications for citation and reporting practice. In: Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (2018). https:\/\/dl.acm.org\/doi\/abs\/10.1145\/3242671.3242683"},{"key":"11_CR28","first-page":"49","volume":"16","author":"J Keebler","year":"2020","unstructured":"Keebler, J., Shelstad, W., Smith, D., Chaparro, B., Phan, M.: Validation of the GUESS-18: a short version of the game user experience satisfaction scale (GUESS). J. Usability Stud. 16, 49\u201362 (2020)","journal-title":"J. Usability Stud."},{"key":"11_CR29","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1016\/j.adolescence.2005.08.010","volume":"29","author":"D Jolliffe","year":"2006","unstructured":"Jolliffe, D., Farrington, D.P.: Development and validation of the basic empathy scale. J. Adolesc. 29, 589\u2013611 (2006). https:\/\/doi.org\/10.1016\/j.adolescence.2005.08.010","journal-title":"J. Adolesc."},{"key":"11_CR30","doi-asserted-by":"publisher","first-page":"321","DOI":"10.1108\/JSOCM-11-2019-0204","volume":"10","author":"P Saleme","year":"2020","unstructured":"Saleme, P., Dietrich, T., Pang, B., Parkinson, J.: A gamified approach to promoting empathy in children. J. Soc. Mark. 10, 321\u2013337 (2020). https:\/\/doi.org\/10.1108\/JSOCM-11-2019-0204","journal-title":"J. Soc. Mark."},{"key":"11_CR31","unstructured":"Unraveling the processes underlying evaluation: attitudes from the perspective of the ape model. Soc. Cogn. https:\/\/guilfordjour-nals.com\/doi\/abs\/10.1521\/soco.2007.25.5.687"},{"key":"11_CR32","doi-asserted-by":"publisher","first-page":"692","DOI":"10.1037\/0033-2909.132.5.692","volume":"132","author":"B Gawronski","year":"2006","unstructured":"Gawronski, B., Bodenhausen, G.V.: Associative and propositional processes in evaluation: an integrative review of implicit and explicit attitude change. Psychol. Bull. 132, 692\u2013731 (2006). https:\/\/doi.org\/10.1037\/0033-2909.132.5.692","journal-title":"Psychol. Bull."},{"key":"11_CR33","doi-asserted-by":"crossref","unstructured":"Gawronski, B., Brannon, S.M.: What is cognitive consistency, and why does it matter? In: Cognitive Dissonance: Reexamining a Pivotal Theory in Psychology, 2nd ed, pp. 91\u2013116. American Psychological Association, Washington, DC (2019)","DOI":"10.1037\/0000135-005"},{"key":"11_CR34","doi-asserted-by":"publisher","first-page":"640","DOI":"10.1038\/s44159-022-00102-7","volume":"1","author":"M H\u00fctter","year":"2022","unstructured":"H\u00fctter, M.: An integrative review of dual- and single-process accounts of evaluative conditioning. Nat. Rev. Psychol. 1, 640\u2013653 (2022). https:\/\/doi.org\/10.1038\/s44159-022-00102-7","journal-title":"Nat. Rev. Psychol."},{"key":"11_CR35","unstructured":"Implicit and Explicit Evaluation: A Brief Review of the Associative\u2013Propositional Evaluation Model - Gawronski - 2014 - Social and Personality Psychology Compass - Wiley Online Library. https:\/\/compass.onlinelibrary.wiley.com\/doi\/full\/10.1111\/spc3.12124"},{"key":"11_CR36","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1037\/0022-3514.91.1.16","volume":"91","author":"A Karpinski","year":"2006","unstructured":"Karpinski, A., Steinman, R.B.: The single category implicit association test as a measure of implicit social cognition. J. Pers. Soc. Psychol. 91, 16\u201332 (2006). https:\/\/doi.org\/10.1037\/0022-3514.91.1.16","journal-title":"J. Pers. Soc. Psychol."},{"key":"11_CR37","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1037\/0033-295X.102.1.4","volume":"102","author":"AG Greenwald","year":"1995","unstructured":"Greenwald, A.G., Banaji, M.R.: Implicit social cognition: attitudes, self-esteem, and stereotypes. Psychol. Rev. 102, 4\u201327 (1995). https:\/\/doi.org\/10.1037\/0033-295X.102.1.4","journal-title":"Psychol. Rev."},{"key":"11_CR38","doi-asserted-by":"publisher","first-page":"1305","DOI":"10.1111\/bjso.12538","volume":"61","author":"PF Kotzur","year":"2022","unstructured":"Kotzur, P.F., Friehs, M.-T., Schmidt, P., Wagner, U., P\u00f6tzschke, S., Wei\u00df, B.: Attitudes towards refugees: introducing a short three-dimensional scale. Br. J. Soc. Psychol. 61, 1305\u20131331 (2022). https:\/\/doi.org\/10.1111\/bjso.12538","journal-title":"Br. J. Soc. Psychol."},{"key":"11_CR39","doi-asserted-by":"publisher","DOI":"10.1016\/j.cedpsych.2023.102225","volume":"75","author":"L Kolek","year":"2023","unstructured":"Kolek, L., Ropovik, I., \u0160isler, V., van Oostendorp, H., Brom, C.: Video games and attitude change: a meta-analysis. Contemp. Educ. Psychol. 75, 102225 (2023). https:\/\/doi.org\/10.1016\/j.cedpsych.2023.102225","journal-title":"Contemp. Educ. Psychol."}],"container-title":["Lecture Notes in Computer Science","HCI International 2024 \u2013 Late Breaking Papers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-76821-7_11","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,16]],"date-time":"2024-12-16T11:06:19Z","timestamp":1734347179000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-76821-7_11"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,17]]},"ISBN":["9783031768200","9783031768217"],"references-count":39,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-76821-7_11","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,17]]},"assertion":[{"value":"17 December 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"HCII","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Human-Computer Interaction","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Washington DC","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"USA","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2024","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"29 June 2024","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"4 July 2024","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"26","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"hcii2024","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/2024.hci.international\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}