{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,28]],"date-time":"2026-03-28T02:47:08Z","timestamp":1774666028003,"version":"3.50.1"},"reference-count":35,"publisher":"Emerald","issue":"2","license":[{"start":{"date-parts":[[2013,6,7]],"date-time":"2013-06-07T00:00:00Z","timestamp":1370563200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2013,6,7]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>This paper aims to briefly outline the seamless evaluation approach and its application during an evaluation of ORIENT, a serious game aimed at young adults.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>In this paper, the authors detail a unobtrusive, embedded evaluation approach that occurs within the game context, adding value and entertainment to the player experience whilst accumulating useful data for the development team.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>The key result from this study was that during the \u201cseamless evaluation\u201d approach, users were unaware that they had been participating in an evaluation, with instruments enhancing rather than detracting from the in\u2010role game experience.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Practical implications<\/jats:title><jats:p>This approach, seamless evaluation, was devised in response to player expectations, perspectives and requirements, recognising that in the evaluation of games the whole process of interaction including its evaluation must be enjoyable and fun for the user.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>Through using seamless evaluation, the authors created an evaluation completely embedded within the \u201cmagic circle\u201d of an in\u2010game experience that added value to the user experience whilst also yielding relevant results for the development team.<\/jats:p><\/jats:sec>","DOI":"10.1108\/itse-02-2013-0003","type":"journal-article","created":{"date-parts":[[2013,6,13]],"date-time":"2013-06-13T10:16:38Z","timestamp":1371118598000},"page":"130-146","source":"Crossref","is-referenced-by-count":7,"title":["Serious game evaluation as a meta\u2010game"],"prefix":"10.1108","volume":"10","author":[{"given":"Lynne","family":"Hall","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Susan Jane","family":"Jones","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ruth","family":"Aylett","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marc","family":"Hall","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sarah","family":"Tazzyman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ana","family":"Paiva","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lynne","family":"Humphries","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"key":"key2022021520003329800_b1","doi-asserted-by":"crossref","unstructured":"Ang, S., Van Dyne, L., Koh, C., Ng, K.Y., Templer, K.J., Tay, C. and Chandrasekar, N.A. (2007), \u201cCultural intelligence: its measurement and effects on cultural judgment and decision making, cultural adaptation and task performance\u201d, Management and Organisation Review, Vol. 3 No. 3, pp. 335\u2010371.","DOI":"10.1111\/j.1740-8784.2007.00082.x"},{"key":"key2022021520003329800_b2","doi-asserted-by":"crossref","unstructured":"Baranowski, T., Buday, R., Thompson, D.I. and Baranowski, J. (2008), \u201cPlaying for real video games and stories for health\u2010related behavior change\u201d, American Journal of Preventive Medicine, Vol. 34 No. 1, pp. 74\u201082.","DOI":"10.1016\/j.amepre.2007.09.027"},{"key":"key2022021520003329800_b3","doi-asserted-by":"crossref","unstructured":"Bellotti, F., Berta, R., De Gloria, A. and Zappi, V. (2008), \u201cExploring gaming mechanisms to enhance knowledge acquisition in virtual worlds\u201d, ACM DIMEA'08 Athens, Greece, pp. 77\u201084.","DOI":"10.1145\/1413634.1413653"},{"key":"key2022021520003329800_b4","doi-asserted-by":"crossref","unstructured":"Boyer, S. (2009), \u201cA virtual failure: evaluating the success of Nintendo's Virtual Boy\u201d, The Velvet Light Trap, No. 64, pp. 23\u201033.","DOI":"10.1353\/vlt.0.0039"},{"key":"key2022021520003329800_b5","unstructured":"Chambliss, D.F. and Schutt, R.K. (2009), Making Sense of the Social World: Methods of Investigation, 3rd ed., Pine Forge Press, Thousand Oaks, CA."},{"key":"key2022021520003329800_b6","doi-asserted-by":"crossref","unstructured":"Chen, P. and Zarki, M.E. (2011), \u201cPerceptual view inconsistency: an objective evaluation framework for online game quality of experience (QoE)\u201d, IEEE Proceedings of NETGAMES, pp. 1\u20106.","DOI":"10.1109\/NetGames.2011.6080978"},{"key":"key2022021520003329800_b7","doi-asserted-by":"crossref","unstructured":"De Freitas, S., Rebolledo\u2010Mendez, G., Liarokapis, F., Magoulas, G. and Poulovassilis, A. (2010), \u201cLearning as immersive experiences: using the four\u2010dimensional framework for designing and evaluating immersive learning experiences in a virtual world\u201d, British Journal of Educational Technology, Vol. 41 No. 1, pp. 69\u201085.","DOI":"10.1111\/j.1467-8535.2009.01024.x"},{"key":"key2022021520003329800_b8","doi-asserted-by":"crossref","unstructured":"Dias, J. and Paiva, A. (2005), \u201cFeeling and reasoning: a computational model\u201d, in Seabra Lopes, L., Lau, N., Mariano, P. and Rocha, L.M. (Eds), 12th Portuguese Conference on Artificial Intelligence, Springer, New York, NY, pp. 127\u2010140.","DOI":"10.1007\/11595014_13"},{"key":"key2022021520003329800_b9","doi-asserted-by":"crossref","unstructured":"Dickey, M.D. (2011), \u201cMurder on Grimm Isle: the impact of game narrative design in an educational game\u2010based learning environment\u201d, British Journal of Educational Technology, Vol. 42 No. 3, pp. 456\u2010469.","DOI":"10.1111\/j.1467-8535.2009.01032.x"},{"key":"key2022021520003329800_b10","unstructured":"D\u00f6rner, D. (2003), \u201cThe mathematics of emotion\u201d, in Detje, F. and Schaub, H. (Eds), The Logic of Cognitive Systems: Proceedings of the Fifth International Conference on Cognitive Modeling, Springer, New York, NY, pp. 127\u2010140."},{"key":"key2022021520003329800_b11","doi-asserted-by":"crossref","unstructured":"Gunter, G.A., Kenny, R.F. and Vick, E.H. (2008), \u201cTaking educational games seriously: using the RETAIN model to design endogenous fantasy into standalone educational games\u201d, Education Technology Research Development, Vol. 56, pp. 511\u2010537.","DOI":"10.1007\/s11423-007-9073-2"},{"key":"key2022021520003329800_b12","unstructured":"Hall, L., Woods, S. and Aylett, R. (2006), \u201cFearNot! Involving children in the design of a Virtual Learning Environment\u201d, Artificial Intelligence & Education, Vol. 16 No. 4, pp. 237\u2010251."},{"key":"key2022021520003329800_b13","doi-asserted-by":"crossref","unstructured":"Hertwig, R. and Ortmann, A. (2008a), \u201cDeception in experiments: revisiting the arguments in its defense\u201d, Ethics and Behaviour, Vol. 18 No. 1, pp. 59\u201092.","DOI":"10.1080\/10508420701712990"},{"key":"key2022021520003329800_b14","doi-asserted-by":"crossref","unstructured":"Hertwig, R. and Ortmann, A. (2008b), \u201cDeception in social psychological experiments: two misconceptions and a research agenda\u201d, Social Psychology Quarterly, Vol. 71 No. 3, pp. 222\u2010227.","DOI":"10.1177\/019027250807100304"},{"key":"key2022021520003329800_b15","doi-asserted-by":"crossref","unstructured":"Hey, J.D. (1998), \u201cExperimental economics and deception: a comment\u201d, Journal of Economic Psychology, Vol. 19, pp. 397\u2010401.","DOI":"10.1016\/S0167-4870(98)00013-0"},{"key":"key2022021520003329800_b16","unstructured":"Hofstede, G., Hofstede, G.J. and Minkov, M. (2010), Cultures and Organisations, Software of the Mind, Intercultural Cooperation and It's Importance for Survival, 3rd ed., McGraw\u2010Hill, New York, NY."},{"key":"key2022021520003329800_b17","doi-asserted-by":"crossref","unstructured":"Hong, J.C., Cheng, C.L., Hwang, M.Y., Lee, C.K. and Chang, H.Y. (2009), \u201cAssessing the educational value of digital games\u201d, Journal of Computer Assisted Learning, Vol. 25, pp. 423\u2010437.","DOI":"10.1111\/j.1365-2729.2009.00319.x"},{"key":"key2022021520003329800_b18","doi-asserted-by":"crossref","unstructured":"Howe, K. (2004), \u201cA critique of experimentalism\u201d, Qualitative Inquiry, Vol. 10 No. 1, pp. 42\u201061.","DOI":"10.1177\/1077800403259491"},{"key":"key2022021520003329800_b19","unstructured":"Huizinga, J. (1950), Homo Ludens: A Study of the Play\u2010Element in Culture, Roy Publishers, New York, NY."},{"key":"key2022021520003329800_b20","doi-asserted-by":"crossref","unstructured":"Johnson, L.F. and Levine, A.H. (2008), \u201cVirtual worlds: inherently immersive, highly social learning spaces\u201d, Theory into Practice, Vol. 47, pp. 161\u2010170.","DOI":"10.1080\/00405840801992397"},{"key":"key2022021520003329800_b21","doi-asserted-by":"crossref","unstructured":"Klesen, M. (2005), \u201cUsing theatrical concepts for role\u2010plays with educational agents\u201d, Applied Artificial Intelligence, Vol. 19 Nos 3\/4, pp. 413\u2010431.","DOI":"10.1080\/08839510590910237"},{"key":"key2022021520003329800_b22","doi-asserted-by":"crossref","unstructured":"Libin, A., Laouderdale, M., Millo, Y., Shamloo, C., Spencer, R., Green, B., Donnelan, J., Wellesley, C. and Groah, S. (2010), \u201cRole\u2010playing simulation as an educational tool for health care personnel: developing and embedded assessment framework\u201d, Cyberpsychology, Behaviour and Social Networking, Vol. 13 No. 2, pp. 217\u2010224.","DOI":"10.1089\/cyber.2009.0040"},{"key":"key2022021520003329800_b23","doi-asserted-by":"crossref","unstructured":"Liu, Z., Feng, Y. and Li, B. (2011), \u201cWhen multi\u2010touch meets streaming\u201d, ACM MUM'11 Beijing, China, pp. 23\u201032.","DOI":"10.1145\/2107596.2107599"},{"key":"key2022021520003329800_b24","doi-asserted-by":"crossref","unstructured":"Moore, L., Graham, A. and Diamond, I. (2003), \u201cOn the feasibility of conducting randomised trials in education: case study of a sex intervention\u201d, British Educational Research Journal, Vol. 29 No. 5, pp. 673\u2010689.","DOI":"10.1080\/0141192032000133712"},{"key":"key2022021520003329800_b25","doi-asserted-by":"crossref","unstructured":"Orb, A., Eisenhauer, L. and Wynaden, D. (2000), \u201cEthics in qualitative research\u201d, Profession and Society, Vol. 33 No. 1, pp. 93\u201096.","DOI":"10.1111\/j.1547-5069.2001.00093.x"},{"key":"key2022021520003329800_b26","doi-asserted-by":"crossref","unstructured":"Ortony, A., Clore, G.L. and Collins, A. (1988), The Cognitive Structure of Emotions, Cambridge University Press, Cambridge.","DOI":"10.1017\/CBO9780511571299"},{"key":"key2022021520003329800_b27","doi-asserted-by":"crossref","unstructured":"Pascual\u2010Leone, A., Singh, T. and Scoboria, A. (2010), \u201cUsing deception ethically: practical research guidelines for researchers and reviewers\u201d, Canadian Psychology, Vol. 51 No. 4, pp. 241\u2010248.","DOI":"10.1037\/a0021119"},{"key":"key2022021520003329800_b28","doi-asserted-by":"crossref","unstructured":"Rankin, Y.A., McNeal, M., Shute, M.W. and Gooch, B. (2008), \u201cUser centred games design: evaluating massive multiplayer online role playing games for second language acquisition\u201d, ACM Sandbox Symposium, Los Angeles, CA, pp. 43\u201049.","DOI":"10.1145\/1401843.1401851"},{"key":"key2022021520003329800_b29","unstructured":"Rojas\u2010Barahona, L.M., Lorenzo, A. and Gardent, C. (2012), \u201cAn end\u2010to\u2010end evaluation of two situated dialog systems\u201d, ACL 13th Annual SIGDIAL, Seoul, South Korea, pp. 10\u201019."},{"key":"key2022021520003329800_b30","unstructured":"Salen, K. and Zimmerman, E. (2003), Rules of Play: Game Design Fundamentals, MIT Press, Cambridge, MA."},{"key":"key2022021520003329800_b31","doi-asserted-by":"crossref","unstructured":"Sheng, S., Magnien, B., Kumaraguru, P., Acquisti, A., Cranor, L.F., Hong, J. and Nunge, E. (2007), \u201cAnti\u2010Phishing Phil: the design and evaluation of a game that teaches people not to fall for phishing\u201d, Privacy and Security, Vol. 229 No. 1, pp. 88\u201099.","DOI":"10.1145\/1280680.1280692"},{"key":"key2022021520003329800_b32","unstructured":"Stephan, W.G. (1985), \u201cIntergroup relations\u201d, The Handbook of Social Psychology, Random House, New York, NY, pp. 559\u2010659."},{"key":"key2022021520003329800_b33","doi-asserted-by":"crossref","unstructured":"Stromberg, H., Vaatanen, A. and Raty, V. (2002), \u201cA group game played in interactive virtual space: design and evaluation\u201d, 4th Conference on Designing Interactive Systems, ACM, New York, NY, pp. 56\u201063.","DOI":"10.1145\/778712.778724"},{"key":"key2022021520003329800_b34","unstructured":"Volk, D. (2008), \u201cCo\u2010creative game development in a participatory Metaverse\u201d, CPSR Indiana University, Bloomington, IN, pp. 262\u2010265."},{"key":"key2022021520003329800_b35","doi-asserted-by":"crossref","unstructured":"Wright, S.C., Aron, A., McLaughlin\u2010Volpe, T. and Ropp, S.A. (1997), \u201cThe extended contact effect: knowledge of cross\u2010group friendships and prejudice\u201d, Journal of Personality and Social Psychology, Vol. 73 No. 1, pp. 73\u201090.","DOI":"10.1037\/0022-3514.73.1.73"}],"container-title":["Interactive Technology and Smart Education"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/ITSE-02-2013-0003","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/ITSE-02-2013-0003\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/ITSE-02-2013-0003\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T21:56:32Z","timestamp":1753394192000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/itse\/article\/10\/2\/130-146\/176232"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,6,7]]},"references-count":35,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2013,6,7]]}},"alternative-id":["10.1108\/ITSE-02-2013-0003"],"URL":"https:\/\/doi.org\/10.1108\/itse-02-2013-0003","relation":{},"ISSN":["1741-5659"],"issn-type":[{"value":"1741-5659","type":"print"}],"subject":[],"published":{"date-parts":[[2013,6,7]]}}}