{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,20]],"date-time":"2025-11-20T06:42:36Z","timestamp":1763620956734,"version":"3.37.3"},"reference-count":33,"publisher":"Walter de Gruyter GmbH","issue":"1","funder":[{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["16SV8396"],"award-info":[{"award-number":["16SV8396"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,4,27]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The development of artificial intelligence, e.\u2009g. for Computer Vision, through supervised learning requires the input of large amounts of annotated or labeled data objects as training data. Usually, the creation of high-quality training data is done manually which can be repetitive and tiring. <jats:italic>Gamification<\/jats:italic>, the use of game elements in a non-game context, is one method to make such tedious tasks more interesting. We propose a multi-step process for gamifying the manual creation of training data for machine learning purposes. In this article, we give an overview of related concepts and existing implementations and present a user-centered approach for a real-life use case. Based on a survey within the target user group we identified annotation use cases and dominant player characteristics. The results served as a foundation for designing the gamification concepts which were then discussed with the participants. The final concept includes levels of increasing difficulty, tutorials, progress indicators and a narrative built around a robot character which at the same time is a user assistant. The implemented prototype is an extension of an existing annotation tool at an AI product company and serves as a basis for further observations.<\/jats:p>","DOI":"10.1515\/icom-2020-0030","type":"journal-article","created":{"date-parts":[[2021,4,23]],"date-time":"2021-04-23T14:10:33Z","timestamp":1619187033000},"page":"33-48","source":"Crossref","is-referenced-by-count":6,"title":["A User-Centered Approach to Gamify the Manual Creation of Training Data for Machine Learning"],"prefix":"10.1515","volume":"20","author":[{"given":"Sarah","family":"Alaghbari","sequence":"first","affiliation":[{"name":"AI4BD Deutschland GmbH , Dresden , Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8667-0926","authenticated-orcid":false,"given":"Annett","family":"Mitschick","sequence":"additional","affiliation":[{"name":"9169 Technische Universit\u00e4t Dresden , Dresden , Germany"}]},{"given":"Gregor","family":"Blichmann","sequence":"additional","affiliation":[{"name":"AI4BD Deutschland GmbH , Dresden , Germany"}]},{"given":"Martin","family":"Voigt","sequence":"additional","affiliation":[{"name":"AI4BD Deutschland GmbH , Dresden , Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2176-876X","authenticated-orcid":false,"given":"Raimund","family":"Dachselt","sequence":"additional","affiliation":[{"name":"Centre for Tactile Internet with Human-in-the-Loop (CeTI) , 9169 Technische Universit\u00e4t Dresden , Dresden , Germany"}]}],"member":"374","published-online":{"date-parts":[[2021,4,22]]},"reference":[{"key":"2023033117384574704_j_icom-2020-0030_ref_001","doi-asserted-by":"crossref","unstructured":"Maximilian Altmeyer, Pascal Lessel, and Antonio Kr\u00fcger. 2016. Expense Control: A Gamified, Semi-Automated, Crowd-Based Approach For Receipt Capturing. In Proceedings of the 21st International Conference on Intelligent User Interfaces (IUI \u201916). Association for Computing Machinery, New York, NY, USA, 31\u201342. https:\/\/doi.org\/10.1145\/2856767.2856790.","DOI":"10.1145\/2856767.2856790"},{"key":"2023033117384574704_j_icom-2020-0030_ref_002","doi-asserted-by":"crossref","unstructured":"Fernando Andrade, Riichiro Mizoguchi, and Seiji Isotani. 2016. The Bright and Dark Sides of Gamification. In Intelligent Tutoring Systems, Alessandro Micarelli, John Stamper, and Kitty Panourgia (Eds.). Lecture Notes in Computer Science, Vol.\u20099684. Springer International Publishing, 1\u201311. https:\/\/doi.org\/10.1007\/978-3-319-39583-8_17.","DOI":"10.1007\/978-3-319-39583-8_17"},{"key":"2023033117384574704_j_icom-2020-0030_ref_003","doi-asserted-by":"crossref","unstructured":"Gokhan Aydin. 2018. Effect of demographics on use intention of gamified systems. International Journal of Technology and Human Interaction (IJTHI) 14, 1, 1\u201321.","DOI":"10.4018\/IJTHI.2018010101"},{"key":"2023033117384574704_j_icom-2020-0030_ref_004","unstructured":"Richard Bartle. 1996. Hearts, clubs, diamonds, spades: Players who suit MUDs. Journal of MUD research 1, 1, 19."},{"key":"2023033117384574704_j_icom-2020-0030_ref_005","doi-asserted-by":"crossref","unstructured":"Martin B\u00f6ckle, Isabel Micheel, Markus Bick, and Jasminko Novak. 2018. A Design Framework for Adaptive Gamification Applications. In Proceedings of the 51st Hawaii International Conference on System Sciences (Proceedings of the Annual Hawaii International Conference on System Sciences), Tung Bui (Ed.). Hawaii International Conference on System Sciences. https:\/\/doi.org\/10.24251\/HICSS.2018.151.","DOI":"10.24251\/HICSS.2018.151"},{"key":"2023033117384574704_j_icom-2020-0030_ref_006","unstructured":"Bunchball. 2015. What are Game Mechanics? https:\/\/www.bunchball.com\/gamification\/game-mechanics."},{"key":"2023033117384574704_j_icom-2020-0030_ref_007","doi-asserted-by":"crossref","unstructured":"Rachel C. Callan, Kristina N. Bauer, and Richard N. Landers. 2015. How to avoid the dark side of gamification: Ten business scenarios and their unintended consequences. In Gamification in education and business. Springer, 553\u2013568.","DOI":"10.1007\/978-3-319-10208-5_28"},{"key":"2023033117384574704_j_icom-2020-0030_ref_008","doi-asserted-by":"crossref","unstructured":"Sebastian Deterding, Staffan Bj\u00f6rk, Lennart Nacke, Dan Dixon, and Elizabeth Lawley. 2013. Designing gamification: creating gameful and playful experiences. In Extended Abstracts on Human Factors in Computing Systems. 3263\u20133266. https:\/\/doi.org\/10.1145\/2468356.2479662.","DOI":"10.1145\/2468356.2479662"},{"key":"2023033117384574704_j_icom-2020-0030_ref_009","doi-asserted-by":"crossref","unstructured":"Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011. From Game Design Elements to Gamefulness: Defining Gamification. In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments (MindTrek \u201911). ACM, 9\u201315. https:\/\/doi.org\/10.1145\/2181037.2181040.","DOI":"10.1145\/2181037.2181040"},{"key":"2023033117384574704_j_icom-2020-0030_ref_010","unstructured":"Dominique Mangiatordi. 2018. Gamification at work: the 8 PLAYER TYPES. https:\/\/www.linkedin.com\/pulse\/gamification-work-8-player-types-dominique-mangiatordi-\/."},{"key":"2023033117384574704_j_icom-2020-0030_ref_011","doi-asserted-by":"crossref","unstructured":"Lauren Ferro, Steffen Walz, and Stefan Greuter. 2013. Towards personalised, gamified systems: An investigation into game design, personality and player typologies. In Proceedings of The 9th Australasian Conference on Interactive Entertainment Matters of Life and Death. https:\/\/doi.org\/10.1145\/2513002.2513024.","DOI":"10.1145\/2513002.2513024"},{"key":"2023033117384574704_j_icom-2020-0030_ref_012","unstructured":"Google. 2020. Google Crowdsource. https:\/\/crowdsource.google.com\/."},{"key":"2023033117384574704_j_icom-2020-0030_ref_013","doi-asserted-by":"crossref","unstructured":"Simone Hantke, Tobias Appel, and Bj\u00f6rn Schuller. 2018. The Inclusion of Gamification Solutions to Enhance User Enjoyment on Crowdsourcing Platforms. In 2018 First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia). 1\u20136. https:\/\/doi.org\/10.1109\/ACIIAsia.2018.8470330.","DOI":"10.1109\/ACIIAsia.2018.8470330"},{"key":"2023033117384574704_j_icom-2020-0030_ref_014","doi-asserted-by":"crossref","unstructured":"Eduardo Herranz, Ricardo Colomo-Palacios, and Antonio de Amescua Seco. 2015. Gamiware: A Gamification Platform for Software Process Improvement. In Systems, Software and Services Process Improvement, Rory V. O\u2019Connor, Mariye Umay Akkaya, Kerem Kemaneci, Murat Yilmaz, Alexander Poth, and Richard Messnarz (Eds.). Springer International Publishing, Cham, 127\u2013139.","DOI":"10.1007\/978-3-319-24647-5_11"},{"key":"2023033117384574704_j_icom-2020-0030_ref_015","doi-asserted-by":"crossref","unstructured":"Karl Kapp. 2012. The gamification of learning and instruction: Game-based methods and strategies for training and education. Pfeiffer, San Francisco, CA.","DOI":"10.1145\/2207270.2211316"},{"key":"2023033117384574704_j_icom-2020-0030_ref_016","doi-asserted-by":"crossref","unstructured":"Alireza Khakpour and Ricardo Colomo-Palacios. 2020. Convergence of Gamification and Machine Learning: A Systematic Literature Review. Technology, Knowledge and Learning. https:\/\/doi.org\/10.1007\/s10758-020-09456-4.","DOI":"10.1007\/s10758-020-09456-4"},{"key":"2023033117384574704_j_icom-2020-0030_ref_017","unstructured":"Selay Ark\u00fcn Kocadere and \u015eeyma \u00c7a\u011flar. 2018. Gamification from player type perspective: A case study. Journal of Educational Technology & Society 21, 3, 12\u201322."},{"key":"2023033117384574704_j_icom-2020-0030_ref_018","doi-asserted-by":"crossref","unstructured":"Jonna Koivisto and Aqdas Malik. 2020. Gamification for Older Adults: A Systematic Literature Review. The Gerontologist. https:\/\/doi.org\/10.1093\/geront\/gnaa047.","DOI":"10.1093\/geront\/gnaa047"},{"key":"2023033117384574704_j_icom-2020-0030_ref_019","doi-asserted-by":"crossref","unstructured":"Janaki Kumar and Mario Herger. 2013. Gamification at Work: Designing Engaging Business Software. In Design, User Experience, and Usability. Health, Learning, Playing, Cultural, and Cross-Cultural User Experience, Aaron Marcus (Ed.). Springer Berlin Heidelberg, 528\u2013537.","DOI":"10.1007\/978-3-642-39241-2_58"},{"key":"2023033117384574704_j_icom-2020-0030_ref_020","doi-asserted-by":"crossref","unstructured":"Alexandra L\u2019Heureux, Katarina Grolinger, Wilson Akio Higashino, and Miriam A.\u2009M. Capretz. 2017. A Gamification Framework for Sensor Data Analytics. In 2017 IEEE International Congress on Internet of Things (ICIOT). 74\u201381. https:\/\/doi.org\/10.1109\/IEEE.ICIOT.2017.18.","DOI":"10.1109\/IEEE.ICIOT.2017.18"},{"key":"2023033117384574704_j_icom-2020-0030_ref_021","unstructured":"Daria Lopukhina. 2018. How Gamification in the Workplace Impacts Employee Productivity. https:\/\/anadea.info\/blog\/how-gamification-in-the-workplace-impacts-employee-productivity."},{"key":"2023033117384574704_j_icom-2020-0030_ref_022","unstructured":"Andrzej Marczewski. 2013. Game Mechanics in Gamification. https:\/\/www.gamified.uk\/2013\/01\/14\/game-mechanics-in-gamification\/."},{"key":"2023033117384574704_j_icom-2020-0030_ref_023","unstructured":"Andrzej Marczewski. 2013. The Intrinsic Motivation RAMP. https:\/\/www.gamified.uk\/gamification-framework\/the-intrinsic-motivation-ramp\/."},{"key":"2023033117384574704_j_icom-2020-0030_ref_024","unstructured":"Andrzej Marczewski. 2015. Even ninja monkeys like to play: Gamification, game thinking & motivational design. Gamified UK. http:\/\/gamified.uk\/user-types\/."},{"key":"2023033117384574704_j_icom-2020-0030_ref_025","unstructured":"Jane McGonigal. 2011. Reality is broken. Penguin Press. http:\/\/www.loc.gov\/catdir\/enhancements\/fy1107\/2010029619-d.html."},{"key":"2023033117384574704_j_icom-2020-0030_ref_026","doi-asserted-by":"crossref","unstructured":"Nina Runge, Dirk Wenig, Danny Zitzmann, and Rainer Malaka. 2015. Tags You Don\u2019t Forget: Gamified Tagging of Personal Images. In Entertainment Computing \u2013 ICEC 2015, Konstantinos Chorianopoulos, Monica Divitini, Jannicke Baalsrud Hauge, Letizia Jaccheri, and Rainer Malaka (Eds.). Springer International Publishing, 301\u2013314.","DOI":"10.1007\/978-3-319-24589-8_23"},{"key":"2023033117384574704_j_icom-2020-0030_ref_027","doi-asserted-by":"crossref","unstructured":"Richard M. Ryan and Edward L. Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist 55, 1, 68\u201378. https:\/\/doi.org\/10.1037\/0003-066X.55.1.68.","DOI":"10.1037\/0003-066X.55.1.68"},{"key":"2023033117384574704_j_icom-2020-0030_ref_028","doi-asserted-by":"crossref","unstructured":"Lorenzo Servadei, Rainer Schmidt, Christina Eidelloth, and Andreas Maier. 2018. In Medical Monkeys: A Crowdsourcing Approach to Medical Big Data. 87\u201397. https:\/\/doi.org\/10.1007\/978-3-319-73805-5_9.","DOI":"10.1007\/978-3-319-73805-5_9"},{"key":"2023033117384574704_j_icom-2020-0030_ref_029","unstructured":"Samuel Suikkanen. 2019-06-17. Gamification in video labeling; Videoiden merkkaamisen pelillist\u00e4minen. G2 pro gradu, diplomity\u00f6. http:\/\/urn.fi\/URN:NBN:fi:aalto-201906233997."},{"key":"2023033117384574704_j_icom-2020-0030_ref_030","unstructured":"Richard H. Thaler and Cass R. Sunstein. 2008. Nudge \u2013 Improving Decisions About Health, Wealth, and Happiness (rev. and expanded ed., with a new afterword and a new chapter ed.). Penguin. 304 Pages."},{"key":"2023033117384574704_j_icom-2020-0030_ref_031","doi-asserted-by":"crossref","unstructured":"Rob van Roy, Sebastian Deterding, and Bieke Zaman. 2018. Uses and Gratifications of Initiating Use of Gamified Learning Platforms. In Asian CHI Symposium 2018, Eunice Sari (Ed.). CHI Conference, ACM, 1\u20136. https:\/\/doi.org\/10.1145\/3170427.3188458.","DOI":"10.1145\/3170427.3188458"},{"key":"2023033117384574704_j_icom-2020-0030_ref_032","doi-asserted-by":"crossref","unstructured":"Luis von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of the 2004 conference on Human factors in computing systems \u2013 CHI \u201904, Elizabeth Dykstra-Erickson and Manfred Tscheligi (Eds.). ACM Press, 319\u2013326. https:\/\/doi.org\/10.1145\/985692.985733.","DOI":"10.1145\/985692.985733"},{"key":"2023033117384574704_j_icom-2020-0030_ref_033","doi-asserted-by":"crossref","unstructured":"Lincoln Wood and Torsten Reiners. 2015. Gamification. In Encyclopedia of Information Science and Technology. 3039\u20133047. https:\/\/doi.org\/10.4018\/978-1-4666-5888-2.ch297.","DOI":"10.4018\/978-1-4666-5888-2.ch297"}],"container-title":["i-com"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.degruyter.com\/document\/doi\/10.1515\/icom-2020-0030\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.degruyter.com\/document\/doi\/10.1515\/icom-2020-0030\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,4,1]],"date-time":"2023-04-01T00:41:10Z","timestamp":1680309670000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.degruyter.com\/document\/doi\/10.1515\/icom-2020-0030\/html"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,1]]},"references-count":33,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,4,22]]},"published-print":{"date-parts":[[2021,4,27]]}},"alternative-id":["10.1515\/icom-2020-0030"],"URL":"https:\/\/doi.org\/10.1515\/icom-2020-0030","relation":{},"ISSN":["2196-6826","1618-162X"],"issn-type":[{"type":"electronic","value":"2196-6826"},{"type":"print","value":"1618-162X"}],"subject":[],"published":{"date-parts":[[2021,4,1]]}}}