{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:19:55Z","timestamp":1750220395444,"version":"3.41.0"},"reference-count":35,"publisher":"Association for Computing Machinery (ACM)","issue":"CHI PLAY","license":[{"start":{"date-parts":[[2021,10,5]],"date-time":"2021-10-05T00:00:00Z","timestamp":1633392000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-sa\/4.0\/"}],"funder":[{"name":"REC-RDAP-GBV-AG-2020"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,10,5]]},"abstract":"<jats:p>Gamification has been recently growing in popularity among researchers investigating Information and Communication Technologies. Scholars have been trying to take advantage of this approach in the field of natural language processing (NLP), developing Games With A Purpose (GWAPs) for corpus annotation that have obtained encouraging results both in annotation quality and overall cost. However, GWAPs implement gamification in different ways and to different degrees. We propose a new framework to investigate the mechanics employed in the gamification process and their magnitude in terms of complexity. This framework is based on an analysis of some of the most important contributions in the field of NLP-related gamified applications and GWAP theory. Its primary purpose is to provide a first step towards classifying mechanics that mimic mainstream video games and may require skills that are not relevant to the annotation task, defined as orthogonal mechanics. In order to test our framework, we develop and evaluate Spacewords, a linguistic space game for synonymy annotation.<\/jats:p>","DOI":"10.1145\/3474692","type":"journal-article","created":{"date-parts":[[2021,10,6]],"date-time":"2021-10-06T22:59:48Z","timestamp":1633561188000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Measuring Orthogonal Mechanics in Linguistic Annotation Games"],"prefix":"10.1145","volume":"5","author":[{"given":"Federico","family":"Bonetti","sequence":"first","affiliation":[{"name":"University of Trento &amp; Fondazione Bruno Kessler, Trento, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sara","family":"Tonelli","sequence":"additional","affiliation":[{"name":"Fondazione Bruno Kessler, Trento, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,10,6]]},"reference":[{"key":"e_1_2_3_1_1","volume-title":"CAT: the CELCT Annotation Tool. In Eight International Conference on Language Resources and Evaluation (LREC'12)","author":"Lenzi Valentina Bartalesi","year":"2012","unstructured":"Valentina Bartalesi Lenzi, Giovanni Moretti, and Rachele Sprugnoli. 2012. CAT: the CELCT Annotation Tool. In Eight International Conference on Language Resources and Evaluation (LREC'12). 333--338."},{"key":"e_1_2_3_2_1","volume-title":"Workshop on Games and Natural Language Processing. European Language Resources Association","author":"Bonetti Federico","year":"2020","unstructured":"Federico Bonetti and Sara Tonelli. 2020. A 3D Role-Playing Game for Abusive Language Annotation. In Workshop on Games and Natural Language Processing. European Language Resources Association, Marseille, France, 39--43. https:\/\/aclanthology.org\/2020.gamnlp-1.6"},{"key":"e_1_2_3_3_1","volume-title":"Proceedings of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing. Association for Computational Linguistics, Online, 60--65","author":"Bonetti Federico","year":"2021","unstructured":"Federico Bonetti and Sara Tonelli. 2021. Challenges in Designing Games with a Purpose for Abusive Language Annotation. In Proceedings of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing. Association for Computational Linguistics, Online, 60--65. https:\/\/aclanthology.org\/2021.hcinlp-1.10"},{"key":"e_1_2_3_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978--3--540--69132--7_28"},{"key":"e_1_2_3_5_1","volume-title":"Proceedings of the 1st International Workshop on Crowdsourcing the Semantic Web (CrowdSem 2013), 12th International Semantic Web Conference .","author":"Dumitrache Anca","year":"2013","unstructured":"Anca Dumitrache, Lora Aroyo, Chris Welty, Robert-Jan Sips, and Anthony Levas. 2013. 'Dr. Detective': combining gamification techniques and crowdsourcing to create a gold standard in medical text. In Proceedings of the 1st International Workshop on Crowdsourcing the Semantic Web (CrowdSem 2013), 12th International Semantic Web Conference ."},{"key":"e_1_2_3_6_1","volume-title":"Proceedings of the EACL 2017 Workshop \"Using Games and Gamification for Natural Language Processing (Games4NLP)\". 2.","author":"Dziedzic Dagmara","year":"2017","unstructured":"Dagmara Dziedzic and Wojciech W\u00c5?odarczyk. 2017. Making NLP games with a purpose fun to play using Free to Play mechanics: RoboCorp case study. In Proceedings of the EACL 2017 Workshop \"Using Games and Gamification for Natural Language Processing (Games4NLP)\". 2."},{"key":"e_1_2_3_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594776.2594777"},{"key":"e_1_2_3_8_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-2002"},{"key":"e_1_2_3_9_1","doi-asserted-by":"publisher","DOI":"10.1080\/10508406.2010.508029"},{"key":"e_1_2_3_10_1","volume-title":"Good Game Feel: An Empirically Grounded Framework for Juicy Design. In Digra '18 - Proceedings of the 218 Digra International Conference. 17","author":"Hicks Kieran","year":"2018","unstructured":"Kieran Hicks, Patrick Dickinson, Juicy Holopainen, and Kathrin Gerling. 2018. Good Game Feel: An Empirically Grounded Framework for Juicy Design. In Digra '18 - Proceedings of the 218 Digra International Conference. 17."},{"key":"e_1_2_3_11_1","doi-asserted-by":"publisher","DOI":"10.3115\/1667583.1667648"},{"key":"e_1_2_3_12_1","volume-title":"AAAI Workshop - Technical Report","volume":"1","author":"Hunicke Robin","year":"2004","unstructured":"Robin Hunicke, Marc Leblanc, and Robert Zubek. 2004. MDA: A Formal Approach to Game Design and Game Research . AAAI Workshop - Technical Report , Vol. 1 (01 2004)."},{"key":"e_1_2_3_13_1","volume-title":"Meaningful Play","author":"Jamieson Peter","year":"2012","unstructured":"Peter Jamieson, Jack Hall, and Lindsay Grace. 2012. Research Directions for Pushing Harnessing Human Computation to Mainstream Video Games. In Meaningful Play 2012. East Lansing, MI."},{"key":"e_1_2_3_14_1","volume-title":"Proceedings of the 2018 LREC Workshop \"Games and Gamification for Natural Language Processing (Games4NLP)\"","author":"Joubert Alain","year":"2018","unstructured":"Alain Joubert, Mathieu Lafourcade, and Nathalie Le Brun. 2018. The JeuxDeMots Project is 10 Years Old: What We have Learned. In Proceedings of the 2018 LREC Workshop \"Games and Gamification for Natural Language Processing (Games4NLP)\". Miyazaki, Japan, 22--26. https:\/\/anawiki.essex.ac.uk\/dali\/games4nlp\/"},{"key":"e_1_2_3_15_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00195"},{"key":"e_1_2_3_16_1","volume-title":"A Casual Revolution: Reinventing Video Games and Their Players","author":"Juul Jesper","year":"2009","unstructured":"Jesper Juul. 2010. A Casual Revolution: Reinventing Video Games and Their Players. MIT Press (MA). 2009009091 https:\/\/books.google.it\/books?id=heEsCwAAQBAJ"},{"key":"e_1_2_3_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3337722.3341868"},{"key":"e_1_2_3_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837893"},{"key":"e_1_2_3_19_1","volume-title":"IWCS 2017 -- 12th International Conference on Computational Semantics -- Short papers . https:\/\/www.aclweb.org\/anthology\/W17--6920","author":"Lafourcade Mathieu","year":"2017","unstructured":"Mathieu Lafourcade and Nathalie Le Brun. 2017. Ambiguss, a game for building a Sense Annotated Corpus for French. In IWCS 2017 -- 12th International Conference on Computational Semantics -- Short papers . https:\/\/www.aclweb.org\/anthology\/W17--6920"},{"key":"e_1_2_3_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/CIG.2019.8848068"},{"key":"e_1_2_3_21_1","volume-title":"Proceedings of the 2018 LREC Workshop 'Games and Gamification for Natural Language Processing (Games4NLP)'","author":"Madge Chris","year":"2018","unstructured":"Chris Madge, Massimo Poesio, Udo Kruschwitz, and Jon Chamberlain. 2018. Testing TileAttack with Three Key Audiences. In Proceedings of the 2018 LREC Workshop 'Games and Gamification for Natural Language Processing (Games4NLP)'. Miyazaki, Japan, 6--11. https:\/\/anawiki.essex.ac.uk\/dali\/games4nlp\/"},{"volume-title":"WordNet: An electronic lexical database","author":"Miller George A","key":"e_1_2_3_22_1","unstructured":"George A Miller. 1998. WordNet: An electronic lexical database .MIT press."},{"key":"e_1_2_3_23_1","volume-title":"Proceedings of the Digital Humanities in the Nordic Countries 4th Conference (DHN 2019","volume":"2084","author":"\u00d6hman Emily","year":"2018","unstructured":"Emily \u00d6hman and Kaisla Kajava. 2018. Sentimentator: Gamifying Fine-grained Sentiment Annotation. In Proceedings of the Digital Humanities in the Nordic Countries 4th Conference (DHN 2019), Vol. 2084. Helsinki, Finland, 98--110. http:\/\/ceur-ws.org\/Vol-2084\/"},{"key":"e_1_2_3_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2448116.2448119"},{"key":"e_1_2_3_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2389176.2389188"},{"key":"e_1_2_3_26_1","unstructured":"James Pustejovsky and Amber Stubbs. 2012. Natural Language Annotation for Machine Learning - a Guide to Corpus-Building for Applications .O'Reilly. http:\/\/www.oreilly.de\/catalog\/9781449306663\/index.html"},{"key":"e_1_2_3_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341215.3356310"},{"key":"e_1_2_3_28_1","volume-title":"Game Studies. The International Journal of Computer Game Research","volume":"8","author":"Sicart Miguel","year":"2008","unstructured":"Miguel Sicart. 2008. Defining Game Mechanics. Game Studies. The International Journal of Computer Game Research , Vol. 8 (12 2008), 1--14."},{"key":"e_1_2_3_29_1","volume-title":"Riedl","author":"Siu Kristin","year":"2017","unstructured":"Kristin Siu, Alexander Zook, and Mark O. Riedl. 2017. A Framework for Exploring and Evaluating Mechanics in Human Computation Games . arXiv:1706.03311 [cs] (June 2017). http:\/\/arxiv.org\/abs\/1706.03311 arXiv: 1706.03311."},{"key":"e_1_2_3_30_1","volume-title":"Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics","author":"Stenetorp Pontus","year":"2012","unstructured":"Pontus Stenetorp, Sampo Pyysalo, Goran Topi\u0107, Tomoko Ohta, Sophia Ananiadou, and Jun'ichi Tsujii. 2012. brat: a Web-based Tool for NLP-Assisted Text Annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Avignon, France, 102--107. https:\/\/www.aclweb.org\/anthology\/E12--2021"},{"key":"e_1_2_3_31_1","volume-title":"Proceedings of the 9th International Conference on the Foundations of Digital Games. http:\/\/www.fdg2014","author":"Tuite Kathleen","year":"2014","unstructured":"Kathleen Tuite. 2014. GWAPs: Games with a Problem. In Proceedings of the 9th International Conference on the Foundations of Digital Games. http:\/\/www.fdg2014.org\/proceedings.html"},{"key":"e_1_2_3_32_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1122"},{"key":"e_1_2_3_33_1","volume-title":"Proceedings of the International Conference on Computational Semantics (IWCS). 397--403","author":"Venhuizen Noortje J","year":"2013","unstructured":"Noortje J Venhuizen, Kilian Evang, Valerio Basile, and Johan Bos. 2013. Gamification for Word Sense Labeling. In Proceedings of the International Conference on Computational Semantics (IWCS). 397--403."},{"key":"e_1_2_3_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.196"},{"key":"e_1_2_3_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985733"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3474692","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3474692","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:42Z","timestamp":1750191522000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3474692"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,5]]},"references-count":35,"journal-issue":{"issue":"CHI PLAY","published-print":{"date-parts":[[2021,10,5]]}},"alternative-id":["10.1145\/3474692"],"URL":"https:\/\/doi.org\/10.1145\/3474692","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2021,10,5]]},"assertion":[{"value":"2021-10-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}