{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,23]],"date-time":"2025-06-23T05:46:01Z","timestamp":1750657561838,"version":"3.41.0"},"reference-count":75,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2021,12,1]],"date-time":"2021-12-01T00:00:00Z","timestamp":1638316800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:p>We present a systematic literature review of gamification in information retrieval, identifying 32 peer-reviewed articles spanning a period from 2012 to present day. Our study highlights that, with the exception of crowdsourcing relevance judgments, very few studies have investigated the application of gamification to search systems. Moreover, in articles that we would strictly define as being related to gamification (i.e. not games with a purpose or serious games), a majority did not present any real-world systems, user studies or empirical results. In numerous other fields, gamification has been used to successfully increase user engagement and productivity. We argue that the potential of gamification has been overlooked in information retrieval and, in particular, in interactive search settings. We conclude by discussing experimental obstacles and the potential benefits of evaluating gamification strategies in future interactive search systems.<\/jats:p>","DOI":"10.1145\/3527546.3527551","type":"journal-article","created":{"date-parts":[[2022,3,17]],"date-time":"2022-03-17T23:24:36Z","timestamp":1647559476000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Game over?"],"prefix":"10.1145","volume":"55","author":[{"given":"Alan","family":"Medlar","sequence":"first","affiliation":[{"name":"University of Helsinki"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dorota","family":"G\u0142owacka","sequence":"additional","affiliation":[{"name":"University of Helsinki"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,3,17]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Serious games","author":"Abt Clark C","year":"1987","unstructured":"Clark C Abt . Serious games . University press of America , 1987 . Clark C Abt. Serious games. University press of America, 1987."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13012-014-0125-9"},{"key":"e_1_2_1_3_1","volume-title":"International journal of medical informatics, 96:62--70","author":"Alah\u00e4iv\u00e4l\u00e4 Tuomas","year":"2016","unstructured":"Tuomas Alah\u00e4iv\u00e4l\u00e4 and Harri Oinas-Kukkonen . Understanding persuasion contexts in health gamification: A systematic analysis of gamified health behavior change support systems literature . International journal of medical informatics, 96:62--70 , 2016 . Tuomas Alah\u00e4iv\u00e4l\u00e4 and Harri Oinas-Kukkonen. Understanding persuasion contexts in health gamification: A systematic analysis of gamified health behavior change support systems literature. International journal of medical informatics, 96:62--70, 2016."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.21236\/ADA439446"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2856767.2856786"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594776.2594784"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the Third International Workshop on Gamification for Information Retrieval (GamifIR)","author":"Barr Matthew","year":"2016","unstructured":"Matthew Barr , Kay Munro , and Frank Hopfgartner . Increasing engagement with the library via gamification . In Proceedings of the Third International Workshop on Gamification for Information Retrieval (GamifIR) , 2016 . Matthew Barr, Kay Munro, and Frank Hopfgartner. Increasing engagement with the library via gamification. In Proceedings of the Third International Workshop on Gamification for Information Retrieval (GamifIR), 2016."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1038\/nbt.2955"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594776.2594781"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-56608-5_9"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocv100"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature09304"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2669711.2669898"},{"key":"e_1_2_1_14_1","first-page":"1285","volume-title":"The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Dalton Jeffrey","year":"2018","unstructured":"Jeffrey Dalton , Victor Ajayi , and Richard Main . Vote goat : Conversational movie recommendation . In The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 1285 -- 1288 , 2018 . Jeffrey Dalton, Victor Ajayi, and Richard Main. Vote goat: Conversational movie recommendation. In The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1285--1288, 2018."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaa013"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2181037.2181040"},{"key":"e_1_2_1_17_1","volume-title":"GamifIR@ SIGIR","author":"Di Nunzio Giorgio Maria","year":"2016","unstructured":"Giorgio Maria Di Nunzio , Maria Maistro , and Daniel Zilio . Gamification for machine learning: The classification game . In GamifIR@ SIGIR , 2016 . Giorgio Maria Di Nunzio, Maria Maistro, and Daniel Zilio. Gamification for machine learning: The classification game. In GamifIR@ SIGIR, 2016."},{"key":"e_1_2_1_18_1","first-page":"123","volume-title":"The trec interactive tracks: Putting the user into search. TREC: Experiment and evaluation in information retrieval","author":"Dumais Susan T","year":"2005","unstructured":"Susan T Dumais and Nicholas J Belkin . The trec interactive tracks: Putting the user into search. TREC: Experiment and evaluation in information retrieval , pages 123 -- 152 , 2005 . Susan T Dumais and Nicholas J Belkin. The trec interactive tracks: Putting the user into search. TREC: Experiment and evaluation in information retrieval, pages 123--152, 2005."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2854946.2855000"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080818"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348400"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594776.2594785"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594776.2594778"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-44564-9_18"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-36973-5_5"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1002\/bjs.8819"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2014.377"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594776.2594780"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDMW.2017.31"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594776.2594787"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12525-015-0212-z"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the Third International Workshop on Gamification for Information Retrieval (GamifIR)","author":"Jin Yuan","year":"2016","unstructured":"Yuan Jin , Mark James Carman , and Lexing Xie . A little competition never hurt anyone's relevance assessments . In Proceedings of the Third International Workshop on Gamification for Information Retrieval (GamifIR) , 2016 . Yuan Jin, Mark James Carman, and Lexing Xie. A little competition never hurt anyone's relevance assessments. In Proceedings of the Third International Workshop on Gamification for Information Retrieval (GamifIR), 2016."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3020165.3022171"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the Third International Workshop on Gamification for Information Retrieval (GamifIR)","author":"Karatassis Ioannis","year":"2016","unstructured":"Ioannis Karatassis and Norbert Fuhr . Gamification for websail . In Proceedings of the Third International Workshop on Gamification for Information Retrieval (GamifIR) , 2016 . Ioannis Karatassis and Norbert Fuhr. Gamification for websail. In Proceedings of the Third International Workshop on Gamification for Information Retrieval (GamifIR), 2016."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2808194.2809465"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2018.10.013"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.290986"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1111\/bjet.12639"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1177\/1046878114563660"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10208-5_9"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2960310.2960328"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594776.2594789"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1645990"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.11613\/BM.2012.031"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the Third International Workshop on Gamification for Information Retrieval (GamifIR)","author":"Meder Michael","year":"2016","unstructured":"Michael Meder , Till Plumbaum , and Sahin Albayrak . Learning gamification design-an usability first approach for the enterprise infoboard experiment . In Proceedings of the Third International Workshop on Gamification for Information Retrieval (GamifIR) , 2016 . Michael Meder, Till Plumbaum, and Sahin Albayrak. Learning gamification design-an usability first approach for the enterprise infoboard experiment. In Proceedings of the Third International Workshop on Gamification for Information Retrieval (GamifIR), 2016."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025171.3025205"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767727"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2583008.2583017"},{"key":"e_1_2_1_49_1","first-page":"26","volume-title":"Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR)","author":"Moazzam Waqas","year":"2015","unstructured":"Waqas Moazzam , Michael Riegler , Sagar Sen , and Mari Nyg\u00e5rd . Scientific hangman : Gamifying scientific evidence for general public . In Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR) , pages 26 -- 33 , 2015 . Waqas Moazzam, Michael Riegler, Sagar Sen, and Mari Nyg\u00e5rd. Scientific hangman: Gamifying scientific evidence for general public. In Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR), pages 26--33, 2015."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1159\/000055014"},{"key":"e_1_2_1_51_1","volume-title":"IIR","author":"Muntean Cristina Ioana","year":"2015","unstructured":"Cristina Ioana Muntean and Franco Maria Nardini . Gamification in information retrieval: State of the art, challenges and opportunities . In IIR , 2015 . Cristina Ioana Muntean and Franco Maria Nardini. Gamification in information retrieval: State of the art, challenges and opportunities. In IIR, 2015."},{"key":"e_1_2_1_52_1","volume-title":"The design of everyday things: Revised and expanded edition. Basic books","author":"Norman Don","year":"2013","unstructured":"Don Norman . The design of everyday things: Revised and expanded edition. Basic books , 2013 . Don Norman. The design of everyday things: Revised and expanded edition. Basic books, 2013."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2020.102226"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.3233\/JIFS-179333"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2362724.2362770"},{"key":"e_1_2_1_56_1","first-page":"19","volume-title":"Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR)","author":"Radu Anca-Livia","year":"2015","unstructured":"Anca-Livia Radu , Aliaksandr Autayeu , Bogdan Ionescu , and Fausto Giunchiglia . High quality photo collection via gamification . In Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR) , pages 19 -- 25 , 2015 . Anca-Livia Radu, Aliaksandr Autayeu, Bogdan Ionescu, and Fausto Giunchiglia. High quality photo collection via gamification. In Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR), pages 19--25, 2015."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10055-016-0298-4"},{"key":"e_1_2_1_58_1","first-page":"34","volume-title":"Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR)","author":"Riegler Michael","year":"2015","unstructured":"Michael Riegler , Ragnhild Eg , Lilian Calvet , Mathias Lux , P\u00e5l Halvorsen , and Carsten Griwodz . Playing around the eye tracker-a serious game based dataset . In Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR) , pages 34 -- 40 , 2015 . Michael Riegler, Ragnhild Eg, Lilian Calvet, Mathias Lux, P\u00e5l Halvorsen, and Carsten Griwodz. Playing around the eye tracker-a serious game based dataset. In Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR), pages 34--40, 2015."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2015.2441003"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.55.1.68"},{"key":"e_1_2_1_61_1","first-page":"7","volume-title":"Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR)","author":"Rybak Jan","year":"2015","unstructured":"Jan Rybak , Krisztian Balog , and Kjetil N\u00f8rv\u00e5g . Ir game : How well do you know information retrieval papers ? In Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR) , pages 7 -- 12 , 2015 . Jan Rybak, Krisztian Balog, and Kjetil N\u00f8rv\u00e5g. Ir game: How well do you know information retrieval papers? In Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR), pages 7--12, 2015."},{"key":"e_1_2_1_62_1","first-page":"13","volume-title":"Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR)","author":"Schl\u00f6tterer J\u00f6rg","year":"2015","unstructured":"J\u00f6rg Schl\u00f6tterer , Christin Seifert , Lisa Wagner , and Michael Granitzer . A game with a purpose to access europe's cultural treasure . In Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR) , pages 13 -- 18 , 2015 . J\u00f6rg Schl\u00f6tterer, Christin Seifert, Lisa Wagner, and Michael Granitzer. A game with a purpose to access europe's cultural treasure. In Proceedings of the Second International Workshop on Gamification for Information Retrieval (GamifIR), pages 13--18, 2015."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2016.7757402"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594776.2594786"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1515\/9783110260588.376"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3406522.3446031"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/1028174.971391"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.196"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985733"},{"key":"e_1_2_1_70_1","volume-title":"Journal of Documentation","author":"Wildemuth Barbara","year":"2014","unstructured":"Barbara Wildemuth , Luanne Freund , and Elaine G Toms . Untangling search task complexity and difficulty in the context of interactive information retrieval studies . Journal of Documentation , 2014 . Barbara Wildemuth, Luanne Freund, and Elaine G Toms. Untangling search task complexity and difficulty in the context of interactive information retrieval studies. Journal of Documentation, 2014."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24027-5_34"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1177\/0038026117728620"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1080\/10548408.2015.1093999"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1089\/cpb.2006.9.772"},{"key":"e_1_2_1_75_1","volume-title":"Gamification by design: Implementing game mechanics in web and mobile apps. \" O'Reilly Media","author":"Zichermann Gabe","year":"2011","unstructured":"Gabe Zichermann and Christopher Cunningham . Gamification by design: Implementing game mechanics in web and mobile apps. \" O'Reilly Media , Inc .\", 2011 . Gabe Zichermann and Christopher Cunningham. Gamification by design: Implementing game mechanics in web and mobile apps. \" O'Reilly Media, Inc.\", 2011."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3527546.3527551","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3527546.3527551","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:53Z","timestamp":1750191533000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3527546.3527551"}},"subtitle":["a review of gamification in information retrieval"],"short-title":[],"issued":{"date-parts":[[2021,12]]},"references-count":75,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["10.1145\/3527546.3527551"],"URL":"https:\/\/doi.org\/10.1145\/3527546.3527551","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2021,12]]},"assertion":[{"value":"2022-03-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}