{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:11:50Z","timestamp":1750306310188,"version":"3.41.0"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2016,3,31]],"date-time":"2016-03-31T00:00:00Z","timestamp":1459382400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2016,7,14]]},"abstract":"<jats:p>Despite the ever-increasing popularity of crowdsourcing (CS) in both industry and academia, procedures that ensure quality in its results are still elusive. We hypothesise that a CS design based on game theory can persuade workers to perform their tasks as quickly as possible with the highest quality. In order to do so, in this article we propose a CS framework inspired by the<jats:italic>n<\/jats:italic>-person Chicken game. Our aim is to address the problem of CS quality without compromising on CS benefits such as low monetary cost and high task completion speed. With that goal in mind, we study the effects of knowledge updates as well as incentives for good workers to continue playing. We define a general task with the characteristics of relevance assessment as a case study, because it has been widely explored in the past with CS due to its potential cost and complexity. In order to investigate our hypotheses, we conduct a simulation where we study the effect of the proposed framework on data accuracy, task completion time, and total monetary rewards. Based on a game-theoretical analysis, we study how different types of individuals would behave under a particular game scenario. In particular, we simulate a population comprised of different types of workers with varying ability to formulate optimal strategies and learn from their experiences. A simulation of the proposed framework produced results that support our hypothesis.<\/jats:p>","DOI":"10.1145\/2873063","type":"journal-article","created":{"date-parts":[[2016,3,31]],"date-time":"2016-03-31T11:55:39Z","timestamp":1459425339000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["A Game-Theory Approach for Effective Crowdsource-Based Relevance Assessment"],"prefix":"10.1145","volume":"7","author":[{"given":"Yashar","family":"Moshfeghi","sequence":"first","affiliation":[{"name":"University of Glasgow, Glasgow, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alvaro Francisco Huertas","family":"Rosero","sequence":"additional","affiliation":[{"name":"University of Glasgow, Glasgow, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joemon M.","family":"Jose","sequence":"additional","affiliation":[{"name":"University of Glasgow, Glasgow, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,3,31]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9204-1"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/1996889.1996910"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation. 557--566","author":"Alonso Omar","year":"2009","unstructured":"Omar Alonso and Stefano Mizzaro . 2009 . Can we get rid of TREC assessors? Using mechanical turk for relevance assessment . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation. 557--566 . Omar Alonso and Stefano Mizzaro. 2009. Can we get rid of TREC assessors? Using mechanical turk for relevance assessment. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation. 557--566."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_2_1_6_1","first-page":"2169","article-title":"Active learning and crowd-sourcing for machine translation","volume":"2","author":"Ambati Vamshi","year":"2010","unstructured":"Vamshi Ambati , Stephan Vogel , and Jaime Carbonell . 2010 . Active learning and crowd-sourcing for machine translation . Lang. Resourc. Eval. 2 , 1 (2010), 2169 -- 2174 . Vamshi Ambati, Stephan Vogel, and Jaime Carbonell. 2010. Active learning and crowd-sourcing for machine translation. Lang. Resourc. Eval. 2, 1 (2010), 2169--2174.","journal-title":"Lang. Resourc. Eval."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/1996889.1996909"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390447"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010039"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835540"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1979742.1979575"},{"volume-title":"The Analysis of International Relations","author":"Deutsch Karl Wolfgang","key":"e_1_2_1_12_1","unstructured":"Karl Wolfgang Deutsch . 1968. The Analysis of International Relations . Vol. 12 . Prentice-Hall , Englewood Cliffs, NJ . Karl Wolfgang Deutsch. 1968. The Analysis of International Relations. Vol. 12. Prentice-Hall, Englewood Cliffs, NJ."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1566374.1566392"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753688"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the Workshop on Crowdsourcing for Search and Data Mining (CSDM) at the 4th ACM International Conference on Web Search and Data Mining (WSDM). 11--14","author":"Eickhoff Carsten","year":"2011","unstructured":"Carsten Eickhoff and Arjen de Vries . 2011 . How crowdsourcable is your task . In Proceedings of the Workshop on Crowdsourcing for Search and Data Mining (CSDM) at the 4th ACM International Conference on Web Search and Data Mining (WSDM). 11--14 . Carsten Eickhoff and Arjen de Vries. 2011. How crowdsourcable is your task. In Proceedings of the Workshop on Crowdsourcing for Search and Data Mining (CSDM) at the 4th ACM International Conference on Web Search and Data Mining (WSDM). 11--14."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-011-9181-9"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348400"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187910"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866723"},{"key":"e_1_2_1_20_1","volume-title":"Harsanyi and Reinhard Selten","author":"John","year":"1988","unstructured":"John C. Harsanyi and Reinhard Selten . 1988 . A General Theory of Equilibrium Selection in Games. MIT Press , Cambridge, MA. John C. Harsanyi and Reinhard Selten. 1988. A General Theory of Equilibrium Selection in Games. MIT Press, Cambridge, MA."},{"key":"e_1_2_1_21_1","volume-title":"Riley","author":"Hirshleifer Jack","year":"1992","unstructured":"Jack Hirshleifer and John G . Riley . 1992 . The Analytics of Uncertainty and Information. Cambridge University Press , Cambridge. Jack Hirshleifer and John G. Riley. 1992. The Analytics of Uncertainty and Information. Cambridge University Press, Cambridge."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807342.1807376"},{"key":"e_1_2_1_23_1","first-page":"159","article-title":"Crowdsourcing: A definition","volume":"33","author":"Howe J.","year":"2006","unstructured":"J. Howe . 2006 . Crowdsourcing: A definition . Crowdsourcing Tracking the Rise of the Amateur 33 (2006), 159 -- 174 . J. Howe. 2006. Crowdsourcing: A definition. Crowdsourcing Tracking the Rise of the Amateur 33 (2006), 159--174.","journal-title":"Crowdsourcing Tracking the Rise of the Amateur"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1869086.1869094"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600171"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600171"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the Worlds 1st Conference on the Future of Distributed Work (CrowdConf\u201910)","author":"Kapelner Adam","year":"2010","unstructured":"Adam Kapelner and Dana Chandler . 2010 . Preventing satisficing in online surveys: A \u2018kapcha\u2019 to ensure higher quality data . In Proceedings of the Worlds 1st Conference on the Future of Distributed Work (CrowdConf\u201910) . Adam Kapelner and Dana Chandler. 2010. Preventing satisficing in online surveys: A \u2018kapcha\u2019 to ensure higher quality data. In Proceedings of the Worlds 1st Conference on the Future of Distributed Work (CrowdConf\u201910)."},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the 17th Americas Conference on Information Systems (AMCIS\u201911)","author":"Kaufmann Nicolas","year":"2011","unstructured":"Nicolas Kaufmann , Thimo Schulze , and Daniel Veit . 2011 . More than fun and money. Worker motivation in crowdsourcing-a study on mechanical turk . In Proceedings of the 17th Americas Conference on Information Systems (AMCIS\u201911) . Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker motivation in crowdsourcing-a study on mechanical turk. In Proceedings of the 17th Americas Conference on Information Systems (AMCIS\u201911)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/1996889.1996911"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2396779"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063860"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398697"},{"volume-title":"Service-Oriented Computing","author":"Kern Robert","key":"e_1_2_1_34_1","unstructured":"Robert Kern , Hans Thies , and Gerhard Satzger . 2010. Statistical quality control for human-based electronic services . In Service-Oriented Computing . Springer , Berlin , 243--257. Robert Kern, Hans Thies, and Gerhard Satzger. 2010. Statistical quality control for human-based electronic services. In Service-Oriented Computing. Springer, Berlin, 243--257."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458160"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010082"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation. 21--26","author":"Le John","year":"2010","unstructured":"John Le , Andy Edmonds , Vaughn Hester , and Lukas Biewald . 2010 . Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution . In Proceedings of the SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation. 21--26 . John Le, Andy Edmonds, Vaughn Hester, and Lukas Biewald. 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In Proceedings of the SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation. 21--26."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-011-0124-6"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.5555\/647301.721460"},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1080\/00031305.1978.10479236","article-title":"Variations of box plots","volume":"32","author":"McGill Robert","year":"1978","unstructured":"Robert McGill , John W. Tukey , and Wayne A. Larsen . 1978 . Variations of box plots . Am. Statist. 32 , 1 (1978), 12 -- 16 . Robert McGill, John W. Tukey, and Wayne A. Larsen. 1978. Variations of box plots. Am. Statist. 32, 1 (1978), 12--16.","journal-title":"Am. Statist."},{"key":"e_1_2_1_43_1","volume-title":"Hanson","author":"Menezes Carmen F.","year":"1970","unstructured":"Carmen F. Menezes and David L . Hanson . 1970 . On the theory of risk aversion. Int. Econ. Review ( 1970), 481--487. Carmen F. Menezes and David L. Hanson. 1970. On the theory of risk aversion. Int. Econ. Review (1970), 481--487."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.paid.2010.05.032"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-36973-5_26"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357225"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1743384.1743478"},{"key":"e_1_2_1_48_1","volume-title":"Osborne and Ariel Rubinstein","author":"Martin","year":"1994","unstructured":"Martin J. Osborne and Ariel Rubinstein . 1994 . A Course in Game Theory. MIT Press , Cambridge, MA. Martin J. Osborne and Ariel Rubinstein. 1994. A Course in Game Theory. MIT Press, Cambridge, MA."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.5555\/2062780.2062813"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1753873"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03342377"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1958824.1958865"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010134"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613715.1613751"},{"volume-title":"Advances in Dynamic Game Theory, Steffen Jorgensen, Marc Quincampoix, and Thomas L","author":"Szilagyi Miklos N.","key":"e_1_2_1_56_1","unstructured":"Miklos N. Szilagyi . 2007. Agent-based simulation of the n-person chicken game . In Advances in Dynamic Game Theory, Steffen Jorgensen, Marc Quincampoix, and Thomas L . Vincent (Eds.). Annals of the International Society of Dynamic Games, Vol. 9 . Birkh\u00e4user Boston , 696--703. DOI:http:\/\/dx.doi.org\/10.1007\/ 978-0-8176-4553-3_34 Miklos N. Szilagyi. 2007. Agent-based simulation of the n-person chicken game. In Advances in Dynamic Game Theory, Steffen Jorgensen, Marc Quincampoix, and Thomas L. Vincent (Eds.). Annals of the International Society of Dynamic Games, Vol. 9. Birkh\u00e4user Boston, 696--703. DOI:http:\/\/dx.doi.org\/10.1007\/ 978-0-8176-4553-3_34"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.196"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/1378704.1378719"},{"volume-title":"Proceedings of the Workshop on Crowdsourcing for Search and Data Mining (CSDM\u201911)","author":"Wang Jing","key":"e_1_2_1_59_1","unstructured":"Jing Wang , Siamak Faridani , and P. Ipeirotis . 2011. Estimating the completion time of crowdsourced tasks using survival analysis models . In Proceedings of the Workshop on Crowdsourcing for Search and Data Mining (CSDM\u201911) . Jing Wang, Siamak Faridani, and P. Ipeirotis. 2011. Estimating the completion time of crowdsourced tasks using survival analysis models. In Proceedings of the Workshop on Crowdsourcing for Search and Data Mining (CSDM\u201911)."},{"key":"e_1_2_1_60_1","volume-title":"Int. J. Sociol. Soc. Policy","author":"Wexler Mark N.","year":"2010","unstructured":"Mark N. Wexler . 2010. Reconfiguring the sociology of the crowd: Exploring crowdsourcing . Int. J. Sociol. Soc. Policy ( 2010 ), 6--20. Mark N. Wexler. 2010. Reconfiguring the sociology of the crowd: Exploring crowdsourcing. Int. J. Sociol. Soc. Policy (2010), 6--20."},{"key":"#cr-split#-e_1_2_1_61_1.1","doi-asserted-by":"crossref","unstructured":"Yuxiang Zhao and Qinghua Zhu. 2012. Evaluation on crowdsourcing research: Current status and future direction. Inform. Syst. Front. (2012) 1--18. DOI:http:\/\/dx.doi.org\/10.1007\/s10796-012-9350-4 10.1007\/s10796-012-9350-4","DOI":"10.1007\/s10796-012-9350-4"},{"key":"#cr-split#-e_1_2_1_61_1.2","doi-asserted-by":"crossref","unstructured":"Yuxiang Zhao and Qinghua Zhu. 2012. Evaluation on crowdsourcing research: Current status and future direction. Inform. Syst. Front. (2012) 1--18. DOI:http:\/\/dx.doi.org\/10.1007\/s10796-012-9350-4","DOI":"10.1007\/s10796-012-9350-4"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2873063","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2873063","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:53:49Z","timestamp":1750222429000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2873063"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,3,31]]},"references-count":62,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,7,14]]}},"alternative-id":["10.1145\/2873063"],"URL":"https:\/\/doi.org\/10.1145\/2873063","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"type":"print","value":"2157-6904"},{"type":"electronic","value":"2157-6912"}],"subject":[],"published":{"date-parts":[[2016,3,31]]},"assertion":[{"value":"2015-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-03-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}