{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T21:32:49Z","timestamp":1776115969565,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":39,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,1,20]],"date-time":"2020-01-20T00:00:00Z","timestamp":1579478400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Australian Research Council (ARC) Discovery Project","award":["DP190102141"],"award-info":[{"award-number":["DP190102141"]}]},{"name":"Erasmus+ project DISKOW","award":["60171990"],"award-info":[{"award-number":["60171990"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,1,20]]},"DOI":"10.1145\/3336191.3371857","type":"proceedings-article","created":{"date-parts":[[2020,1,22]],"date-time":"2020-01-22T19:08:16Z","timestamp":1579720096000},"page":"241-249","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":29,"title":["Crowd Worker Strategies in Relevance Judgment Tasks"],"prefix":"10.1145","author":[{"given":"Lei","family":"Han","sequence":"first","affiliation":[{"name":"University of Queensland, Brisbane, Australia"}]},{"given":"Eddy","family":"Maddalena","sequence":"additional","affiliation":[{"name":"University of Southampton, Southampton, United Kingdom"}]},{"given":"Alessandro","family":"Checco","sequence":"additional","affiliation":[{"name":"University of Sheffield, Sheffield, United Kingdom"}]},{"given":"Cristina","family":"Sarasua","sequence":"additional","affiliation":[{"name":"University of Zurich, Zurich, Switzerland"}]},{"given":"Ujwal","family":"Gadiraju","sequence":"additional","affiliation":[{"name":"L3S Research Center, Hannover, Germany"}]},{"given":"Kevin","family":"Roitero","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, Italy"}]},{"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[{"name":"University of Queensland, Brisbane, Australia"}]}],"member":"320","published-online":{"date-parts":[[2020,1,22]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Design and implementation of relevance assessments using crowdsourcing","author":"Alonso Omar","unstructured":"Omar Alonso and Ricardo Baeza-Yates . 2011. Design and implementation of relevance assessments using crowdsourcing . In ECIR. Springer , 153--164. Omar Alonso and Ricardo Baeza-Yates. 2011. Design and implementation of relevance assessments using crowdsourcing. In ECIR. Springer, 153--164."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2012.01.004"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_3_2_1_4_1","volume-title":"Using thematic analysis in psychology. Qualitative research in psychology","author":"Braun Virginia","year":"2006","unstructured":"Virginia Braun and Victoria Clarke . 2006. Using thematic analysis in psychology. Qualitative research in psychology , Vol. 3 , 2 ( 2006 ), 77--101. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology , Vol. 3, 2 (2006), 77--101."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1561\/1800000025"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159661"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741685"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3201064.3201074"},{"key":"e_1_3_2_1_9_1","volume-title":"mbox","author":"Ujwal Gadiraju","year":"2018","unstructured":"Ujwal Gadiraju et al mbox . 2018 . Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection. CSCW ( 2018), 1--27. Ujwal Gadiraju et almbox. 2018. Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection. CSCW (2018), 1--27."},{"key":"e_1_3_2_1_10_1","first-page":"49","article-title":"Modus operandi of crowd workers: The invisible role of microtask work environments","volume":"1","author":"Gadiraju Ujwal","year":"2017","unstructured":"Ujwal Gadiraju , Alessandro Checco , Neha Gupta , and Gianluca Demartini . 2017 . Modus operandi of crowd workers: The invisible role of microtask work environments . Proceedings of IMWUT , Vol. 1 , 3 (2017), 49 . Ujwal Gadiraju, Alessandro Checco, Neha Gupta, and Gianluca Demartini. 2017. Modus operandi of crowd workers: The invisible role of microtask work environments. Proceedings of IMWUT , Vol. 1, 3 (2017), 49.","journal-title":"Proceedings of IMWUT"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027385.3027402"},{"key":"e_1_3_2_1_12_1","volume-title":"HCOMP","author":"Goyal Tanya","year":"2018","unstructured":"Tanya Goyal , Tyler McDonnell , Mucahid Kutlu , Tamer Elsayed , and Matthew Lease . 2018 . Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations . In HCOMP 2018. 41--49. Tanya Goyal, Tyler McDonnell, Mucahid Kutlu, Tamer Elsayed, and Matthew Lease. 2018. Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations. In HCOMP 2018. 41--49."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291035"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300637"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470742"},{"key":"e_1_3_2_1_16_1","volume-title":"TOIS","volume":"20","author":"Jaana Kalervo","year":"2002","unstructured":"Kalervo J\"arvelin and Jaana Kek\"al\"ainen. 2002 . Cumulated gain-based evaluation of IR techniques . TOIS , Vol. 20 , 4 (2002), 422--446. Kalervo J\"arvelin and Jaana Kek\"al\"ainen. 2002. Cumulated gain-based evaluation of IR techniques. TOIS , Vol. 20, 4 (2002), 422--446."},{"key":"e_1_3_2_1_17_1","volume-title":"HCOMP","author":"Kaplan Toni","year":"2018","unstructured":"Toni Kaplan , Susumu Saito , Kotaro Hara , and Jeffrey P Bigham . 2018 . Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers .. In HCOMP 2018. 70--78. Toni Kaplan, Susumu Saito, Kotaro Hara, and Jeffrey P Bigham. 2018. Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers.. In HCOMP 2018. 70--78."},{"key":"e_1_3_2_1_18_1","first-page":"1","article-title":"More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk","volume":"11","author":"Kaufmann Nicolas","year":"2011","unstructured":"Nicolas Kaufmann , Thimo Schulze , and Daniel Veit . 2011 . More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk .. In AMCIS , Vol. 11. 1 -- 11 . Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk.. In AMCIS, Vol. 11. 1--11.","journal-title":"AMCIS"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063860"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398697"},{"key":"e_1_3_2_1_21_1","volume-title":"Content analysis: An introduction to its methodology","author":"Kirppendorff K","year":"1989","unstructured":"K Kirppendorff . 1989. Content analysis: An introduction to its methodology . Beverley Hills : Sage ( 1989 ). K Kirppendorff. 1989. Content analysis: An introduction to its methodology. Beverley Hills: Sage (1989)."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_3_2_1_23_1","volume-title":"HCOMP 2016 .","author":"Maddalena Eddy","year":"2016","unstructured":"Eddy Maddalena , Marco Basaldella , Dario De Nart , Dante Degl'Innocenti , Stefano Mizzaro , and Gianluca Demartini . 2016 . Crowdsourcing Relevance Assessments: The Unexpected Benefits of Limiting the Time to Judge . In HCOMP 2016 . Eddy Maddalena, Marco Basaldella, Dario De Nart, Dante Degl'Innocenti, Stefano Mizzaro, and Gianluca Demartini. 2016. Crowdsourcing Relevance Assessments: The Unexpected Benefits of Limiting the Time to Judge. In HCOMP 2016 ."},{"key":"e_1_3_2_1_24_1","volume-title":"On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics","author":"Mann Henry B","year":"1947","unstructured":"Henry B Mann and Donald R Whitney . 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics ( 1947 ), 50--60. Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50--60."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2464464.2464485"},{"key":"e_1_3_2_1_26_1","volume-title":"Collecting Annotator Rationales for Relevance Judgments. In HCOMP","author":"McDonnell Tyler","year":"2016","unstructured":"Tyler McDonnell , Matthew Lease , Mucahid Kutlu , and Tamer Elsayed . 2016 . Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments. In HCOMP 2016. 139--148. Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments. In HCOMP 2016. 139--148."},{"key":"e_1_3_2_1_27_1","first-page":"46","article-title":"Pattern-matching-the gestalt approach","volume":"13","author":"Ratcliff John W","year":"1988","unstructured":"John W Ratcliff and David E Metzener . 1988 . Pattern-matching-the gestalt approach . Dr Dobbs Journal , Vol. 13 , 7 (1988), 46 . John W Ratcliff and David E Metzener. 1988. Pattern-matching-the gestalt approach. Dr Dobbs Journal , Vol. 13, 7 (1988), 46.","journal-title":"Dr Dobbs Journal"},{"key":"e_1_3_2_1_28_1","first-page":"17","article-title":"An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets","volume":"11","author":"Rogstadius Jakob","year":"2011","unstructured":"Jakob Rogstadius , Vassilis Kostakos , Aniket Kittur , Boris Smus , Jim Laredo , and Maja Vukovic . 2011 . An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets . ICWSM , Vol. 11 (2011), 17 -- 21 . Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. ICWSM , Vol. 11 (2011), 17--21.","journal-title":"ICWSM"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753846.1753873"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380125"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047199"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/274644.274680"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564433"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767760"},{"key":"e_1_3_2_1_35_1","volume-title":"Process Mining","author":"Van der Aalst Wil MP","unstructured":"Wil MP Van der Aalst . 2011. Process discovery: An introduction . In Process Mining . Springer , 125--156. Wil MP Van der Aalst. 2011. Process discovery: An introduction. In Process Mining . Springer, 125--156."},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.SP.500-242"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1645967"},{"key":"e_1_3_2_1_38_1","volume-title":"HCOMP 2018 .","author":"Yang Jie","unstructured":"Jie Yang , Carlo van der Valk, Tobias Ho\u00dffeld, Judith Redi, and Alessandro Bozzon. 2018. How Do Crowdworker Communities and Microtask Markets Influence Each Other? A Data-Driven Study on Amazon Mechanical Turk . In HCOMP 2018 . Jie Yang, Carlo van der Valk, Tobias Ho\u00dffeld, Judith Redi, and Alessandro Bozzon. 2018. How Do Crowdworker Communities and Microtask Markets Influence Each Other? A Data-Driven Study on Amazon Mechanical Turk. In HCOMP 2018 ."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883036"}],"event":{"name":"WSDM '20: The Thirteenth ACM International Conference on Web Search and Data Mining","location":"Houston TX USA","acronym":"WSDM '20","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 13th International Conference on Web Search and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3336191.3371857","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3336191.3371857","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:23:14Z","timestamp":1750202594000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3336191.3371857"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,20]]},"references-count":39,"alternative-id":["10.1145\/3336191.3371857","10.1145\/3336191"],"URL":"https:\/\/doi.org\/10.1145\/3336191.3371857","relation":{},"subject":[],"published":{"date-parts":[[2020,1,20]]},"assertion":[{"value":"2020-01-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}