{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:34:09Z","timestamp":1750221249920,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":30,"publisher":"ACM","license":[{"start":{"date-parts":[[2018,6,27]],"date-time":"2018-06-27T00:00:00Z","timestamp":1530057600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Qatar National Research Fund","award":["NPRP 7-1313-1-245"],"award-info":[{"award-number":["NPRP 7-1313-1-245"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2018,6,27]]},"DOI":"10.1145\/3209978.3210033","type":"proceedings-article","created":{"date-parts":[[2018,7,2]],"date-time":"2018-07-02T12:12:40Z","timestamp":1530533560000},"page":"805-814","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Crowd vs. Expert"],"prefix":"10.1145","author":[{"given":"Mucahid","family":"Kutlu","sequence":"first","affiliation":[{"name":"Qatar University, Doha, Qatar"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tyler","family":"McDonnell","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, Austin, TX, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yassmine","family":"Barkallah","sequence":"additional","affiliation":[{"name":"Qatar University, Doha, Qatar"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tamer","family":"Elsayed","sequence":"additional","affiliation":[{"name":"Qatar University, Doha, Qatar"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matthew","family":"Lease","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, Austin, TX, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,6,27]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2637002.2637025"},{"volume-title":"Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","author":"Alonso Omar","key":"e_1_3_2_1_2_1","unstructured":"Omar Alonso and Stefano Mizzaro . 2009. Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , Vol. Vol. 15 . 16. Omar Alonso and Stefano Mizzaro . 2009. Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, Vol. Vol. 15. 16."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2012.01.004"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390447"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914671"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010039"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/PERCOMW.2011.5766944"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835540"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484161"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2012.95"},{"key":"e_1_3_2_1_11_1","volume-title":"Proceedings of the Twenty-Third NIST Text REtrieval Conference (TREC).","author":"Collins-Thompson Kevyn","year":"2015","unstructured":"Kevyn Collins-Thompson , Craig Macdonald , Paul Bennett , Fernando Diaz , and Ellen M Voorhees . 2015 . TREC 2014 Web Track Overview . In Proceedings of the Twenty-Third NIST Text REtrieval Conference (TREC). Kevyn Collins-Thompson, Craig Macdonald, Paul Bennett, Fernando Diaz, and Ellen M Voorhees . 2015. TREC 2014 Web Track Overview. In Proceedings of the Twenty-Third NIST Text REtrieval Conference (TREC)."},{"key":"e_1_3_2_1_12_1","volume-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics","author":"Dawid Alexander","year":"1979","unstructured":"Alexander Dawid and Allan Skene . 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics ( 1979 ), 20--28. Alexander Dawid and Allan Skene . 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics (1979), 20--28."},{"key":"e_1_3_2_1_13_1","volume-title":"6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP). 10 pages.","author":"Goyal Tanya","year":"2018","unstructured":"Tanya Goyal , Tyler McDonnell , Mucahid Kutlu , Tamer Elsayed , and Matthew Lease . 2018 . Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations . In 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP). 10 pages. Tanya Goyal, Tyler McDonnell, Mucahid Kutlu, Tamer Elsayed, and Matthew Lease . 2018. Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations. In 6th AAAI Conference on Human Computation and Crowdsourcing (HCOMP). 10 pages."},{"key":"e_1_3_2_1_14_1","volume-title":"Crowdsourcing Document Relevance Assessment with Mechanical Turk Proc. of the NAACL-HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk. 172--179","author":"Grady Catherine","year":"2010","unstructured":"Catherine Grady and Matthew Lease . 2010 . Crowdsourcing Document Relevance Assessment with Mechanical Turk Proc. of the NAACL-HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk. 172--179 . Catherine Grady and Matthew Lease . 2010. Crowdsourcing Document Relevance Assessment with Mechanical Turk Proc. of the NAACL-HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk. 172--179."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"crossref","first-page":"267","DOI":"10.58948\/2331-3528.1800","article-title":"Inconsistent responsiveness determination in document review: Difference of opinion or human error","volume":"32","author":"Grossman Maura R","year":"2012","unstructured":"Maura R Grossman and Gordon V Cormak . 2012 . Inconsistent responsiveness determination in document review: Difference of opinion or human error . Pace L. Rev. Vol. 32 (2012), 267 . Maura R Grossman and Gordon V Cormak . 2012. Inconsistent responsiveness determination in document review: Difference of opinion or human error. Pace L. Rev. Vol. 32 (2012), 267.","journal-title":"Pace L. Rev."},{"key":"e_1_3_2_1_16_1","volume-title":"EveTAR: building a large-scale multi-task test collection over Arabic tweets. Information Retrieval Journal (21 Dec","author":"Hasanain Maram","year":"2017","unstructured":"Maram Hasanain , Reem Suwaileh , Tamer Elsayed , Mucahid Kutlu , and Hind Almerekhi . 2017. EveTAR: building a large-scale multi-task test collection over Arabic tweets. Information Retrieval Journal (21 Dec . 2017 ). Maram Hasanain, Reem Suwaileh, Tamer Elsayed, Mucahid Kutlu, and Hind Almerekhi . 2017. EveTAR: building a large-scale multi-task test collection over Arabic tweets. Information Retrieval Journal (21 Dec . 2017)."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741102"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2396779"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458160"},{"key":"e_1_3_2_1_20_1","volume-title":"The Many Benefits of Annotator Rationales for Relevance Judgments Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)","author":"McDonnell Tyler","year":"2017","unstructured":"Tyler McDonnell , Mucahid Kutlu , Tamer Elsayed , and Matthew Lease . 2017 . The Many Benefits of Annotator Rationales for Relevance Judgments Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17) . AAAI, 4909--4913. Tyler McDonnell, Mucahid Kutlu, Tamer Elsayed, and Matthew Lease . 2017. The Many Benefits of Annotator Rationales for Relevance Judgments Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17). AAAI, 4909--4913."},{"key":"e_1_3_2_1_21_1","volume-title":"Collecting Annotator Rationales for Relevance Judgments Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP). AAAI, 139--148","author":"McDonnell Tyler","year":"2016","unstructured":"Tyler McDonnell , Matthew Lease , Mucahid Kutlu , and Tamer Elsayed . 2016 . Why is That Relevant? Collecting Annotator Rationales for Relevance Judgments Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP). AAAI, 139--148 . Best Paper Award. Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed . 2016. Why is That Relevant? Collecting Annotator Rationales for Relevance Judgments Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP). AAAI, 139--148. Best Paper Award."},{"key":"e_1_3_2_1_22_1","volume-title":"Proceedings of CrowdConf.","author":"Paiement Jean-Francois","year":"2010","unstructured":"Jean-Francois Paiement , James G Shanahan , and Remi Zajac . 2010 . Crowdsourcing local search relevance . In Proceedings of CrowdConf. Jean-Francois Paiement, James G Shanahan, and Remi Zajac . 2010. Crowdsourcing local search relevance. In Proceedings of CrowdConf."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484090"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010057"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564433"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(00)00010-8"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2854946.2854968"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.v57:7"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390435"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609577"}],"event":{"name":"SIGIR '18: The 41st International ACM SIGIR conference on research and development in Information Retrieval","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"],"location":"Ann Arbor MI USA","acronym":"SIGIR '18"},"container-title":["The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3209978.3210033","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3209978.3210033","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:07:49Z","timestamp":1750212469000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3209978.3210033"}},"subtitle":["What Can Relevance Judgment Rationales Teach Us About Assessor Disagreement?"],"short-title":[],"issued":{"date-parts":[[2018,6,27]]},"references-count":30,"alternative-id":["10.1145\/3209978.3210033","10.1145\/3209978"],"URL":"https:\/\/doi.org\/10.1145\/3209978.3210033","relation":{},"subject":[],"published":{"date-parts":[[2018,6,27]]},"assertion":[{"value":"2018-06-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}