{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T18:15:01Z","timestamp":1775326501396,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":32,"publisher":"ACM","license":[{"start":{"date-parts":[[2017,11,28]],"date-time":"2017-11-28T00:00:00Z","timestamp":1511827200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2017,11,28]]},"DOI":"10.1145\/3152771.3156146","type":"proceedings-article","created":{"date-parts":[[2017,12,20]],"date-time":"2017-12-20T14:54:02Z","timestamp":1513781642000},"page":"407-411","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Evaluating crowdsourced relevance assessments using self-reported traits and task speed"],"prefix":"10.1145","author":[{"given":"Christopher","family":"Chow","sequence":"first","affiliation":[{"name":"The Australian National University, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tom","family":"Gedeon","sequence":"additional","affiliation":[{"name":"The Australian National University, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,11,28]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1631272.1631336"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CogInfoCom.2015.7390631"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1924421.1924442"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511816796"},{"key":"e_1_3_2_1_6_1","volume-title":"Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5), 378","author":"Fleiss J. L.","year":"1971","unstructured":"Fleiss , J. L. ( 1971 ). Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5), 378 . Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5), 378."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1004932107"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187914"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCA.2002.802821"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398697"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_3_2_1_13_1","volume-title":"The measurement of observer agreement for categorical data. biometrics, 159--174","author":"Landis J. R.","year":"1977","unstructured":"Landis , J. R. , & Koch , G. G. ( 1977 ). The measurement of observer agreement for categorical data. biometrics, 159--174 . Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. biometrics, 159--174."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998197"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10639-013-9263-y"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2007.06.015"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v4i1.13284"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/1394399"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2873063"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1081870.1081899"},{"key":"e_1_3_2_1_21_1","volume-title":"Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks. arXiv preprint arXiv:1707.01836","author":"Rajpurkar P.","year":"2017","unstructured":"Rajpurkar , P. , Hannun , A. Y. , Haghpanahi , M. , Bourn , C. , & Ng , A. Y. ( 2017 ). Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks. arXiv preprint arXiv:1707.01836 . Rajpurkar, P., Hannun, A. Y., Haghpanahi, M., Bourn, C., & Ng, A. Y. (2017). Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks. arXiv preprint arXiv:1707.01836."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(92)90004-J"},{"key":"e_1_3_2_1_23_1","first-page":"261","volume-title":"Proceedings of WSOM (Vol. 3","author":"Saloj\u00e4rvi J.","year":"2003","unstructured":"Saloj\u00e4rvi , J. , Kojo , I. , Simola , J. , & Kaski , S. ( 2003 , September). Can relevance be inferred from eye movements in information retrieval . In Proceedings of WSOM (Vol. 3 , pp. 261 -- 266 ). Saloj\u00e4rvi, J., Kojo, I., Simola, J., & Kaski, S. (2003, September). Can relevance be inferred from eye movements in information retrieval. In Proceedings of WSOM (Vol. 3, pp. 261--266)."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0377-2217(01)00113-8"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613715.1613751"},{"issue":"5","key":"e_1_3_2_1_26_1","first-page":"360","article-title":"Understanding interobserver agreement: the kappa statistic","volume":"37","author":"Viera A. J.","year":"2005","unstructured":"Viera , A. J. , & Garrett , J. M. ( 2005 ). Understanding interobserver agreement: the kappa statistic . Fam Med , 37 ( 5 ), 360 -- 363 . Viera, A. J., & Garrett, J. M. (2005). Understanding interobserver agreement: the kappa statistic. Fam Med, 37(5), 360--363.","journal-title":"Fam Med"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2449396.2449398"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998234"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.3115\/1034678.1034721"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.v59:8"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819953"},{"key":"e_1_3_2_1_32_1","volume-title":"Risk and Trust (PASSAT) and 2011 IEEE Third Inernational Conference on Social Computing (SocialCom), 2011 IEEE Third International Conference on (pp. 766--773)","author":"Yuen M. C.","year":"2011","unstructured":"Yuen , M. C. , King , I. , & Leung , K. S. ( 2011 , October). A survey of crowdsourcing systems. In Privacy, Security , Risk and Trust (PASSAT) and 2011 IEEE Third Inernational Conference on Social Computing (SocialCom), 2011 IEEE Third International Conference on (pp. 766--773) . IEEE. Yuen, M. C., King, I., & Leung, K. S. (2011, October). A survey of crowdsourcing systems. In Privacy, Security, Risk and Trust (PASSAT) and 2011 IEEE Third Inernational Conference on Social Computing (SocialCom), 2011 IEEE Third International Conference on (pp. 766--773). IEEE."}],"event":{"name":"OzCHI '17: 29th Australian Conference on Human-Computer Interaction","location":"Brisbane Queensland Australia","acronym":"OzCHI '17"},"container-title":["Proceedings of the 29th Australian Conference on Computer-Human Interaction"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3152771.3156146","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3152771.3156146","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:11:10Z","timestamp":1750212670000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3152771.3156146"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,11,28]]},"references-count":32,"alternative-id":["10.1145\/3152771.3156146","10.1145\/3152771"],"URL":"https:\/\/doi.org\/10.1145\/3152771.3156146","relation":{},"subject":[],"published":{"date-parts":[[2017,11,28]]},"assertion":[{"value":"2017-11-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}