{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:12:02Z","timestamp":1750306322241,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":39,"publisher":"ACM","license":[{"start":{"date-parts":[[2016,5,7]],"date-time":"2016-05-07T00:00:00Z","timestamp":1462579200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2016,5,7]]},"DOI":"10.1145\/2858036.2858199","type":"proceedings-article","created":{"date-parts":[[2016,5,5]],"date-time":"2016-05-05T14:08:22Z","timestamp":1462457302000},"page":"261-271","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Setwise Comparison"],"prefix":"10.1145","author":[{"given":"Advait","family":"Sarkar","sequence":"first","affiliation":[{"name":"Microsoft Research Cambridge & University of Cambridge, Cambridge, United Kingdom"}]},{"given":"Cecily","family":"Morrison","sequence":"additional","affiliation":[{"name":"Microsoft Research, Cambridge, United Kingdom"}]},{"given":"Jonas F.","family":"Dorn","sequence":"additional","affiliation":[{"name":"Novartis Pharma AG, Basel, Switzerland"}]},{"given":"Rishi","family":"Bedi","sequence":"additional","affiliation":[{"name":"Novartis Pharma AG & Stanford University, Basel, Switzerland"}]},{"given":"Saskia","family":"Steinheimer","sequence":"additional","affiliation":[{"name":"Inselspital, Bern University Hospital, Bern, Switzerland"}]},{"given":"Jacques","family":"Boisvert","sequence":"additional","affiliation":[{"name":"Novartis Pharma AG, Basel, Switzerland"}]},{"given":"Jessica","family":"Burggraaff","sequence":"additional","affiliation":[{"name":"VU University Medical Center, Amsterdam, Netherlands"}]},{"given":"Marcus","family":"D'Souza","sequence":"additional","affiliation":[{"name":"University Hospital Basel, Basel, Switzerland"}]},{"given":"Peter","family":"Kontschieder","sequence":"additional","affiliation":[{"name":"Microsoft Research, Cambridge, United Kingdom"}]},{"given":"Samuel","family":"Rota Bul\u00f2","sequence":"additional","affiliation":[{"name":"Microsoft Research, Cambridge, United Kingdom"}]},{"given":"Lorcan","family":"Walsh","sequence":"additional","affiliation":[{"name":"Norvatis Pharma AG, Basel, Switzerland"}]},{"given":"Christian P.","family":"Kamm","sequence":"additional","affiliation":[{"name":"University Hospital Bern, Bern, Switzerland"}]},{"given":"Yordan","family":"Zaykov","sequence":"additional","affiliation":[{"name":"Microsoft Research, Cambridge, United Kingdom"}]},{"given":"Abigail","family":"Sellen","sequence":"additional","affiliation":[{"name":"Microsoft Research, Cambridge, United Kingdom"}]},{"given":"Si\u00e2n","family":"Lindley","sequence":"additional","affiliation":[{"name":"Microsoft Research, Cambridge, United Kingdom"}]}],"member":"320","published-online":{"date-parts":[[2016,5,7]]},"reference":[{"volume-title":"Accessed: Friday 8th","year":"2016","key":"e_1_3_2_2_1_1","unstructured":"2016. TrueSkill Python Code. http:\/\/trueskill.org\/. (2016) . Accessed: Friday 8th January , 2016 . 2016. TrueSkill Python Code. http:\/\/trueskill.org\/. (2016). Accessed: Friday 8th January, 2016."},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1167\/2.1.5"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"crossref","unstructured":"S. Afzal and P. Robinson. 2014. Emotion Data Collection and Its Implications for Affective Computing. In The Oxford Handbook of Affective Computing. 359--369.  S. Afzal and P. Robinson. 2014. Emotion Data Collection and Its Implications for Affective Computing. In The Oxford Handbook of Affective Computing. 359--369.","DOI":"10.1093\/oxfordhb\/9780199942237.013.002"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995403"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/2900423.2900664"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526727"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"crossref","unstructured":"R. Bogacz E. Brown J. Moehlis P. Holmes and J.D. Cohen. 2006. The physics of optimal decision making: A formal analysis of models of performance in two-alternative forced-choice tasks. Psychological review 113 4 (2006) 700.  R. Bogacz E. Brown J. Moehlis P. Holmes and J.D. Cohen. 2006. The physics of optimal decision making: A formal analysis of models of performance in two-alternative forced-choice tasks. Psychological review 113 4 (2006) 700.","DOI":"10.1037\/0033-295X.113.4.700"},{"key":"e_1_3_2_2_8_1","first-page":"324","article-title":"Rank Analysis of Incomplete Block Designs","volume":"39","author":"Bradley RA","year":"1952","unstructured":"RA Bradley . 1952 . Rank Analysis of Incomplete Block Designs : The Method of Paired Comparisons. Biometrika 39 (1952), 324 -- 345 . RA Bradley. 1952. Rank Analysis of Incomplete Block Designs: The Method of Paired Comparisons. Biometrika 39 (1952), 324--345.","journal-title":"The Method of Paired Comparisons. Biometrika"},{"key":"e_1_3_2_2_9_1","volume-title":"Friedl","author":"Brodley Carla E.","year":"1999","unstructured":"Carla E. Brodley and Mark A . Friedl . 1999 . Identifying mislabeled training data. Journal of Artificial Intelligence Research ( 1999), 131--167. Carla E. Brodley and Mark A. Friedl. 1999. Identifying mislabeled training data. Journal of Artificial Intelligence Research (1999), 131--167."},{"key":"e_1_3_2_2_10_1","volume-title":"David Maxwell Chickering, and Susan T. Dumais","author":"Carterette Ben","year":"2008","unstructured":"Ben Carterette , Paul N. Bennett , David Maxwell Chickering, and Susan T. Dumais . 2008 . Here or there preference judgments for relevance. Lecture Notes in Computer Science 4956 LNCS ( 2008), 16--27. DOI: http:\/\/dx.doi.org\/10.1007\/978--3--540--78646--7{_}5 10.1007\/978--3--540--78646--7 Ben Carterette, Paul N. Bennett, David Maxwell Chickering, and Susan T. Dumais. 2008. Here or there preference judgments for relevance. Lecture Notes in Computer Science 4956 LNCS (2008), 16--27. DOI: http:\/\/dx.doi.org\/10.1007\/978--3--540--78646--7{_}5"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1474-4422(12)70059-5"},{"key":"e_1_3_2_2_12_1","unstructured":"R. Cowie S. Douglas-Cowie E. Savvidou E. McMahon M. Sawey and M. Schr\u00f6der. 2000. 'FEELTRACE': An instrument for recording perceived emotion in real time.. In ISCA tutorial and research workshop (ITRW) on speech and emotion.  R. Cowie S. Douglas-Cowie E. Savvidou E. McMahon M. Sawey and M. Schr\u00f6der. 2000. 'FEELTRACE': An instrument for recording perceived emotion in real time.. In ISCA tutorial and research workshop (ITRW) on speech and emotion."},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/604045.604056"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357061"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-89689-0_41"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2013.2292894"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.59"},{"key":"e_1_3_2_2_18_1","volume-title":"Staveland","author":"Hart Sandra G.","year":"1988","unstructured":"Sandra G. Hart and Lowell E . Staveland . 1988 . Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology 52 (1988), 139--183. Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology 52 (1988), 139--183."},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00422218"},{"key":"e_1_3_2_2_20_1","volume-title":"Advances in Neural Information Processing Systems (NIPS2006).","author":"Herbrich Ralf","year":"2006","unstructured":"Ralf Herbrich , Tom Minka , and Thore Graepel . TrueSkill(TM) : A Bayesian Skill Rating System . In Advances in Neural Information Processing Systems (NIPS2006). 2006 . Ralf Herbrich, Tom Minka, and Thore Graepel. TrueSkill(TM): A Bayesian Skill Rating System. In Advances in Neural Information Processing Systems (NIPS2006). 2006."},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-013-0306-1"},{"key":"e_1_3_2_2_22_1","volume-title":"Bernard MJ Uitdehaag, and Chris H Polman","author":"Kamm Christian P","year":"2014","unstructured":"Christian P Kamm , Bernard MJ Uitdehaag, and Chris H Polman . 2014 . Multiple sclerosis: current knowledge and future outlook. European neurology 72, 3--4 (2014), 132--141. Christian P Kamm, Bernard MJ Uitdehaag, and Chris H Polman. 2014. Multiple sclerosis: current knowledge and future outlook. European neurology 72, 3--4 (2014), 132--141."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10470-6_54"},{"key":"e_1_3_2_2_24_1","first-page":"249","article-title":"Supervised machine learning: A review of classification techniques","volume":"31","author":"Kotsiantis S. B.","year":"2007","unstructured":"S. B. Kotsiantis , I. Zaharakis , and P. Pintelas . 2007 . Supervised machine learning: A review of classification techniques . Informatica 31 (2007), 249 -- 268 . S. B. Kotsiantis, I. Zaharakis, and P. Pintelas. 2007. Supervised machine learning: A review of classification techniques. Informatica 31 (2007), 249--268.","journal-title":"Informatica"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557238"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1212\/WNL.33.11.1444"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2008.4587756"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647367"},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-68504-3_30"},{"key":"e_1_3_2_2_30_1","volume-title":"Forming inferences about some intraclass correlation coefficients. Psychological methods 1, 1","author":"McGraw Kenneth O","year":"1996","unstructured":"Kenneth O McGraw and Seok P Wong . 1996. Forming inferences about some intraclass correlation coefficients. Psychological methods 1, 1 ( 1996 ), 30. Kenneth O McGraw and Seok P Wong. 1996. Forming inferences about some intraclass correlation coefficients. Psychological methods 1, 1 (1996), 30."},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2011.20"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13735-012-0028-y"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1080\/07370024.2015.1093421"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1212\/WNL.40.6.971"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2015.7357211"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0070288"},{"key":"e_1_3_2_2_37_1","volume-title":"Accessed: Friday 8th","author":"Exel Job Van","year":"2005","unstructured":"Job Van Exel and Gjalt de Graaf . 2005 . Q methodology: A sneak preview. http:\/\/www.qmethodology.net\/PDF\/Q-methodology. (2005) . Accessed: Friday 8th January, 2016. Job Van Exel and Gjalt de Graaf. 2005. Q methodology: A sneak preview. http:\/\/www.qmethodology.net\/PDF\/Q-methodology. (2005). Accessed: Friday 8th January, 2016."},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-012-0564-1"},{"volume-title":"International conference on artificial intelligence and statistics. 932--939","author":"Yan Y.","key":"e_1_3_2_2_39_1","unstructured":"Y. Yan , R. Rosales , G. Fung , M. W. Schmidt , G. H. Valadez , L. Bogoni , L Moy , and J. G. Dy . 2010. Modeling annotator expertise: Learning when everybody knows a bit of something. (pp. 932--939) .. In International conference on artificial intelligence and statistics. 932--939 . Y. Yan, R. Rosales, G. Fung, M. W. Schmidt, G. H. Valadez, L. Bogoni, L Moy, and J. G. Dy. 2010. Modeling annotator expertise: Learning when everybody knows a bit of something. (pp. 932--939).. In International conference on artificial intelligence and statistics. 932--939."}],"event":{"name":"CHI'16: CHI Conference on Human Factors in Computing Systems","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"],"location":"San Jose California USA","acronym":"CHI'16"},"container-title":["Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2858036.2858199","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2858036.2858199","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:54:20Z","timestamp":1750222460000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2858036.2858199"}},"subtitle":["Consistent, Scalable, Continuum Labels for Computer Vision"],"short-title":[],"issued":{"date-parts":[[2016,5,7]]},"references-count":39,"alternative-id":["10.1145\/2858036.2858199","10.1145\/2858036"],"URL":"https:\/\/doi.org\/10.1145\/2858036.2858199","relation":{},"subject":[],"published":{"date-parts":[[2016,5,7]]},"assertion":[{"value":"2016-05-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}