{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,1]],"date-time":"2025-08-01T03:58:23Z","timestamp":1754020703842,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":30,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,3,14]],"date-time":"2020-03-14T00:00:00Z","timestamp":1584144000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,3,14]]},"DOI":"10.1145\/3343413.3377998","type":"proceedings-article","created":{"date-parts":[[2020,3,12]],"date-time":"2020-03-12T18:39:33Z","timestamp":1584038373000},"page":"363-367","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Collecting High-Quality Dialogue User Satisfaction Ratings with Third-Party Annotators"],"prefix":"10.1145","author":[{"given":"Mickey","family":"van Zeelt","sequence":"first","affiliation":[{"name":"ING Group N.V., Amsterdam, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Floris","family":"den Hengst","sequence":"additional","affiliation":[{"name":"ING Group N.V., Amsterdam, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Seyyed Hadi","family":"Hashemi","sequence":"additional","affiliation":[{"name":"University of Amsterdam, Amsterdam, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,3,14]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.21437\/Interspeech.2017-725"},{"volume-title":"Frames: A Corpus for Adding Memory to Goal-Oriented Dialogue Systems . arXiv:1704.00057 [cs] (March","year":"2017","author":"Asri Layla El","key":"e_1_3_2_1_2_1"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691610393980"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2015.2493525"},{"volume-title":"A coefficient of agreement for nominal scales. Educational and psychological measurement","year":"1960","author":"Cohen Jacob","key":"e_1_3_2_1_5_1"},{"volume-title":"Reinforcement Learning for Personalized Dialogue Management. In IEEE\/WIC\/ACM International Conference on Web Intelligence. ACM, 59--67","year":"2019","author":"den Hengst Floris","key":"e_1_3_2_1_6_1"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2499149.2499168"},{"volume-title":"Measuring nominal scale agreement among many raters. Psychological bulletin","year":"1971","author":"Fleiss Joseph L","key":"e_1_3_2_1_8_1"},{"key":"e_1_3_2_1_9_1","unstructured":"Riste Gligorov. 2010. Towards Integration of End-User Tags with Professional Annotations . http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.188.3495&rep=rep1&type=pdf  Riste Gligorov. 2010. Towards Integration of End-User Tags with Professional Annotations . http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.188.3495&rep=rep1&type=pdf"},{"volume-title":"Proceedings of the 27th ACM International Conference on Information and Knowledge Management - CIKM '18. ACM Press","year":"1802","author":"Hashemi Seyyed Hadi","key":"e_1_3_2_1_10_1"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1080\/19312450709336664"},{"key":"e_1_3_2_1_12_1","unstructured":"Ryuichiro Higashinaka Kotaro Funakoshi Yuka Kobayashi and Michimasa Inaba. 2016. The dialogue breakdown detection challenge: Task description datasets and evaluation metrics. In LREC .  Ryuichiro Higashinaka Kotaro Funakoshi Yuka Kobayashi and Michimasa Inaba. 2016. The dialogue breakdown detection challenge: Task description datasets and evaluation metrics. In LREC ."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2018.09.004"},{"key":"e_1_3_2_1_14_1","unstructured":"Gabriella Kazai. 2011. In Search of Quality in Crowdsourcing for Search Engine Evaluation . In Advances in Information Retrieval Paul Clough Colum Foley Cathal Gurrin Gareth J. F. Jones Wessel Kraaij Hyowon Lee and Vanessa Mudoch (Eds.). Vol. 6611. Springer Berlin Heidelberg Berlin Heidelberg 165--176. https:\/\/doi.org\/10.1007\/978--3--642--20161--5_17  Gabriella Kazai. 2011. In Search of Quality in Crowdsourcing for Search Engine Evaluation . In Advances in Information Retrieval Paul Clough Colum Foley Cathal Gurrin Gareth J. F. Jones Wessel Kraaij Hyowon Lee and Vanessa Mudoch (Eds.). Vol. 6611. Springer Berlin Heidelberg Berlin Heidelberg 165--176. https:\/\/doi.org\/10.1007\/978--3--642--20161--5_17"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000012"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_3_2_1_17_1","unstructured":"John Le Andy Edmonds Vaughn Hester and Lukas Biewald. 2011. Ensuring quality in crowdsourced search relevance evaluation : The effects of training question distribution.  John Le Andy Edmonds Vaughn Hester and Lukas Biewald. 2011. Ensuring quality in crowdsourced search relevance evaluation : The effects of training question distribution."},{"volume-title":"Computer Vision -- ECCV 2014 , , David Fleet, Tomas Pajdla","author":"Lin Tsung-Yi","key":"e_1_3_2_1_18_1"},{"volume-title":"Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '09)","author":"Mason Winter","key":"e_1_3_2_1_19_1"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/505282.505285"},{"volume-title":"On the Role of User-generated Metadata in Audio Visual Collections. ACM","author":"Mark A Musen and Association for Computing Machinery. 2011.","key":"e_1_3_2_1_21_1"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0378-7206(03)00089-2"},{"volume-title":"Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)","year":"2006","author":"Passonneau Rebecca","key":"e_1_3_2_1_23_1"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Esteban Real Jonathon Shlens Stefano Mazzocchi Xin Pan and Vincent Vanhoucke. 2017. YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video . (2017) 5296--5305.  Esteban Real Jonathon Shlens Stefano Mazzocchi Xin Pan and Vincent Vanhoucke. 2017. YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video . (2017) 5296--5305.","DOI":"10.1109\/CVPR.2017.789"},{"volume-title":"Modeling and Predicting Quality in Spoken Human-Computer Interaction. In SIGDIAL Conference .","year":"2011","author":"Schmitt Alexander","key":"e_1_3_2_1_25_1"},{"key":"e_1_3_2_1_26_1","unstructured":"Mohammad Soleymani. 2010. Crowdsourcing for Affective Annotation of Video: Development of a Viewer-reported Boredom Corpus .  Mohammad Soleymani. 2010. Crowdsourcing for Affective Annotation of Video: Development of a Viewer-reported Boredom Corpus ."},{"volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . 569--578","year":"2013","author":"Ultes Stefan","key":"e_1_3_2_1_27_1"},{"volume-title":"Department of Parks","year":"2006","author":"Vagias Wade M","key":"e_1_3_2_1_28_1"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/SLT.2010.5700864"},{"volume-title":"Measuring inter-rater reliability for nominal data--which coefficients and confidence intervals are appropriate? BMC medical research methodology","year":"2016","author":"Zapf Antonia","key":"e_1_3_2_1_30_1"}],"event":{"name":"CHIIR '20: Conference on Human Information Interaction and Retrieval","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval","SIGCHI ACM Special Interest Group on Computer-Human Interaction"],"location":"Vancouver BC Canada","acronym":"CHIIR '20"},"container-title":["Proceedings of the 2020 Conference on Human Information Interaction and Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3343413.3377998","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3343413.3377998","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:13:07Z","timestamp":1750201987000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3343413.3377998"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,14]]},"references-count":30,"alternative-id":["10.1145\/3343413.3377998","10.1145\/3343413"],"URL":"https:\/\/doi.org\/10.1145\/3343413.3377998","relation":{},"subject":[],"published":{"date-parts":[[2020,3,14]]},"assertion":[{"value":"2020-03-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}