{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,1]],"date-time":"2025-12-01T02:50:36Z","timestamp":1764557436596,"version":"3.41.0"},"reference-count":34,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2018,6,13]],"date-time":"2018-06-13T00:00:00Z","timestamp":1528848000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2018,6,30]]},"abstract":"<jats:p>When searching on the web or in an app, results are often returned as lists of hundreds to thousands of items, making it difficult for users to understand or navigate the space of results. Research has demonstrated that using clustering to partition search results into coherent, topical clusters can aid in both exploration and discovery. Yet clusters generated by an algorithm for this purpose are often of poor quality and do not satisfy users. To achieve acceptable clustered search results, experts must manually evaluate and refine the clustered results for each search query, a process that does not scale to large numbers of search queries. In this article, we investigate using crowd-based human evaluation to inspect, evaluate, and improve clusters to create high-quality clustered search results at scale. We introduce a workflow that begins by using a collection of well-known clustering algorithms to produce a set of clustered search results for a given query. Then, we use crowd workers to holistically assess the quality of each clustered search result to find the best one. Finally, the workflow has the crowd spot and fix problems in the best result to produce a final output. We evaluate this workflow on 120 top search queries from the Google Play Store, some of whom have clustered search results as a result of evaluations and refinements by experts. Our evaluations demonstrate that the workflow is effective at reproducing the evaluation of expert judges and also improves clusters in a way that agrees with experts and crowds alike.<\/jats:p>","DOI":"10.1145\/3158226","type":"journal-article","created":{"date-parts":[[2018,6,13]],"date-time":"2018-06-13T15:30:12Z","timestamp":1528903812000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Evaluation and Refinement of Clustered Search Results with the Crowd"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9462-9835","authenticated-orcid":false,"given":"Amy X.","family":"Zhang","sequence":"first","affiliation":[{"name":"Massachusetts Institute of Technology, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jilin","family":"Chen","sequence":"additional","affiliation":[{"name":"Google, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wei","family":"Chai","sequence":"additional","affiliation":[{"name":"Google, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jinjun","family":"Xu","sequence":"additional","affiliation":[{"name":"Google, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lichan","family":"Hong","sequence":"additional","affiliation":[{"name":"Google, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ed","family":"CHI","sequence":"additional","affiliation":[{"name":"Google, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,6,13]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531653"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"volume-title":"1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201913)","year":"2013","author":"Bragg Jonathan","key":"e_1_2_2_3_1"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1027527.1027747"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1541880.1541884"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858411"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2856767.2856783"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/332040.332418"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466265"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2013.212"},{"volume-title":"Proceedings of the International AAAI Conference on Weblogs and Social Media (ICWSM\u201912)","year":"2012","author":"Cranshaw Justin","key":"e_1_2_2_11_1"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/133160.133214"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.1979.4766909"},{"volume-title":"Proceedings of Workshop on Biomedical Data Mining, Modeling, and Semantic Integration at the 16th International Semantic Web Conference (ISWC\u201915)","year":"2015","author":"Dumitrache Anca","key":"e_1_2_2_14_1"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/568574.568575"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1062745.1062760"},{"volume-title":"Crowdclustering. In Proceedings of the 24th International Conference on Neural Information Processing Systems (NIPS\u201911)","year":"2011","author":"Gomes Ryan","key":"e_1_2_2_17_1"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/243199.243216"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-013-5413-0"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2017.03.007"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600159"},{"key":"e_1_2_2_22_1","doi-asserted-by":"crossref","unstructured":"Christopher D. Manning Prabhakar Raghavan Hinrich Sch\u00fctze etal 2008. Introduction to Information Retrieval Vol. 1. Cambridge University Press Cambridge.   Christopher D. Manning Prabhakar Raghavan Hinrich Sch\u00fctze et al. 2008. Introduction to Information Retrieval Vol. 1. Cambridge University Press Cambridge.","DOI":"10.1017\/CBO9780511809071"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1121949.1121979"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2005.38"},{"volume-title":"A Bayesian truth serum for subjective data. Science 306, 5695","year":"2004","author":"Prelec Dra\u017een","key":"e_1_2_2_25_1"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/0377-0427(87)90125-7"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2124295.2124324"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00042"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.5555\/1757448.1757479"},{"volume-title":"Proceedings of the 28th International Conference on International Conference on Machine Learning (ICML\u201911)","year":"2011","author":"Tamuz Omer","key":"e_1_2_2_30_1"},{"volume-title":"Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP\u201914)","author":"Verroios Vasilis","key":"e_1_2_2_31_1"},{"volume-title":"Proceedings of the 25th International Conference on Neural Information Processing Systems\u2014Volume 2 (NIPS\u201912)","year":"2012","author":"Yi Jinfeng","key":"e_1_2_2_32_1"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1389-1286(99)00054-7"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.2426"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3158226","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3158226","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:13:22Z","timestamp":1750212802000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3158226"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,6,13]]},"references-count":34,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2018,6,30]]}},"alternative-id":["10.1145\/3158226"],"URL":"https:\/\/doi.org\/10.1145\/3158226","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"type":"print","value":"2160-6455"},{"type":"electronic","value":"2160-6463"}],"subject":[],"published":{"date-parts":[[2018,6,13]]},"assertion":[{"value":"2016-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-06-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}