{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:09:20Z","timestamp":1750306160498,"version":"3.41.0"},"reference-count":38,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2016,11,15]],"date-time":"2016-11-15T00:00:00Z","timestamp":1479168000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-sa\/4.0\/"}],"funder":[{"name":"European Union's Seventh Framework Programme for research, technological development and demonstration","award":["316097[ENGINE]"],"award-info":[{"award-number":["316097[ENGINE]"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Web"],"published-print":{"date-parts":[[2016,12,27]]},"abstract":"<jats:p>Machine learning algorithms and recommender systems trained on human ratings are widely in use today. However, human ratings may be associated with a high level of uncertainty and are subjective, influenced by demographic or psychological factors. We propose a new approach to the design of object classes from human ratings: the use of entire distributions to construct classes. By avoiding aggregation for class definition, our approach loses no information and can deal with highly volatile or conflicting ratings. The approach is based the concept of the Earth Mover's Distance (EMD), a measure of distance for distributions. We evaluate the proposed approach based on four datasets obtained from diverse Web content or movie quality evaluation services or experiments. We show that clusters discovered in these datasets using the EMD measure are characterized by a consistent and simple interpretation. Quality classes defined using entire rating distributions can be fitted to clusters of distributions in the four datasets using two parameters, resulting in a good overall fit. We also consider the impact of the composition of small samples on the distributions that are the basis of our classification approach. We show that using distributions based on small samples of 10 evaluations is still robust to several demographic and psychological variables. This observation suggests that the proposed approach can be used in practice for quality evaluation, even for highly uncertain and subjective ratings.<\/jats:p>","DOI":"10.1145\/2994132","type":"journal-article","created":{"date-parts":[[2016,11,17]],"date-time":"2016-11-17T16:33:35Z","timestamp":1479400415000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Web Content Classification Using Distributions of Subjective Quality Evaluations"],"prefix":"10.1145","volume":"10","author":[{"given":"Maria","family":"Rafalak","sequence":"first","affiliation":[{"name":"Polish-Japanese Academy of Information Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dominik","family":"Deja","sequence":"additional","affiliation":[{"name":"Polish-Japanese Academy of Information Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Adam","family":"Wierzbicki","sequence":"additional","affiliation":[{"name":"Wroc\u0142aw University of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rados\u0142aw","family":"Nielek","sequence":"additional","affiliation":[{"name":"Polish-Japanese Academy of Information Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Micha\u0142","family":"K\u0105kol","sequence":"additional","affiliation":[{"name":"Polish-Japanese Academy of Information Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,11,15]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2013.03.012"},{"key":"e_1_2_1_2_1","volume-title":"Skene","author":"Dawid Alexander P.","year":"1979","unstructured":"Alexander P. Dawid and Allan M . Skene . 1979 . Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics . (1979), 20--28. Alexander P. Dawid and Allan M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics. (1979), 20--28."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-011-9181-9"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567972"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2013.2292894"},{"volume-title":"The Elements of Statistical Learning","author":"Friedman Jerome","key":"e_1_2_1_6_1","unstructured":"Jerome Friedman , Trevor Hastie , and Robert Tibshirani . 2001. The Elements of Statistical Learning . Springer , Berlin . Jerome Friedman, Trevor Hastie, and Robert Tibshirani. 2001. The Elements of Statistical Learning. Springer, Berlin."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1177\/0894439310397146"},{"key":"e_1_2_1_8_1","unstructured":"Dirk Hovy Taylor Berg-Kirkpatrick Ashish Vaswani and Eduard Hovy. 2013. Learning whom to trust with MACE. In HLT-NAACL 1120--1130.  Dirk Hovy Taylor Berg-Kirkpatrick Ashish Vaswani and Eduard Hovy. 2013. Learning whom to trust with MACE. In HLT-NAACL 1120--1130."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1562764.1562800"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-013-0306-1"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2929.2004.02012.x"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2639968.2640074"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487788.2488133"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/CSE.2009.491"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10044-002-0173-7"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.2307\/1388242"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2015.07.010"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btl346"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/11944874_33"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187863"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1743384.1743478"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-03260-3_36"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-36973-5_47"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2567948.2578997"},{"volume-title":"Analysis of demographical factors","author":"Rafalak Maria","key":"e_1_2_1_26_1","unstructured":"Maria Rafalak , Piotr Bilski , and Adam Wierzbicki . 2014b. Analysis of demographical factors \u2019 influence on websites\u2019 credibility evaluation. Human-Computer Interaction. Applications and Services. Springer International Publishing , 57--68. Maria Rafalak, Piotr Bilski, and Adam Wierzbicki. 2014b. Analysis of demographical factors\u2019 influence on websites\u2019 credibility evaluation. Human-Computer Interaction. Applications and Services. Springer International Publishing, 57--68."},{"key":"e_1_2_1_27_1","first-page":"491","article-title":"Eliminating spammers and ranking annotators for crowdsourced labeling tasks","volume":"13","author":"Raykar Vikas C.","year":"2012","unstructured":"Vikas C. Raykar and Shipeng Yu . 2012 . Eliminating spammers and ranking annotators for crowdsourced labeling tasks . Journal of Machine Learning Research 13 , 491 -- 518 . Vikas C. Raykar and Shipeng Yu. 2012. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. Journal of Machine Learning Research 13, 491--518.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1026543900054"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2011.08.020"},{"key":"e_1_2_1_30_1","unstructured":"Padhraic Smyth Usama Fayyad Michael Burl Pietro Perona and Pierre Baldi. 1995. Inferring ground truth from subjective labelling of Venus images. In Advances in Neural Information Processing Systems. 1085--1092.   Padhraic Smyth Usama Fayyad Michael Burl Pietro Perona and Pierre Baldi. 1995. Inferring ground truth from subjective labelling of Venus images. In Advances in Neural Information Processing Systems. 1085--1092."},{"volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistic, 254--263","author":"Snow Rion","key":"e_1_2_1_31_1","unstructured":"Rion Snow , Brendan O\u2019Connor , Daniel Jurafsky , and Andrew Y. Ng . 2008. Cheap and fast\u2014but is it good?: Evaluating non-expert annotations for natural language tasks . In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistic, 254--263 . Rion Snow, Brendan O\u2019Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast\u2014but is it good?: Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistic, 254--263."},{"key":"e_1_2_1_32_1","first-page":"820","article-title":"Utility data annotation with Amazon Mechanical Turk","volume":"51","author":"Sorokin Alexander","year":"2008","unstructured":"Alexander Sorokin and David Forsyth . 2008 . Utility data annotation with Amazon Mechanical Turk . Urbana 51 , 61(2008), 820 . Alexander Sorokin and David Forsyth. 2008. Utility data annotation with Amazon Mechanical Turk. Urbana 51, 61(2008), 820.","journal-title":"Urbana"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijar.2006.06.024"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010374114305"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2567948.2579000"},{"key":"e_1_2_1_36_1","unstructured":"Peter Welinder Steve Branson Serge Belongie and Pietro Perona. 2010. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems. 2424--2432.   Peter Welinder Steve Branson Serge Belongie and Pietro Perona. 2010. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems. 2424--2432."},{"key":"e_1_2_1_37_1","unstructured":"Jacob Whitehill Paul Ruvolo Tingfan Wu Jacob Bergsma and Javier Movellan. 2009. Whose evaluation should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems. 2035--2043.   Jacob Whitehill Paul Ruvolo Tingfan Wu Jacob Bergsma and Javier Movellan. 2009. Whose evaluation should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems. 2035--2043."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487600"}],"container-title":["ACM Transactions on the Web"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2994132","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2994132","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:39:50Z","timestamp":1750217990000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2994132"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,11,15]]},"references-count":38,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,12,27]]}},"alternative-id":["10.1145\/2994132"],"URL":"https:\/\/doi.org\/10.1145\/2994132","relation":{},"ISSN":["1559-1131","1559-114X"],"issn-type":[{"type":"print","value":"1559-1131"},{"type":"electronic","value":"1559-114X"}],"subject":[],"published":{"date-parts":[[2016,11,15]]},"assertion":[{"value":"2015-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-11-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}