{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,27]],"date-time":"2025-09-27T10:55:17Z","timestamp":1758970517583,"version":"3.41.0"},"reference-count":31,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T00:00:00Z","timestamp":1541030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["W911NF-17-C-0094"],"award-info":[{"award-number":["W911NF-17-C-0094"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:p>Popularity systems, like Twitter retweets, Reddit upvotes, and Pinterest pins have the potential to guide people toward posts that others liked. That, however, creates a feedback loop that reduces their informativeness: items marked as more popular get more attention, so that additional upvotes and retweets may simply reflect the increased attention and not independent information about the fraction of people that like the items. How much information remains? For example, how confident can we be that more people prefer item A to item B if item A had hundreds of upvotes on Reddit and item B had only a few? We investigate using an Internet game called GuessTheKarma that collects independent preference judgments (N=20,674) for 400 pairs of images, approximately 50 per pair. Unlike the rating systems that dominate social media services, GuessTheKarma is devoid of social and ranking effects that influence ratings. Overall, Reddit scores were not very good predictors of the true population preferences for items as measured by GuessTheKarma: the image with higher score was preferred by a majority of independent raters only 68% of the time. However, when one image had a low score and the other was one of the highest scoring in its subreddit, the higher scoring image was preferred nearly 90% of the time by the majority of independent raters. Similarly, Imgur view counts for the images were poor predictors except when there were orders of magnitude differences between the pairs. We conclude that popularity systems marked by feedback loops may convey a strong signal about population preferences, but only when comparing items that received vastly different popularity scores.<\/jats:p>","DOI":"10.1145\/3274328","type":"journal-article","created":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T21:21:27Z","timestamp":1541107287000},"page":"1-15","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["GuessTheKarma"],"prefix":"10.1145","volume":"2","author":[{"given":"Maria","family":"Glenski","sequence":"first","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN, USA"}]},{"given":"Greg","family":"Stoddard","sequence":"additional","affiliation":[{"name":"Chicago Crime Lab, New York City, NY, USA"}]},{"given":"Paul","family":"Resnick","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}]},{"given":"Tim","family":"Weninger","sequence":"additional","affiliation":[{"name":"University of Notre Dame, Notre Dame, IN, USA"}]}],"member":"320","published-online":{"date-parts":[[2018,11]]},"reference":[{"volume-title":"Social Informatics","author":"Arapakis Ioannis","key":"e_1_2_1_1_1","unstructured":"Ioannis Arapakis, B Barla Cambazoglu, and Mounia Lalmas. 2014. On the Feasibility of Predicting News Popularity at Cold Start. In Social Informatics . Springer, 290--299."},{"key":"e_1_2_1_2_1","volume-title":"A theory of fads, fashion, custom, and cultural change as informational cascades. Journal of political Economy","author":"Bikhchandani Sushil","year":"1992","unstructured":"Sushil Bikhchandani, David Hirshleifer, and Ivo Welch. 1992. A theory of fads, fashion, custom, and cultural change as informational cascades. Journal of political Economy (1992), 992--1026."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.2736568"},{"key":"e_1_2_1_4_1","volume-title":"Sequential Voting Promotes Collective Discovery in Social Recommendation Systems. arXiv preprint arXiv:1603.04466","author":"Celis L Elisa","year":"2016","unstructured":"L Elisa Celis, Peter M Krafft, and Nathan Kobe. 2016. Sequential Voting Promotes Collective Discovery in Social Recommendation Systems. arXiv preprint arXiv:1603.04466 (2016)."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488417"},{"key":"e_1_2_1_6_1","unstructured":"P Alex Dow Lada A Adamic and Adrien Friggeri. 2013. The Anatomy of Large Facebook Cascades.. In ICWSM ."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557285"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441866"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557081"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSS.2017.2742242"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2963104"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1518882"},{"key":"e_1_2_1_13_1","volume-title":"Groupthink: Psychological studies of policy decisions and fiascoes.","author":"Janis Irving Lester","year":"1982","unstructured":"Irving Lester Janis. 1982. Groupthink: Psychological studies of policy decisions and fiascoes. Vol. 349. Houghton Mifflin Boston."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/0167-2681(94)00068-P"},{"key":"e_1_2_1_15_1","first-page":"3","article-title":"What's in a Name? Understanding the Interplay between Titles, Content, and Communities in Social Media","volume":"1","author":"Lakkaraju Himabindu","year":"2013","unstructured":"Himabindu Lakkaraju, Julian J McAuley, and Jure Leskovec. 2013. What's in a Name? Understanding the Interplay between Titles, Content, and Communities in Social Media. ICWSM , Vol. 1, 2 (2013), 3.","journal-title":"ICWSM"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985761"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0098914"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1008636108"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.5555\/1759957.1759999"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.1102081"},{"key":"e_1_2_1_21_1","volume-title":"Social influence bias: A randomized experiment. Science 341, 6146","author":"Muchnik Lev","year":"2013","unstructured":"Lev Muchnik, Sinan Aral, and Sean J Taylor. 2013. Social influence bias: A randomized experiment. Science 341, 6146 (2013), 647-651."},{"key":"e_1_2_1_22_1","volume-title":"A Bayesian Truth Serum for Subjective Data. Science 306, 5695","author":"Prelec Drazen","year":"2004","unstructured":"{22} Drazen Prelec. 2004. A Bayesian Truth Serum for Subjective Data. Science 306, 5695 (2004), 462-466. arXiv:http:\/\/science.sciencemag.org\/content\/306\/5695\/462.full.pdf"},{"key":"e_1_2_1_23_1","volume-title":"Forecasting elections: Voter intentions versus expectations. Available at SSRN 1884644","author":"Rothschild David M","year":"2011","unstructured":"David M Rothschild and Justin Wolfers. 2011. Forecasting elections: Voter intentions versus expectations. Available at SSRN 1884644 (2011)."},{"key":"e_1_2_1_24_1","volume-title":"Peter Sheridan Dodds, and Duncan J Watts","author":"Salganik Matthew J","year":"2006","unstructured":"Matthew J Salganik, Peter Sheridan Dodds, and Duncan J Watts. 2006. Experimental study of inequality and unpredictability in an artificial cultural market. science , Vol. 311, 5762 (2006), 854--856."},{"key":"e_1_2_1_25_1","volume-title":"Levy","author":"Salganik Matthew J.","year":"2015","unstructured":"Matthew J. Salganik and Karen E. C. Levy. 2015. Wiki Surveys: Open and Quantifiable Social Data Collection. PLOS ONE , Vol. 10, 5 (05 2015), 1--17."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1958824.1958865"},{"key":"e_1_2_1_27_1","volume-title":"AAAI Conference on Web and Social Media (ICWSM) .","author":"Stoddard Greg","year":"2015","unstructured":"Greg Stoddard. 2015. Popularity dynamics and intrinsic quality in reddit and hacker news. In AAAI Conference on Web and Social Media (ICWSM) ."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","unstructured":"James Surowiecki. 2005. The wisdom of crowds .Anchor.","DOI":"10.5555\/1095645"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1017"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1316836111"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985733"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274328","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274328","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274328","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:07Z","timestamp":1750207447000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274328"}},"subtitle":["A Game to Assess Social Rating Systems"],"short-title":[],"issued":{"date-parts":[[2018,11]]},"references-count":31,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1145\/3274328"],"URL":"https:\/\/doi.org\/10.1145\/3274328","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2018,11]]},"assertion":[{"value":"2018-11-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}