{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T14:59:47Z","timestamp":1776092387989,"version":"3.50.1"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2019,11,7]],"date-time":"2019-11-07T00:00:00Z","timestamp":1573084800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2019,11,7]]},"abstract":"<jats:p>In content-based online platforms, use of aggregate user feedback (say, the sum of votes) is commonplace as the \"gold standard\" for measuring content quality. Use of vote aggregates, however, is at odds with the existing empirical literature, which suggests that voters are susceptible to different biases-reputation (e.g., of the poster), social influence (e.g., votes thus far), and position (e.g., answer position). Our goal is to quantify, in an observational setting, the degree of these biases in online platforms. Specifically, what are the causal effects of different impression signals-such as the reputation of the contributing user, aggregate vote thus far, and position of content-on a participant's vote on content? We adopt an instrumental variable (IV) framework to answer this question. We identify a set of candidate instruments, carefully analyze their validity, and then use the valid instruments to reveal the effects of the impression signals on votes. Our empirical study using log data from Stack Exchange websites shows that the bias estimates from our IV approach differ from the bias estimates from the ordinary least squares (OLS) method. In particular, OLS underestimates reputation bias (1.6-2.2x for gold badges) and position bias (up to 1.9x for the initial position) and overestimates social influence bias (1.8-2.3x for initial votes). The implications of our work include: redesigning user interface to avoid voter biases; making changes to platforms' policy to mitigate voter biases; detecting other forms of biases in online platforms.<\/jats:p>","DOI":"10.1145\/3359222","type":"journal-article","created":{"date-parts":[[2019,11,8]],"date-time":"2019-11-08T13:40:21Z","timestamp":1573220421000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Quantifying Voter Biases in Online Platforms"],"prefix":"10.1145","volume":"3","author":[{"given":"Himel","family":"Dev","sequence":"first","affiliation":[{"name":"University of Illinois at Urbana-Champaign, Urbana, IL, USA"}]},{"given":"Karrie","family":"Karahalios","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, Urbana, IL, USA"}]},{"given":"Hari","family":"Sundaram","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, Urbana, IL, USA"}]}],"member":"320","published-online":{"date-parts":[[2019,11,7]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052680"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341557"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2339530.2339665"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488398"},{"key":"e_1_2_2_5_1","volume-title":"Mostly harmless econometrics: An empiricist's companion","author":"Angrist Joshua D","unstructured":"Joshua D Angrist and J\u00f6rn-Steffen Pischke . 2008. Mostly harmless econometrics: An empiricist's companion . Princeton university press . Part 4. Joshua D Angrist and J\u00f6rn-Steffen Pischke. 2008. Mostly harmless econometrics: An empiricist's companion. Princeton university press. Part 4."},{"key":"e_1_2_2_6_1","doi-asserted-by":"crossref","unstructured":"Joshua D Angrist and J\u00f6rn-Steffen Pischke. 2010. The credibility revolution in empirical economics: how better research design is taking the con out of econometrics.Journal of Economic Perspectives 24 2 3--30.  Joshua D Angrist and J\u00f6rn-Steffen Pischke. 2010. The credibility revolution in empirical economics: how better research design is taking the con out of econometrics.Journal of Economic Perspectives 24 2 3--30.","DOI":"10.1257\/jep.24.2.3"},{"key":"e_1_2_2_7_1","volume-title":"Proceedings of the Third International AAAI Conference on Weblogs and Social Media(ICWSM).","author":"Beuscart Jean-Samuel","year":"2009","unstructured":"Jean-Samuel Beuscart and Thomas Couronn\u00e9 . 2009 . The distribution of online reputation: audience and influence of musicians on myspace . In Proceedings of the Third International AAAI Conference on Weblogs and Social Media(ICWSM). Jean-Samuel Beuscart and Thomas Couronn\u00e9. 2009. The distribution of online reputation: audience and influence of musicians on myspace. In Proceedings of the Third International AAAI Conference on Weblogs and Social Media(ICWSM)."},{"key":"e_1_2_2_8_1","doi-asserted-by":"crossref","unstructured":"Oliver Budzinski and Sophia Gaenssle. 2018. The economics of social media stars: an empirical investigation of stardom popularity and success on youtube.  Oliver Budzinski and Sophia Gaenssle. 2018. The economics of social media stars: an empirical investigation of stardom popularity and success on youtube.","DOI":"10.1080\/08997764.2020.1849228"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0173610"},{"key":"e_1_2_2_10_1","volume-title":"Proceedings of the Twelfth International AAAI Conference on Web and Social Media (ICWSM).","author":"Burghardt Keith","year":"2018","unstructured":"Keith Burghardt , Tad Hogg , and Kristina Lerman . 2018 . Quantifying the impact of cognitive biases in question-answering systems . In Proceedings of the Twelfth International AAAI Conference on Web and Social Media (ICWSM). Keith Burghardt, Tad Hogg, and Kristina Lerman. 2018. Quantifying the impact of cognitive biases in question-answering systems. In Proceedings of the Twelfth International AAAI Conference on Web and Social Media (ICWSM)."},{"key":"e_1_2_2_11_1","volume-title":"Handbook of Labor Economics.","author":"Card David","year":"1801","unstructured":"David Card . 1999. The causal effect of education on earnings . In Handbook of Labor Economics. Vol. 3 . Elsevier , 1801 --1863. David Card. 1999. The causal effect of education on earnings. In Handbook of Labor Economics. Vol. 3. Elsevier, 1801--1863."},{"key":"e_1_2_2_12_1","volume-title":"Proceedings of the Tenth International AAAI Conference on Web and Social Media (ICWSM),42--51","author":"Celis L Elisa","year":"2016","unstructured":"L Elisa Celis , Peter M Krafft , and Nathan Kobe . 2016 . Sequential voting promotes collective discovery in social recommendation systems . In Proceedings of the Tenth International AAAI Conference on Web and Social Media (ICWSM),42--51 . L Elisa Celis, Peter M Krafft, and Nathan Kobe. 2016. Sequential voting promotes collective discovery in social recommendation systems. In Proceedings of the Tenth International AAAI Conference on Web and Social Media (ICWSM),42--51."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186037"},{"key":"e_1_2_2_14_1","doi-asserted-by":"crossref","unstructured":"Matt Dickson. 2013. The causal effect of education on wages revisited.Oxford Bulletin of Economics and Statistics 75 4 477--498.  Matt Dickson. 2013. The causal effect of education on wages revisited.Oxford Bulletin of Economics and Statistics 75 4 477--498.","DOI":"10.1111\/j.1468-0084.2012.00708.x"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174006"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2950290.2950331"},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0200162"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.2307\/2585672"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441866"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2615569.2615681"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSS.2017.2742242"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274328"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2963104"},{"key":"e_1_2_2_24_1","unstructured":"Benjamin Grosser. 2014. What do metrics want? how quantification prescribes social interaction on facebook. Computational Culture.  Benjamin Grosser. 2014. What do metrics want? how quantification prescribes social interaction on facebook. Computational Culture."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/HICSS.2014.206"},{"key":"e_1_2_2_26_1","unstructured":"Jay Hanlon. 2019. Stack overflow isn't very welcoming. it's time for that to change. https:\/\/stackoverflow.blog\/2018\/04\/26\/stack-overflow-isnt-very-welcoming-its-time-for-that-to-change\/. (2019).  Jay Hanlon. 2019. Stack overflow isn't very welcoming. it's time for that to change. https:\/\/stackoverflow.blog\/2018\/04\/26\/stack-overflow-isnt-very-welcoming-its-time-for-that-to-change\/. (2019)."},{"key":"e_1_2_2_27_1","volume-title":"Causal inference","author":"Hern\u00e1n MA","unstructured":"MA Hern\u00e1n and JM Robins . 2019. Causal inference . In Chapman & Hall\/CRC. Part 16. MA Hern\u00e1n and JM Robins. 2019. Causal inference. In Chapman & Hall\/CRC. Part 16."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.15346\/hc.v2i2.4"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0403723101"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148212"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018699"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1177\/000312240507000107"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2645710.2645740"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0033785"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186160"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0098914"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998217"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1008636108"},{"key":"e_1_2_2_39_1","unstructured":"Glenn M MacDonald. 1988. The economics of rising stars. The American Economic Review 155--166.  Glenn M MacDonald. 1988. The economics of rising stars. The American Economic Review 155--166."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313583"},{"key":"e_1_2_2_41_1","doi-asserted-by":"crossref","unstructured":"Robert K Merton. 1968. The matthew effect in science: the reward and communication systems of science a reconsidered. Science 159 3810 56--63.  Robert K Merton. 1968. The matthew effect in science: the reward and communication systems of science a reconsidered. Science 159 3810 56--63.","DOI":"10.1126\/science.159.3810.56"},{"key":"e_1_2_2_42_1","volume-title":"Proceedings of the 2013IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). ACM, 886--893","author":"Movshovitz-Attias Dana","year":"2013","unstructured":"Dana Movshovitz-Attias , Yair Movshovitz-Attias , Peter Steenkiste , and Christos Faloutsos . 2013 . Analysis of the reputation system and user contributions on a question answering website: stack overflow . In Proceedings of the 2013IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). ACM, 886--893 . Dana Movshovitz-Attias, Yair Movshovitz-Attias, Peter Steenkiste, and Christos Faloutsos. 2013. Analysis of the reputation system and user contributions on a question answering website: stack overflow. In Proceedings of the 2013IEEE\/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). ACM, 886--893."},{"key":"e_1_2_2_43_1","doi-asserted-by":"crossref","unstructured":"Lev Muchnik Sinan Aral and Sean J Taylor. 2013. Social influence bias: a randomized experiment. Science 341 6146 647--651.  Lev Muchnik Sinan Aral and Sean J Taylor. 2013. Social influence bias: a randomized experiment. Science 341 6146 647--651.","DOI":"10.1126\/science.1240466"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1964858.1964859"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935843"},{"key":"e_1_2_2_46_1","volume-title":"H Chi","author":"Paul Sharoda A","year":"2012","unstructured":"Sharoda A Paul , Lichan Hong , and Ed H Chi . 2012 . Who is authoritative? understanding reputation mechanisms in quora. In Collective Intelligence . Sharoda A Paul, Lichan Hong, and Ed H Chi. 2012. Who is authoritative? understanding reputation mechanisms in quora. In Collective Intelligence."},{"key":"e_1_2_2_47_1","volume-title":"Foundations of Bayesianism","author":"Pearl Judea","unstructured":"Judea Pearl . 2001. Bayesianism and causality, or, why i am only a half-bayesian . In Foundations of Bayesianism . Springer , 19--36. Judea Pearl. 2001. Bayesianism and causality, or, why i am only a half-bayesian. In Foundations of Bayesianism. Springer, 19--36."},{"key":"e_1_2_2_48_1","volume-title":"Proceedings of the National Conference on Artificial Intelligence (AAAI)number 2.","volume":"21","author":"Radlinski Filip","year":"2006","unstructured":"Filip Radlinski and Thorsten Joachims . 2006 . Minimally invasive randomization for collecting unbiased preferences from clickthrough logs . In Proceedings of the National Conference on Artificial Intelligence (AAAI)number 2. Vol. 21 . Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 1406. Filip Radlinski and Thorsten Joachims. 2006. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In Proceedings of the National Conference on Artificial Intelligence (AAAI)number 2. Vol. 21. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 1406."},{"key":"e_1_2_2_49_1","first-page":"1","article-title":"karma, precious karma!' karmawhoring on reddit and the front page's econometrisation","volume":"4","author":"Richterich Annika","year":"2014","unstructured":"Annika Richterich . 2014 . ' karma, precious karma!' karmawhoring on reddit and the front page's econometrisation . Journal of Peer Production , 4 , 1 . Annika Richterich. 2014. 'karma, precious karma!' karmawhoring on reddit and the front page's econometrisation. Journal of Peer Production, 4, 1.","journal-title":"Journal of Peer Production"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2013.49"},{"key":"e_1_2_2_51_1","volume-title":"Peter Sheridan Dodds, and Duncan J Watts","author":"Salganik Matthew J","year":"2006","unstructured":"Matthew J Salganik , Peter Sheridan Dodds, and Duncan J Watts . 2006 . Experimental study of inequality and unpredictability in an artificial cultural market. Science , 311, 5762, 854--856. Matthew J Salganik, Peter Sheridan Dodds, and Duncan J Watts. 2006. Experimental study of inequality and unpredictability in an artificial cultural market. Science, 311, 5762, 854--856."},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835518"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567998"},{"key":"e_1_2_2_54_1","volume-title":"Proceedings of the Ninth International AAAI Conference on Web and Social Media (ICWSM), 416--425","author":"Stoddard Greg","year":"2015","unstructured":"Greg Stoddard . 2015 . Popularity dynamics and intrinsic quality in reddit and hacker news . In Proceedings of the Ninth International AAAI Conference on Web and Social Media (ICWSM), 416--425 . Greg Stoddard. 2015. Popularity dynamics and intrinsic quality in reddit and hacker news. In Proceedings of the Ninth International AAAI Conference on Web and Social Media (ICWSM), 416--425."},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979215"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134736"},{"key":"e_1_2_2_57_1","volume-title":"Proceedings of the Tenth International AAAI Conference on Web and Social Media (ICWSM), 398--407","author":"Hentenryck Pascal Van","year":"2016","unstructured":"Pascal Van Hentenryck , Andr\u00e9s Abeliuk , Franco Berbeglia , Felipe Maldonado , and Gerardo Berbeglia . 2016 . Aligning popularity and quality in online cultural markets . In Proceedings of the Tenth International AAAI Conference on Web and Social Media (ICWSM), 398--407 . Pascal Van Hentenryck, Andr\u00e9s Abeliuk, Franco Berbeglia, Felipe Maldonado, and Gerardo Berbeglia. 2016. Aligning popularity and quality in online cultural markets. In Proceedings of the Tenth International AAAI Conference on Web and Social Media (ICWSM), 398--407."},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/SocialInformatics.2012.81"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623720"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.soc.25.1.659"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-92185-1_39"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0149151"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772793"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3359222","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3359222","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:12:54Z","timestamp":1750201974000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3359222"}},"subtitle":["An Instrumental Variable Approach"],"short-title":[],"issued":{"date-parts":[[2019,11,7]]},"references-count":63,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2019,11,7]]}},"alternative-id":["10.1145\/3359222"],"URL":"https:\/\/doi.org\/10.1145\/3359222","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11,7]]},"assertion":[{"value":"2019-11-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}