{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T22:42:58Z","timestamp":1757544178956,"version":"3.41.0"},"reference-count":9,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2020,10,14]],"date-time":"2020-10-14T00:00:00Z","timestamp":1602633600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["W911NF-17-1-0077"],"award-info":[{"award-number":["W911NF-17-1-0077"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]},{"name":"US Army Research Office MURI","award":["W911NF-13-1-0340"],"award-info":[{"award-number":["W911NF-13-1-0340"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2020,10,14]]},"abstract":"<jats:p>Crowdsourcing systems aggregate decisions of many people to help users quickly identify high-quality options, such as the best answers to questions or interesting news stories. A long-standing issue in crowdsourcing is how option quality and human judgement heuristics interact to affect collective outcomes, such as the perceived popularity of options. We address this limitation by conducting a controlled experiment where subjects choose between two ranked options whose quality can be independently varied. We use this data to construct a model that quantifies how judgement heuristics and option quality combine when deciding between two options. The model reveals popularity-ranking can be unstable: unless the quality difference between the two options is sufficiently high, the higher quality option is not guaranteed to be eventually ranked on top. To rectify this instability, we create an algorithm that accounts for judgement heuristics to infer the best option and rank it first. This algorithm is guaranteed to be optimal if data matches the model. When the data does not match the model, however, simulations show that in practice this algorithm performs better or at least as well as popularity-based and recency-based ranking for any two-choice question. Our work suggests that algorithms relying on inference of mathematical models of user behavior can substantially improve outcomes in crowdsourcing systems.<\/jats:p>","DOI":"10.1145\/3415237","type":"journal-article","created":{"date-parts":[[2020,10,15]],"date-time":"2020-10-15T22:27:49Z","timestamp":1602800869000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Origins of Algorithmic Instabilities in Crowdsourced Ranking"],"prefix":"10.1145","volume":"4","author":[{"given":"Keith","family":"Burghardt","sequence":"first","affiliation":[{"name":"USC Information Sciences Institute, Marina del Rey, CA, USA"}]},{"given":"Tad","family":"Hogg","sequence":"additional","affiliation":[{"name":"Institute for Molecular Manufacturing, Palo Alto, CA, USA"}]},{"given":"Raissa","family":"D'Souza","sequence":"additional","affiliation":[{"name":"University of California, Davis &amp; Sante Fe Institute, Davis, CA, USA"}]},{"given":"Kristina","family":"Lerman","sequence":"additional","affiliation":[{"name":"USC Information Sciences Institute, Marina del Rey, CA, USA"}]},{"given":"Marton","family":"Posfai","sequence":"additional","affiliation":[{"name":"Central European University &amp; University of California, Davis, Budapest, Hungary"}]}],"member":"320","published-online":{"date-parts":[[2020,10,15]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3109859.3109912"},{"volume-title":"Managing Popularity Bias in Recommender Systems with Personalized Re-ranking. arXiv preprint","year":"1901","author":"Abdollahpouri Himan","key":"e_1_2_1_2_1"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052680"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1093\/restud\/rdr004"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1316836111"},{"key":"e_1_2_1_7_1","unstructured":"D. J. Watts. 2012. Everything Is Obvious: How Common Sense Fails Us Random House LLC.  D. J. Watts. 2012. Everything Is Obvious: How Common Sense Fails Us .Random House LLC."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484053"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177732360"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2014.12.038"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3415237","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3415237","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3415237","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:03:11Z","timestamp":1750197791000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3415237"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,14]]},"references-count":9,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2020,10,14]]}},"alternative-id":["10.1145\/3415237"],"URL":"https:\/\/doi.org\/10.1145\/3415237","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2020,10,14]]},"assertion":[{"value":"2020-10-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}