{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T20:13:02Z","timestamp":1780344782988,"version":"3.54.1"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T00:00:00Z","timestamp":1541030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Xi'an Jiaotong - Liverpool University","award":["RDF 15-02-17"],"award-info":[{"award-number":["RDF 15-02-17"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:p>Following successful crowd ideation contests, organizations in search of the \"next big thing\" are left with hundreds of ideas. Expert-based idea filtering is lengthy and costly; therefore, crowd-based strategies are often employed. Unfortunately, these strategies typically (1) do not separate the mediocre from the excellent, and (2) direct all the attention to certain idea concepts, while others starve. We introduce DBLemons - a crowd-based idea filtering strategy that addresses these issues by (1) asking voters to identify the worst rather than the best ideas using a \"bag of lemons'' voting approach, and (2) by exposing voters to a wider idea spectrum, thanks to a dynamic diversity-based ranking system balancing idea quality and coverage. We compare DBLemons against two state-of-the-art idea filtering strategies in a real-world setting. Results show that DBLemons is more accurate, less time-consuming, and reduces the idea space in half while still retaining 94% of the top ideas.<\/jats:p>","DOI":"10.1145\/3274384","type":"journal-article","created":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T21:21:27Z","timestamp":1541107287000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["When Crowds Give You Lemons"],"prefix":"10.1145","volume":"2","author":[{"given":"Ioanna","family":"Lykourentzou","sequence":"first","affiliation":[{"name":"Utrecht University, Utrecht, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Faez","family":"Ahmed","sequence":"additional","affiliation":[{"name":"University of Maryland, College Park, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Costas","family":"Papastathis","sequence":"additional","affiliation":[{"name":"University of Peloponnese, Tripoli-Arcadia, Greece"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Irwyn","family":"Sadien","sequence":"additional","affiliation":[{"name":"Xi'an Jiaotong - Liverpool University, Suzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Konstantinos","family":"Papangelis","sequence":"additional","affiliation":[{"name":"Xi'an Jiatong-Liverpool University, Suzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,11]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Community heuristics for user interface evaluation of crowdsourcing platforms. Future Generation Computer Systems","author":"Campo Simon","year":"2018","unstructured":"Simon \u00e0 Campo, Vasssilis-Javed Khan, Konstantinos Papangelis, and Panos Markopoulos. 2018. Community heuristics for user interface evaluation of crowdsourcing platforms. Future Generation Computer Systems (2018)."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998249"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1115\/1.4038070"},{"key":"e_1_2_1_4_1","volume-title":"ASME International Design Engineering Technical Conferences. ASME","author":"Ahmed Faez","unstructured":"Faez Ahmed, Mark Fuge, and Lev D. Gorbunov. 2016. Discovering diverse, high quality design ideas from a large corpus. In ASME International Design Engineering Technical Conferences. ASME, Charlotte, USA."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1014052.1014097"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2012.07.021"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.2015.0605"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.2014.1909"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1115\/1.4029065"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"D. Cer M. Diab E. Agirre I. Lopez-Gazpio and L. Specia. 2017. SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation. ArXiv e-prints (July 2017). arXiv:cs.CL\/1708.00055","DOI":"10.18653\/v1\/S17-2001"},{"key":"e_1_2_1_11_1","volume-title":"Comparing strategies for winning expert-rated and crowd-rated crowdsourcing contests: First findings","author":"Chen Liang","unstructured":"Liang Chen and De Liu. 2012. Comparing strategies for winning expert-rated and crowd-rated crowdsourcing contests: First findings. Vol. 1. 97--107."},{"key":"e_1_2_1_12_1","unstructured":"Liang Chen Pei Xu and De Liu. 2016. Experts versus the crowd: a comparison of selection mechanisms in crowdsourcing contests. (2016)."},{"key":"e_1_2_1_13_1","volume-title":"Single-peaked functions and the theory of preference. Psychological review 84, 2","author":"Coombs Clyde H","year":"1977","unstructured":"Clyde H Coombs and George S Avrunin. 1977. Single-peaked functions and the theory of preference. Psychological review 84, 2 (1977), 216."},{"key":"e_1_2_1_14_1","volume-title":"Crowdsourcing for Participatory Democracies: Efficient Elicitation of Social Choice Functions. Second AAAI Conference on Human Computation and Crowdsourcing","author":"Helene Landemore David Timothy Tanja Aitamurto","year":"2014","unstructured":"Tanja Aitamurto Helene Landemore David Timothy Lee, Ashish Goel. 2014. Crowdsourcing for Participatory Democracies: Efficient Elicitation of Social Choice Functions. Second AAAI Conference on Human Computation and Crowdsourcing (2014)."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.17705\/1jais.00106"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5120\/ijca2017913766"},{"key":"e_1_2_1_17_1","first-page":"52","article-title":"The wisdom of crowds in the movie industry: Towards new solutions to reduce uncertainties","volume":"17","author":"Escoffier Nadine","year":"2015","unstructured":"Nadine Escoffier and Bill McKelvey. 2015. The wisdom of crowds in the movie industry: Towards new solutions to reduce uncertainties. International Journal of Arts Management 17, 2 (2015), 52.","journal-title":"International Journal of Arts Management"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1115\/1.4026510"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.24840\/2183-0606_004.004_0006"},{"key":"e_1_2_1_21_1","volume-title":"Open Innovation: Researching a New Paradigm","author":"Henry Chesbrough Joel West","year":"2006","unstructured":"Joel West Henry Chesbrough, Wim Vanhaverbeke. 2006. Open Innovation: Researching a New Paradigm. Oxford University Press."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1162"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098038"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.technovation.2010.10.002"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9310.2009.00563.x"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_1_27_1","first-page":"1","article-title":"A roadmap for open innovation systems","volume":"2","author":"Klein Mark","year":"2015","unstructured":"Mark Klein and Gregorio Convertino. 2015. A roadmap for open innovation systems. Journal of Social Media for Organizations 2, 1 (2015), 1.","journal-title":"Journal of Social Media for Organizations"},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Mark Klein and Ana Cristina Bicharra Garcia. 2014. The bag of stars: High-speed idea filtering for open innovation. (2014).","DOI":"10.2139\/ssrn.2387180"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2015.06.005"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-6486.2009.00854.x"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.5555\/2002472.2002537"},{"key":"e_1_2_1_32_1","volume-title":"Collective Wisdom: Lessons From the Theory of Judgment Aggregation. In Collective Wisdom: Principles and Mechanisms","author":"List Christian","year":"2012","unstructured":"Christian List. 2012. Collective Wisdom: Lessons From the Theory of Judgment Aggregation. In Collective Wisdom: Principles and Mechanisms, Helene Landemore and Jon Elster (Eds.). Cambridge University Press."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0142-694X(03)00003-6"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jsis.2013.07.004"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijresmar.2004.01.002"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF01588971"},{"key":"e_1_2_1_37_1","volume-title":"ASME 2016 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference","author":"Pang Michelle A","unstructured":"Michelle A Pang and Carolyn C Seepersad. 2016. Crowdsourcing the Evaluation of Design Concepts With Empathic Priming. In ASME 2016 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. American Society of Mechanical Engineers, V007T06A004--V007T06A004."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1982.10477845"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/0001-6918(92)90043-D"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.2753\/JEC1086-4415170301"},{"key":"e_1_2_1_41_1","volume-title":"Ideas are free: How the idea revolution is liberating people and transforming organizations","author":"Robinson Alan G","unstructured":"Alan G Robinson and DeanMSchroeder. 2004. Ideas are free: How the idea revolution is liberating people and transforming organizations. Berrett-Koehler Publishers."},{"key":"e_1_2_1_42_1","volume-title":"Levy","author":"Salganik Matthew J.","year":"2015","unstructured":"Matthew J. Salganik and Karen E. C. Levy. 2015. Wiki Surveys: Open and Quantifiable Social Data Collection. PLOS ONE 10, 5 (05 2015), 1--17."},{"key":"e_1_2_1_44_1","volume-title":"Constraining effects of examples in a creative generation task. Memory & cognition 21, 6","author":"Ward Thomas","year":"1993","unstructured":"StevenMSmith, Thomas BWard, and Jay S Schumacher. 1993. Constraining effects of examples in a creative generation task. Memory & cognition 21, 6 (1993), 837--845."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.5555\/1095645"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog1202_4"},{"key":"e_1_2_1_47_1","volume-title":"Google struggles to give away 10 million","author":"van Buskirk Eliot","unstructured":"Eliot van Buskirk. 2010. Google struggles to give away 10 million. http:\/\/www.wired.com\/2010\/06\/google-struggles-togive-away-10-million\/all\/1. (2010). Archived on: 2016-02-04."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15326985EP3801_2"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043955"},{"key":"e_1_2_1_50_1","volume-title":"13th International Conference on Wirtschaftsinformatik.","author":"Wagenknecht Thomas","year":"2017","unstructured":"Thomas Wagenknecht, Jan Crommelinck, Timm Teubner, and Christof Weinhardt. 2017. When Life Gives You Lemons: How rating scales affect user activity and frustration in collaborative evaluation processes. In 13th International Conference on Wirtschaftsinformatik."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145380"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2124295.2124300"},{"key":"e_1_2_1_53_1","volume-title":"Jurgen Van Gael, and David Andrzejewski","author":"Zhu Xiaojin","year":"2007","unstructured":"Xiaojin Zhu, Andrew B Goldberg, Jurgen Van Gael, and David Andrzejewski. 2007. Improving Diversity in Ranking using Absorbing Random Walks. In HLT-NAACL. Citeseer, 97--104."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274384","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274384","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:35Z","timestamp":1750207475000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274384"}},"subtitle":["Filtering Innovative Ideas using a Diverse-Bag-of-Lemons Strategy"],"short-title":[],"issued":{"date-parts":[[2018,11]]},"references-count":51,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1145\/3274384"],"URL":"https:\/\/doi.org\/10.1145\/3274384","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11]]},"assertion":[{"value":"2018-11-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}