{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:01:38Z","timestamp":1755838898048,"version":"3.41.0"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"GROUP","license":[{"start":{"date-parts":[[2020,1,4]],"date-time":"2020-01-04T00:00:00Z","timestamp":1578096000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100003593","name":"Conselho Nacional de Desenvolvimento Cient\u00edfico e Tecnol\u00f3gico","doi-asserted-by":"crossref","award":["420801\/2016-2 and 311256\/2018-0"],"award-info":[{"award-number":["420801\/2016-2 and 311256\/2018-0"]}],"id":[{"id":"10.13039\/501100003593","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2020,1,4]]},"abstract":"<jats:p>Crowdsourcing divides a task into small pieces that are carried out by the crowd. In Software Engineering, crowdsourcing divides the software development tasks of to be carried out online by the crow and is simply called Software Crowdsourcing (SW CS). Most SW CS platforms support this emerging software development strategy and operate within a framework of competition among the crowd. Competitive SW CS platforms intentionally minimize communication and collaboration among the parties involved (customer, platform, and crowd) while they compete in the software development tasks. The goal of this paper is to investigate platform moderators in SW CS challenges. Platform moderators are individuals who work for the SW CS platforms to mediate customer and crowd. A qualitative analysis of the content of the communication forums hosted on the TopCoder platform was performed to analyze the messages exchanged by the platform moderators and the crowd. Our empirical results indicate that co-pilots enforce and, at the same time, extend the limitations of the documentation associated with the tasks to support crowd members, provide technical help to crowd members during the competitions, and engage the crowd in the challenges. Co-pilots are organized, work diligently, worrying about being fair, and, at the same time, seeking to find a balance between autonomy and dependency on the customer. We conclude by providing insights to improve the design of software crowdsourcing platforms.<\/jats:p>","DOI":"10.1145\/3375194","type":"journal-article","created":{"date-parts":[[2020,1,4]],"date-time":"2020-01-04T22:05:18Z","timestamp":1578175518000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["On Moderating Software Crowdsourcing Challenges"],"prefix":"10.1145","volume":"4","author":[{"given":"Cleidson R. B.","family":"de Souza","sequence":"first","affiliation":[{"name":"Universidade Federal do Par\u00e1, Bel\u00e9m, PA, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Leticia S.","family":"Machado","sequence":"additional","affiliation":[{"name":"Universidade Federal do Par\u00e1, Bel\u00e9m, PA, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ricardo Rodrigo M.","family":"Melo","sequence":"additional","affiliation":[{"name":"Universidade Federal do Par\u00e1, Bel\u00e9m, PA, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,1,4]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"crossref","unstructured":"Nikolay Archak. 2010. Money Glory and Cheap Talk: Analyzing Strategic Behavior of Contestants in Simultaneous Crowdsourcing Contests on TopCoder.com. 21--30. https:\/\/doi.org\/10.1145\/1772690.1772694  Nikolay Archak. 2010. Money Glory and Cheap Talk: Analyzing Strategic Behavior of Contestants in Simultaneous Crowdsourcing Contests on TopCoder.com. 21--30. https:\/\/doi.org\/10.1145\/1772690.1772694","DOI":"10.1145\/1772690.1772694"},{"key":"e_1_2_1_2_1","unstructured":"Joost Bianchi Yuri Knopper Ozgur Eris Petra Badke-Schaub and Lampros Roussos. 2015. Online Ways of Sharedness: A Syntactic Analysis of Design Collaboration in OpenIDEO.  Joost Bianchi Yuri Knopper Ozgur Eris Petra Badke-Schaub and Lampros Roussos. 2015. Online Ways of Sharedness: A Syntactic Analysis of Design Collaboration in OpenIDEO."},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","unstructured":"Kevin Boudreau Patrick Gaul\u00e9 Karim Lakhani Christoph Riedl and AnitaWoolley. 2014. From Crowds to Collaborators: Initiating Effort Catalyzing Interactions Among Online Creative Workers. SSRN Electronic Journal (01 2014). https:\/\/doi.org\/10.2139\/ssrn.2384068  Kevin Boudreau Patrick Gaul\u00e9 Karim Lakhani Christoph Riedl and AnitaWoolley. 2014. From Crowds to Collaborators: Initiating Effort Catalyzing Interactions Among Online Creative Workers. SSRN Electronic Journal (01 2014). https:\/\/doi.org\/10.2139\/ssrn.2384068","DOI":"10.2139\/ssrn.2384068"},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","unstructured":"Brabham and Daren C. 2008. Crowdsourcing as a Model for Problem Solving: An Introduction and Cases. The International Journal of Research into New Media Technologies 14 (1) (01 2008) 75--90.  Brabham and Daren C. 2008. Crowdsourcing as a Model for Problem Solving: An Introduction and Cases. The International Journal of Research into New Media Technologies 14 (1) (01 2008) 75--90.","DOI":"10.1177\/1354856507084420"},{"key":"e_1_2_1_5_1","doi-asserted-by":"crossref","unstructured":"Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psychology. Qualitative research in psychology 3 (01 2006) 77--101. https:\/\/doi.org\/10.1191\/1478088706qp063oa  Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psychology. Qualitative research in psychology 3 (01 2006) 77--101. https:\/\/doi.org\/10.1191\/1478088706qp063oa","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1924421.1924442"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1924421.1924442"},{"key":"e_1_2_1_8_1","doi-asserted-by":"crossref","unstructured":"Alpana Dubey Kumar Abhinav Sakshi Taneja Gurdeep Virdi Anurag Dwarakanath Alex Kass and Suma Mani. 2016. Dynamics of Software Development Crowdsourcing. https:\/\/doi.org\/10.1109\/ICGSE.2016.13  Alpana Dubey Kumar Abhinav Sakshi Taneja Gurdeep Virdi Anurag Dwarakanath Alex Kass and Suma Mani. 2016. Dynamics of Software Development Crowdsourcing. https:\/\/doi.org\/10.1109\/ICGSE.2016.13","DOI":"10.1109\/ICGSE.2016.13"},{"key":"e_1_2_1_9_1","doi-asserted-by":"crossref","unstructured":"Mark Fuge Kevin Tee Alice Agogino and Nathan Maton. 2014. Analysis of Collaborative Design Networks: A Case Study of OpenIDEO. Journal of Computing and Information Science in Engineering 14 (06 2014) 021009. https:\/\/doi.org\/10.1115\/1.4026510  Mark Fuge Kevin Tee Alice Agogino and Nathan Maton. 2014. Analysis of Collaborative Design Networks: A Case Study of OpenIDEO. Journal of Computing and Information Science in Engineering 14 (06 2014) 021009. https:\/\/doi.org\/10.1115\/1.4026510","DOI":"10.1115\/1.4026510"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819942"},{"volume-title":"Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business (1 ed.)","year":"2008","author":"Howe Jeff","key":"e_1_2_1_11_1"},{"volume-title":"Communitition: The Tension Between Competition and Collaboration in Community Based Design Contests. Creativity and Innovation Management 20 (02","year":"2011","author":"Hutter Katja","key":"e_1_2_1_12_1"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CTS.2011.5928663"},{"key":"e_1_2_1_14_1","unstructured":"Evgeny Kaganer Erran Carmel Rudy Hirschheim and Olsen Timothy. 2013. Managing the Human Cloud. MIT Sloan Management Review 54 (12 2013) 23.  Evgeny Kaganer Erran Carmel Rudy Hirschheim and Olsen Timothy. 2013. Managing the Human Cloud. MIT Sloan Management Review 54 (12 2013) 23."},{"volume-title":"Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW '08)","author":"Kittur Aniket","key":"e_1_2_1_15_1"},{"volume-title":"Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (CSCW '10)","year":"1891","author":"Kittur Aniket","key":"e_1_2_1_16_1"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"volume-title":"Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW '08)","author":"Kriplean Travis","key":"e_1_2_1_18_1"},{"key":"e_1_2_1_19_1","unstructured":"Karim Lakhani David Garvin and Eric Lonstein. 2010. TopCoder (A): Developing Software through Crowdsourcing. (01 2010).  Karim Lakhani David Garvin and Eric Lonstein. 2010. TopCoder (A): Developing Software through Crowdsourcing. (01 2010)."},{"key":"e_1_2_1_20_1","doi-asserted-by":"crossref","unstructured":"Thomas LaToza Micky Chen Luxi Jiang Mengyao Zhao and Andre van der Hoek. 2015. Borrowing from the Crowd: A Study of Recombination in Software Design Competitions. https:\/\/doi.org\/10.1109\/ICSE.2015.72  Thomas LaToza Micky Chen Luxi Jiang Mengyao Zhao and Andre van der Hoek. 2015. Borrowing from the Crowd: A Study of Recombination in Software Design Competitions. https:\/\/doi.org\/10.1109\/ICSE.2015.72","DOI":"10.1109\/ICSE.2015.72"},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Leticia Machado Alexandre Zanatta Sabrina Marczak and Rafael Prikladnicki. 2017. The Good the Bad and the Ugly: An Onboard Journey in Software Crowdsourcing Competitive Model. https:\/\/doi.org\/10.1109\/CSI-SE.2017.6  Leticia Machado Alexandre Zanatta Sabrina Marczak and Rafael Prikladnicki. 2017. The Good the Bad and the Ugly: An Onboard Journey in Software Crowdsourcing Competitive Model. https:\/\/doi.org\/10.1109\/CSI-SE.2017.6","DOI":"10.1109\/CSI-SE.2017.6"},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Thomas Malone Robert Laubacher and Chrysanthos Dellarocas. 2010. The Collective Intelligence Genome. IEEE Engineering Management Review 38 (03 2010) 38--52. https:\/\/doi.org\/10.1109\/EMR.2010.5559142  Thomas Malone Robert Laubacher and Chrysanthos Dellarocas. 2010. The Collective Intelligence Genome. IEEE Engineering Management Review 38 (03 2010) 38--52. https:\/\/doi.org\/10.1109\/EMR.2010.5559142","DOI":"10.1109\/EMR.2010.5559142"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Ke Mao Licia Capra Mark Harman and Yue Jia. 2016. A Survey of the Use of Crowdsourcing in Software Engineering. Journal of Systems and Software 126 (09 2016). https:\/\/doi.org\/10.1016\/j.jss.2016.09.015  Ke Mao Licia Capra Mark Harman and Yue Jia. 2016. A Survey of the Use of Crowdsourcing in Software Engineering. Journal of Systems and Software 126 (09 2016). https:\/\/doi.org\/10.1016\/j.jss.2016.09.015","DOI":"10.1016\/j.jss.2016.09.015"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470728"},{"key":"e_1_2_1_25_1","unstructured":"Ricardo Melo Leticia Machado Rafael Prikladnicki and Cleidson de Souza. 2018. Um Estudo Qualitativo sobre o Crowdsourcing: An\u00e1lise da Colabora\u00e7\u00e3o na plataforma TopCoder. http:\/\/toc.proceedings.com\/40795webtoc.pdf  Ricardo Melo Leticia Machado Rafael Prikladnicki and Cleidson de Souza. 2018. Um Estudo Qualitativo sobre o Crowdsourcing: An\u00e1lise da Colabora\u00e7\u00e3o na plataforma TopCoder. http:\/\/toc.proceedings.com\/40795webtoc.pdf"},{"key":"e_1_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Sreeja Nag Ira Heffan Alvar Saenz-Otero and Mike Lydon. 2012. SPHERES Zero Robotics Software Development: Lessons on Crowdsourcing and Collaborative Competition. https:\/\/doi.org\/10.1109\/AERO.2012.6187452  Sreeja Nag Ira Heffan Alvar Saenz-Otero and Mike Lydon. 2012. SPHERES Zero Robotics Software Development: Lessons on Crowdsourcing and Collaborative Competition. https:\/\/doi.org\/10.1109\/AERO.2012.6187452","DOI":"10.1109\/AERO.2012.6187452"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/143457.143549"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2017.2"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1180875.1180904"},{"key":"e_1_2_1_30_1","doi-asserted-by":"crossref","unstructured":"Klaas-Jan Stol Bora Caglayan and Brian Fitzgerald. 2017. Competition-Based Crowdsourcing Software Development: A Multi-Method Study from a Customer Perspective. IEEE Transactions on Software Engineering PP (11 2017). https:\/\/doi.org\/10.1109\/TSE.2017.2774297  Klaas-Jan Stol Bora Caglayan and Brian Fitzgerald. 2017. Competition-Based Crowdsourcing Software Development: A Multi-Method Study from a Customer Perspective. IEEE Transactions on Software Engineering PP (11 2017). https:\/\/doi.org\/10.1109\/TSE.2017.2774297","DOI":"10.1109\/TSE.2017.2774297"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568249"},{"key":"e_1_2_1_32_1","unstructured":"A. Strauss and J. Corbin. 1990. Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Sage Publications Newbury Park California.  A. Strauss and J. Corbin. 1990. Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Sage Publications Newbury Park California."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134735"},{"key":"e_1_2_1_34_1","unstructured":"TopCoder. 2008. TopCoder. Retrieved March 2 2008 from https:\/\/www.topcoder.com  TopCoder. 2008. TopCoder. Retrieved March 2 2008 from https:\/\/www.topcoder.com"},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Luis Vaz Sabrina Marczak and Igor Steinmacher. 2018. An empirical study on task documentation in software crowdsourcing: the case of the topcoder platform. 62--71. https:\/\/doi.org\/10.1145\/3266237.3266265  Luis Vaz Sabrina Marczak and Igor Steinmacher. 2018. An empirical study on task documentation in software crowdsourcing: the case of the topcoder platform. 62--71. https:\/\/doi.org\/10.1145\/3266237.3266265","DOI":"10.1145\/3266237.3266265"},{"key":"e_1_2_1_36_1","unstructured":"Martin Wattenberg Fernanda Vi\u00e9gas and Katherine Hollenbach. 2007. Visualizing Activity on Wikipedia with Chromograms Vol. 4663. 272--287. https:\/\/doi.org\/10.1007\/978--3--540--74800--7_23  Martin Wattenberg Fernanda Vi\u00e9gas and Katherine Hollenbach. 2007. Visualizing Activity on Wikipedia with Chromograms Vol. 4663. 272--287. https:\/\/doi.org\/10.1007\/978--3--540--74800--7_23"},{"key":"e_1_2_1_37_1","doi-asserted-by":"crossref","unstructured":"Ye Yang Muhammad Rezaul Karim Razieh Saremi and Guenther Ruhe. 2016. Who Should Take This Task?: Dynamic Decision Support for Crowd Workers. 1--10. https:\/\/doi.org\/10.1145\/2961111.2962594  Ye Yang Muhammad Rezaul Karim Razieh Saremi and Guenther Ruhe. 2016. Who Should Take This Task?: Dynamic Decision Support for Crowd Workers. 1--10. https:\/\/doi.org\/10.1145\/2961111.2962594","DOI":"10.1145\/2961111.2962594"},{"volume-title":"Worker Behaviors in Competitive Crowdsourcing Tasks. Empirical Software Engineering and Measurement (ESEM), 2015 ACM\/IEEE International Symposium on (10","year":"2015","author":"Yang Ye","key":"e_1_2_1_38_1"},{"key":"e_1_2_1_39_1","doi-asserted-by":"crossref","unstructured":"Alexandre Zanatta Igor Steinmacher Leticia Machado Cleidson Souza and Rafael Prikladnicki. 2017. Barriers Faced by Newcomers to Software-Crowdsourcing Projects. IEEE Software 34 (03 2017) 37--43. https:\/\/doi.org\/10.1109\/MS.2017.32  Alexandre Zanatta Igor Steinmacher Leticia Machado Cleidson Souza and Rafael Prikladnicki. 2017. Barriers Faced by Newcomers to Software-Crowdsourcing Projects. IEEE Software 34 (03 2017) 37--43. https:\/\/doi.org\/10.1109\/MS.2017.32","DOI":"10.1109\/MS.2017.32"},{"key":"e_1_2_1_40_1","unstructured":"P\u00e4r \u00c5gerfalk Brian Fitzgerald and Klaas-Jan Stol. 2015. Software Sourcing in the Age of Open Leveraging the Unknown Workforce. https:\/\/doi.org\/10.1007\/978--3--319--17266--8  P\u00e4r \u00c5gerfalk Brian Fitzgerald and Klaas-Jan Stol. 2015. Software Sourcing in the Age of Open Leveraging the Unknown Workforce. https:\/\/doi.org\/10.1007\/978--3--319--17266--8"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3375194","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3375194","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:32:48Z","timestamp":1750199568000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3375194"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,1,4]]},"references-count":40,"journal-issue":{"issue":"GROUP","published-print":{"date-parts":[[2020,1,4]]}},"alternative-id":["10.1145\/3375194"],"URL":"https:\/\/doi.org\/10.1145\/3375194","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2020,1,4]]},"assertion":[{"value":"2020-01-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}