{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T18:00:31Z","timestamp":1776103231538,"version":"3.50.1"},"reference-count":90,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T00:00:00Z","timestamp":1541030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"FY2018 Research Challenge Grant award"},{"name":"J. Wayne and Kathy Richards Faculty Fellowship in Engineering"},{"name":"Research Grant from Leidos Laboratories"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:p>Traditional employment usually provides mechanisms for workers to improve their skills to access better opportunities. However, crowd work platforms like Amazon Mechanical Turk (AMT) generally do not support skill development (i.e., becoming faster and better at work). While researchers have started to tackle this problem, most solutions are dependent on experts or requesters willing to help. However, requesters generally lack the necessary knowledge, and experts are rare and expensive. To further facilitate crowd workers' skill growth, we present Crowd Coach, a system that enables workers to receive peer coaching while on the job. We conduct a field experiment and real world deployment to study Crowd Coach in the wild. Hundreds of workers used Crowd Coach in a variety of tasks, including writing, doing surveys, and labeling images. We find that Crowd Coach enhances workers' speed without sacrificing their work quality, especially in audio transcription tasks. We posit that peer coaching systems hold potential for better supporting crowd workers' skill development while on the job. We finish with design implications from our research.<\/jats:p>","DOI":"10.1145\/3274306","type":"journal-article","created":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T21:21:27Z","timestamp":1541107287000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":37,"title":["Crowd Coach"],"prefix":"10.1145","volume":"2","author":[{"given":"Chun-Wei","family":"Chiang","sequence":"first","affiliation":[{"name":"West Virginia University, Morgantown, WV, USA"}]},{"given":"Anna","family":"Kasunic","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"given":"Saiph","family":"Savage","sequence":"additional","affiliation":[{"name":"Universidad Nacional Autonoma de Mexico (UNAM), Mexico City, Mexico"}]}],"member":"320","published-online":{"date-parts":[[2018,11]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025974"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2685553.2699021"},{"key":"e_1_2_2_3_1","volume-title":"Classroom assessment techniques: A handbookfor college teachers","author":"Angelo Thomas A","year":"1993","unstructured":"Thomas A Angelo and K Patricia Cross. 1993. Classroom assessment techniques: A handbookfor college teachers. San Francisco: Jossey-Bas (1993)."},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1300\/J075v07n03_05"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026057"},{"key":"e_1_2_2_6_1","first-page":"543","article-title":"Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers","volume":"37","author":"Berg Janine","year":"2015","unstructured":"Janine Berg. 2015. Income security in the on-demand economy: Findings and policy lessons from a survey of crowdworkers. Comparative Labor Law and Policy Journal , Vol. 37 (2015), 543.","journal-title":"Comparative Labor Law and Policy Journal"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132525.3134821"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3058555.3058562"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1108\/EUM0000000005548"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1080\/0260293990240405"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1177\/1354856507084420"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.2196\/mental.4822"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858198"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","unstructured":"Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design .Morgan Kaufmann.","DOI":"10.5555\/1526229"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858237"},{"key":"e_1_2_2_16_1","volume-title":"Second AAAI Conference on Human Computation and Crowdsourcing .","author":"Callison-Burch Chris","year":"2014","unstructured":"Chris Callison-Burch. 2014. Crowd-workers: Aggregating information across turkers to help them find higher paying work. In Second AAAI Conference on Human Computation and Crowdsourcing ."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2003.1264740"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.hrmr.2006.03.012"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858178"},{"key":"e_1_2_2_20_1","volume-title":"Risks and Rewards of Crowdsourcing Marketplaces","author":"Chandler Jesse","unstructured":"Jesse Chandler, Gabriele Paolacci, and Pam Mueller. 2013. Risks and Rewards of Crowdsourcing Marketplaces .Springer New York, New York, NY, 377--392."},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3170427.3188660"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675251"},{"key":"e_1_2_2_23_1","unstructured":"Xuefei Nancy Deng and KD Joshi. 2013. Is Crowdsourcing a Source of Worker Empowerment or Exploitation? Understanding Crowd Workers' Perceptions of Crowdsourcing Career. (2013)."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741685"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134675"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557217"},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1640233.1640260"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12599-016-0438-0"},{"key":"e_1_2_2_30_1","first-page":"538","article-title":"The value of lifelong learning: key element in professional career development","volume":"99","author":"Duyff Roberta L","year":"1999","unstructured":"Roberta L Duyff. 1999. The value of lifelong learning: key element in professional career development. Journal of the Academy of Nutrition and Dietetics , Vol. 99, 5 (1999), 538.","journal-title":"Journal of the Academy of Nutrition and Dietetics"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979413"},{"key":"e_1_2_2_32_1","volume-title":"Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of educational research","author":"Falchikov Nancy","year":"2000","unstructured":"Nancy Falchikov and Judy Goldfinch. 2000. Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of educational research , Vol. 70, 3 (2000), 287--322."},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702279"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174023"},{"key":"e_1_2_2_35_1","volume-title":"Proceedings of the Artificial Intelligence and Interactive Digital Entertainment","author":"Harrison Brent","year":"2016","unstructured":"Brent Harrison and Mark O Riedl. 2016. Learning from stories: using crowdsourced narratives to train virtual agents. Proceedings of the Artificial Intelligence and Interactive Digital Entertainment (2016), 183--189."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.3102\/003465430298487"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556420.2556484"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1037\/0021-9010.86.6.1232"},{"key":"e_1_2_2_39_1","volume-title":"Civic engagement in the digital age. Pew Internet & American Life Project","author":"Horrigan John B.","year":"2016","unstructured":"John B. Horrigan. 2016. Civic engagement in the digital age. Pew Internet & American Life Project (2016)."},{"key":"e_1_2_2_40_1","article-title":"Social media, civic engagement, and the slacktivism hypothesis: Lessons from Mexico's \u201cEl Bronco","volume":"70","author":"Howard Philip N","year":"2016","unstructured":"Philip N Howard, Saiph Savage, Claudia Flores Saviaga, Carlos Toxtli, and Andr\u00e9s Monroy-Hem\u00e1ndez. 2016. Social media, civic engagement, and the slacktivism hypothesis: Lessons from Mexico's \u201cEl Bronco\u201d. Journal of International Affairs , Vol. 70, 1 (2016).","journal-title":"Journal of International Affairs"},{"key":"e_1_2_2_41_1","volume-title":"Fourth AAAI Conference on Human Computation and Crowdsourcing .","author":"Kenneth Huang Ting-Hao","year":"2016","unstructured":"Ting-Hao Kenneth Huang, Walter S Lasecki, Amos Azaria, and Jeffrey P Bigham. 2016. \" Is There Anything Else I Can Help You With?\" Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. In Fourth AAAI Conference on Human Computation and Crowdsourcing ."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858327"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470742"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858592"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3058555.3058562"},{"key":"e_1_2_2_46_1","first-page":"1","article-title":"More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk","volume":"11","author":"Kaufmann Nicolas","year":"2011","unstructured":"Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk.. In AMCIS, Vol. 11. 1--11.","journal-title":"AMCIS"},{"key":"e_1_2_2_47_1","volume-title":"For better or for worse? An analysis of how flexible working practices influence employees' perceptions of job quality. The International Journal of Human Resource Management","author":"Kelliher Clare","year":"2008","unstructured":"Clare Kelliher and Deirdre Anderson. 2008. For better or for worse? An analysis of how flexible working practices influence employees' perceptions of job quality. The International Journal of Human Resource Management, , Vol. 19, 3 (2008), 419--431."},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_2_49_1","volume-title":"The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological bulletin","author":"Kluger Avraham N","year":"1996","unstructured":"Avraham N Kluger and Angelo DeNisi. 1996. The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological bulletin , Vol. 119, 2 (1996), 254."},{"key":"e_1_2_2_50_1","volume-title":"The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science","author":"Koedinger Kenneth R","year":"2012","unstructured":"Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science , Vol. 36, 5 (2012), 757--798."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145354"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505057"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2724660.2724670"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818052.2855527"},{"key":"e_1_2_2_55_1","volume-title":"Self-regulation through goal setting. Organizational behavior and human decision processes","author":"Latham Gary P","year":"1991","unstructured":"Gary P Latham and Edwin A Locke. 1991. Self-regulation through goal setting. Organizational behavior and human decision processes , Vol. 50, 2 (1991), 212--247."},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1353\/jhe.2003.0045"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531663"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/1809400.1809422"},{"key":"e_1_2_2_59_1","volume-title":"Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476","author":"Miller Alexander H","year":"2017","unstructured":"Alexander H Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. 2017. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476 (2017)."},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025687"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11251-008-9053-x"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145352"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441886"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047198"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.5555\/1857999.1858023"},{"key":"e_1_2_2_66_1","doi-asserted-by":"crossref","unstructured":"Gabriele Paolacci Jesse Chandler and Panagiotis G Ipeirotis. 2010. Running experiments on Amazon Mechanical Turk. (2010).","DOI":"10.1017\/S1930297500002205"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11251-015-9353-x"},{"key":"e_1_2_2_68_1","first-page":"52","article-title":"A review of employee motivation theories and their implications for employee retention within organizations","volume":"5","author":"Ramlall Sunil","year":"2004","unstructured":"Sunil Ramlall. 2004. A review of employee motivation theories and their implications for employee retention within organizations. Journal of American Academy of Business , Vol. 5, 1\/2 (2004), 52--63.","journal-title":"Journal of American Academy of Business"},{"key":"e_1_2_2_69_1","unstructured":"Alex Rosenblat and Luke Stark. 2016. Algorithmic labor and information asymmetries: A case study of Uber's drivers. (2016)."},{"key":"e_1_2_2_70_1","volume-title":"Who are the turkers? Worker demographics in amazon mechanical turk. Department of Informatics","author":"Ross Joel","year":"2009","unstructured":"Joel Ross, Andrew Zaldivar, Lilly Irani, and Bill Tomlinson. 2009. Who are the turkers? Worker demographics in amazon mechanical turk. Department of Informatics, University of California, Irvine, USA, Tech. Rep (2009)."},{"key":"e_1_2_2_71_1","volume-title":"Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist","author":"Ryan Richard M","year":"2000","unstructured":"Richard M Ryan and Edward L Deci. 2000. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist , Vol. 55, 1 (2000), 68."},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819985"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/2641580.2641614"},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/1869086.1869100"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837891"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858121"},{"key":"e_1_2_2_77_1","volume-title":"CHI Workshop on Crowdsourcing and Human Computation .","author":"Thies William","year":"2011","unstructured":"William Thies, Aishwarya Ratan, and James Davis. 2011. Paid crowdsourcing as a vehicle for global development. In CHI Workshop on Crowdsourcing and Human Computation ."},{"key":"e_1_2_2_78_1","first-page":"29","article-title":"CritViz: Web-based software supporting peer critique in large creative classrooms","volume":"15","author":"Tinapple David","year":"2013","unstructured":"David Tinapple, Loren Olson, and John Sadauskas. 2013. CritViz: Web-based software supporting peer critique in large creative classrooms. Bulletin of the IEEE Technical Committee on Learning Technology , Vol. 15, 1 (2013), 29.","journal-title":"Bulletin of the IEEE Technical Committee on Learning Technology"},{"key":"e_1_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1548-1425.2008.00031.x"},{"key":"e_1_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2556996"},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2017.1294194"},{"key":"e_1_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675219"},{"key":"e_1_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998234"},{"key":"e_1_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/2876034.2876042"},{"key":"e_1_2_2_85_1","unstructured":"Bill Williamson. 1998. Lifeworlds and Learning. Essays in the Theory Philosophy and Practice of Lifelong Learning. (1998)."},{"key":"e_1_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/2901790.2901820"},{"key":"e_1_2_2_87_1","doi-asserted-by":"crossref","unstructured":"Dongwook Yoon Nicholas Chen Bernie Randles Amy Cheatle Corinna E. L\u00f6ckenhoff Steven J. Jackson Abigail Sellen and Franccois Guimbreti\u00e8re. 2016. RichReview","DOI":"10.1145\/2818048.2819951"},{"key":"e_1_2_2_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819951"},{"key":"e_1_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531718"},{"key":"e_1_2_2_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2481311"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274306","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274306","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:06Z","timestamp":1750207446000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274306"}},"subtitle":["Peer Coaching for Crowd Workers' Skill Growth"],"short-title":[],"issued":{"date-parts":[[2018,11]]},"references-count":90,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1145\/3274306"],"URL":"https:\/\/doi.org\/10.1145\/3274306","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11]]},"assertion":[{"value":"2018-11-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}