{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T07:08:39Z","timestamp":1781334519167,"version":"3.54.1"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T00:00:00Z","timestamp":1541030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:p>Real-time collaboration between a requester and crowd workers expands the scope of tasks that crowdsourcing can be used for by letting requesters and crowd workers interactively create various artifacts (e.g., a sketch prototype, writing, or program code). In such systems, it is increasingly common to allow requesters to verbally describe their requests, receive responses from workers, and provide immediate and continuous feedback to enhance the overall outcome of the two groups' real-time collaboration. This work is motivated by the lack of a deep understanding of the challenges that end users of such systems face in their communication with workers and the need of design implications that can address such challenges for other similar systems. In this paper, we investigate how requesters verbally communicate and collaborate with crowd workers to solve a complex task. Using a crowd-powered UI design tool, we conducted a qualitative user study to explore how requesters with varying expertise communicate and collaborate with crowd workers. Our work also identifies the unique challenges that collaborative crowdsourcing systems pose: potential expertise differences between requesters and crowd workers, the asymmetry of two-way communication (e.g., speech versus text), and the shared artifact's concurrent modification by two disparate groups. Finally, we make design recommendations that can inform the design of future real-time collaboration processes in crowdsourcing systems.<\/jats:p>","DOI":"10.1145\/3274373","type":"journal-article","created":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T21:21:27Z","timestamp":1541107287000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Exploring Real-Time Collaboration in Crowd-Powered Systems Through a UI Design Tool"],"prefix":"10.1145","volume":"2","author":[{"given":"Sang Won","family":"Lee","sequence":"first","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rebecca","family":"Krosnick","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sun Young","family":"Park","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brandon","family":"Keelean","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sach","family":"Vaidya","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephanie D.","family":"O'Keefe","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Walter S.","family":"Lasecki","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,11]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2017. Amazon's Mechanical Turk. http:\/\/www.mturk.com."},{"key":"e_1_2_1_2_1","unstructured":"2017. Voice Qualities. http:\/\/www.ncvs.org\/ncvs\/tutorials\/voiceprod\/tutorial\/quality.html."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3059454.3059477"},{"key":"e_1_2_1_4_1","volume-title":"W3C 91","author":"Bergkvist Adam","year":"2012","unstructured":"Adam Bergkvist, Daniel C Burnett, Cullen Jennings, and Anant Narayanan. 2012. Webrtc 1.0: Real-time communication between browsers. Working draft, W3C 91 (2012)."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047201"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2791285"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","unstructured":"Jeffrey P. Bigham Chandrika Jayant Hanjie Ji Greg Little Andrew Miller Robert C. Miller Robin Miller Aubrey Tatarowicz Brandyn White Samual White and Tom Yeh. 2010. VizWiz: nearly real-time answers to visual questions .In User Interface Software and Technology (UIST). 333--342. 10.1145\/1866029.1866080","DOI":"10.1145\/1866029.1866080"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025972"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1037\/1040-3590.6.4.284"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Herbert H Clark Susan E Brennan et al. 1991. Grounding in communication. Perspectives on socially shared cognition13 1991 (1991) 127--149.","DOI":"10.1037\/10096-006"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/99977.99987"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/358916.358947"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1031607.1031687"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1177\/0261927X04269589"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815585.2815729"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the Sixth AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2018","author":"Gouravajhala Sai R.","year":"2018","unstructured":"Sai R. Gouravajhala, Jinyeong Yim, Karthik Desingh, Yanda Huang, Odest Chadwicke Jenkins, and Walter S. Lasecki.2018. EURECA: Enhanced Understanding of Real Environments via Crowd Assistance. In Proceedings of the Sixth AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2018, Z\u00fcrich, Switzerland, July 5--8, 2018. 31--40. https:\/\/aaai.org\/ocs\/index.php\/HCOMP\/HCOMP18\/paper\/view\/17933"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/192844.193011"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.5555\/1241867.1241883"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1021271517844"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/587078.587086"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025642"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557048"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1037\/1076-898X.5.2.205"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1037\/0021-9010.86.6.1232"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1080\/07370024.2017.1344126"},{"key":"e_1_2_1_27_1","volume-title":"Fourth AAAI Conference on Human Computation and Crowdsourcing.","author":"Kenneth Huang Ting-Hao","year":"2016","unstructured":"Ting-Hao Kenneth Huang, Walter S. Lasecki, Amos Azaria, and Jeffrey P Bigham. 2016. \"Is There Anything Else I Can Help You With?\" Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. In Fourth AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1037\/0096-3445.116.1.26"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.5555\/1858681.1858689"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327051HCI1403_2"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the AAAI Conference on Human Computation (HCOMP","volume":"17","author":"Kaur Harmanpreet","year":"2017","unstructured":"Harmanpreet Kaur, Mitchell Gordon, Yiwei Yang, Jeffrey P. Bigham, Jaime Teevan, Ece Kamar, and Walter S. Lasecki.2017. Crowdmask: Using crowds to preserve privacy in crowd-powered systems via progressive filtering. In Proceedings of the AAAI Conference on Human Computation (HCOMP 2017)., HCOMP, Vol. 17."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327051HCI1812_2"},{"key":"e_1_2_1_35_1","volume-title":"Lasecki and Jeffrey P Bigham","author":"Walter","year":"2013","unstructured":"Walter S. Lasecki and Jeffrey P Bigham. 2013. Interactive crowds: Real-time crowdsourcing and crowd agents. In Handbook of human computation. Springer, 509--521."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647367"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702605"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702565"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047200"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2513383.2517033"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501988.2502057"},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","unstructured":"Matthew Lease Jessica Hullman Jeffrey P. Bigham Michael S. Bernstein Juho Kim Walter S. Lasecki Saeideh Bakhshi Tanushree Mitra and Robert C Miller. 2013. Mechanical turk is not anonymous. (2013).","DOI":"10.2139\/ssrn.2228728"},{"key":"e_1_2_1_44_1","volume-title":"Work-in-Progress, the AAAI Conference on Human Computation (HCOMP","author":"Lee Sang Won","year":"2017","unstructured":"Sang Won Lee, Yan Chen, and Walter S Lasecki. 2017. The Need for Real-Time Crowd Generation of Task Lists from Speech. In Work-in-Progress, the AAAI Conference on Human Computation (HCOMP 2017)."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126594.3126595"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600159"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174041"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1031607.1031648"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858169"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/169059.169166"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15327051HCI1504_1"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.pragma.2017.01.013"},{"key":"e_1_2_1_53_1","volume-title":"Reed","author":"Reid Fraser J. M.","year":"2007","unstructured":"Fraser J. M. Reid and Susan E. Reed. 2007. Conversational grounding and visual access in collaborative design. CoDesign3, 2 (2007), 111--122."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647409"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998300"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998332"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1031607.1031655"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/1718918.1718969"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1016\/0020-7373(91)90039-A"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025811"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819953"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274373","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274373","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:35Z","timestamp":1750207475000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274373"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,11]]},"references-count":60,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1145\/3274373"],"URL":"https:\/\/doi.org\/10.1145\/3274373","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11]]},"assertion":[{"value":"2018-11-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}