{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T19:56:45Z","timestamp":1777406205655,"version":"3.51.4"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2017,12,6]],"date-time":"2017-12-06T00:00:00Z","timestamp":1512518400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1351131"],"award-info":[{"award-number":["1351131"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100005492","name":"Stanford University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100005492","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2017,12,6]]},"abstract":"<jats:p>The dominant crowdsourcing infrastructure today is the workflow, which decomposes goals into small independent tasks. However, complex goals such as design and engineering have remained stubbornly difficult to achieve with crowdsourcing workflows. Is this due to a lack of imagination, or a more fundamental limit? This paper explores this question through in-depth case studies of 22 workers across six workflow-based crowd teams, each pursuing a complex and interdependent web development goal. We used an inductive mixed method approach to analyze behavior trace data, chat logs, survey responses and work artifacts to understand how workers enacted and adapted the crowdsourcing workflows. Our results indicate that workflows served as useful coordination artifacts, but in many cases critically inhibited crowd workers from pursuing real-time adaptations to their work plans. However, the CSCW and organizational behavior literature argues that all sufficiently complex goals require open-ended adaptation. If complex work requires adaptation but traditional static crowdsourcing workflows can't support it, our results suggest that complex work may remain a fundamental limitation of workflow-based crowdsourcing infrastructures.<\/jats:p>","DOI":"10.1145\/3134724","type":"journal-article","created":{"date-parts":[[2017,12,6]],"date-time":"2017-12-06T21:23:15Z","timestamp":1512595395000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":62,"title":["No Workflow Can Ever Be Enough"],"prefix":"10.1145","volume":"1","author":[{"given":"Daniela","family":"Retelny","sequence":"first","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}]},{"given":"Michael S.","family":"Bernstein","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}]},{"given":"Melissa A.","family":"Valentine","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}]}],"member":"320","published-online":{"date-parts":[[2017,12,6]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Third AAAI Conference on Human Computation and Crowdsourcing (AAAI '15)","author":"Agapie Elena","year":"2015"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047203"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.2307\/2392320"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1287\/orsc.1050.0149"},{"key":"e_1_2_1_5_1","volume-title":"or, Linux and The Nature of the Firm. Yale Law Journal","author":"Benkler Yochai","year":"2002"},{"key":"e_1_2_1_6_1","unstructured":"Yochai Benkler. 2013. Peer Production and Cooperation. In Handbook on the Economics of the Internet. 1--29.  Yochai Benkler. 2013. Peer Production and Cooperation. In Handbook on the Economics of the Internet. 1--29."},{"key":"e_1_2_1_7_1","volume-title":"Peer Production: A Modality of Collective Intelligence. In Handbook of Collective Intelligence, Thomas W","author":"Benkler Yochai","year":"2015"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047201"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_10_1","volume-title":"Handbook of Collective Intelligence","author":"Bigham Jeffrey P."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866080"},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","first-page":"1281","DOI":"10.2307\/3069401","article-title":"The incident command system: High-reliability organizing for complex and volatile task environments","volume":"44","author":"Bigley Gregory A.","year":"2001","journal-title":"Academy of Management Journal"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.2307\/2393807"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820023"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858512"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466265"},{"key":"e_1_2_1_17_1","unstructured":"Richard Daft. 2015. Organization Theory and Design. Cengage Learning.  Richard Daft. 2015. Organization Theory and Design. Cengage Learning."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI '10)","author":"Dai Peng"},{"key":"e_1_2_1_19_1","unstructured":"Pete Deemer Gabrielle Benefield Craig Larman and Bas Vodde. 2012. The Scrum Primer: A Lightweight Guide to the Theory and Practice of Scrum. (2012) 20 pages.  Pete Deemer Gabrielle Benefield Craig Larman and Bas Vodde. 2012. The Scrum Primer: A Lightweight Guide to the Theory and Practice of Scrum. (2012) 20 pages."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_1_21_1","volume-title":"Teaming: How organizations learn, innovate, and compete in the knowledge economy","author":"Edmondson Amy C.","year":"2012"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5465\/amr.1989.4308385"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.1060.0526"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984532"},{"key":"e_1_2_1_25_1","volume-title":"Strauss","author":"Glaser Barney G.","year":"1967"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819942"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/302405.302455"},{"key":"e_1_2_1_28_1","article-title":"The Rise of Crowdsourcing","volume":"14","author":"Howe Jeff","year":"2006","journal-title":"Wired Magazine"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979111"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998196"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145357"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441922"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/203330.203345"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145354"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702565"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380122"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647349"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v25i1.8092"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.2307\/2391211"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence (AAAI'12)","author":"Lin Christopher H."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837907"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866040"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819979"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/174666.174668"},{"key":"e_1_2_1_47_1","volume-title":"Simon","author":"March James G.","year":"1958"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858169"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047198"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.5465\/19416520903047533"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979148"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647409"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF01405730"},{"key":"e_1_2_1_54_1","volume-title":"Holacracy: The new management system for a rapidly changing world","author":"Robertson Brian J.","year":"2015"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998300"},{"key":"e_1_2_1_56_1","volume-title":"The Reflective Practitioner: How Professionals Think In Action","author":"Schon Donald A."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568249"},{"key":"e_1_2_1_58_1","volume-title":"Human-Machine Reconfigurations: Plans and Situated Actions","author":"Suchman Lucy","edition":"2"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858121"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858108"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531680"},{"key":"e_1_2_1_62_1","doi-asserted-by":"crossref","volume-title":"Organizations in Action: Social Science Bases of Administrative Theory","author":"Thompson James D.","DOI":"10.4324\/9781315125930"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1287\/orsc.2014.0947"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025811"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.2307\/2094477"},{"key":"e_1_2_1_66_1","volume-title":"reCAPTCHA: Human-Based Character Recognition via Web Security Measures. Science 321, 5895","author":"von Ahn Luis","year":"2008"},{"key":"e_1_2_1_67_1","unstructured":"Daniel S. Weld Christopher H. Lin and Jonathan Bragg. 2015. Artificial Intelligence and Collective Intelligence. In Handbook of Collective Intelligence. 89--114.  Daniel S. Weld Christopher H. Lin and Jonathan Bragg. 2015. Artificial Intelligence and Collective Intelligence. In Handbook of Collective Intelligence. 89--114."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327051hci0301_2"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2835201"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820025"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979147"},{"key":"e_1_2_1_72_1","volume-title":"2nd International Workshop on Crowdsourcing in Software Engineering. 45--46","author":"Zhao Mengyao"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3134724","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3134724","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3134724","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:11:24Z","timestamp":1750212684000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3134724"}},"subtitle":["How Crowdsourcing Workflows Constrain Complex Work"],"short-title":[],"issued":{"date-parts":[[2017,12,6]]},"references-count":72,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2017,12,6]]}},"alternative-id":["10.1145\/3134724"],"URL":"https:\/\/doi.org\/10.1145\/3134724","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,12,6]]},"assertion":[{"value":"2017-12-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}