{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:11:28Z","timestamp":1750306288716,"version":"3.41.0"},"reference-count":46,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2016,4,25]],"date-time":"2016-04-25T00:00:00Z","timestamp":1461542400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2016,7,14]]},"abstract":"<jats:p>Crowdsourcing is increasingly being adopted to solve simple tasks such as image labeling and object tagging, as well as more complex tasks, where crowd workers collaborate in processes with interdependent steps. For the whole range of complexity, research has yielded numerous patterns for coordinating crowd workers in order to optimize crowd accuracy, efficiency, and cost. Process designers, however, often don't know which pattern to apply to a problem at hand when designing new applications for crowdsourcing.<\/jats:p>\n          <jats:p>In this article, we propose to solve this problem by systematically exploring the design space of complex crowdsourced tasks via automated recombination and auto-experimentation for an issue at hand. Specifically, we propose an approach to finding the optimal process for a given problem by defining the deep structure of the problem in terms of its abstract operators, generating all possible alternatives via the (re)combination of the abstract deep structure with concrete implementations from a Process Repository, and then establishing the best alternative via auto-experimentation.<\/jats:p>\n          <jats:p>To evaluate our approach, we implemented PPLib (pronounced \u201cPeople Lib\u201d), a program library that allows for the automated recombination of known processes stored in an easily extensible Process Repository. We evaluated our work by generating and running a plethora of process candidates in two scenarios on Amazon's Mechanical Turk followed by a meta-evaluation, where we looked at the differences between the two evaluations. Our first scenario addressed the problem of text translation, where our automatic recombination produced multiple processes whose performance almost matched the benchmark established by an expert translation. In our second evaluation, we focused on text shortening; we automatically generated 41 crowd process candidates, among them variations of the well-established Find-Fix-Verify process. While Find-Fix-Verify performed well in this setting, our recombination engine produced five processes that repeatedly yielded better results. We close the article by comparing the two settings where the Recombinator was used, and empirically show that the individual processes performed differently in the two settings, which led us to contend that there is no unifying formula, hence emphasizing the necessity for recombination.<\/jats:p>","DOI":"10.1145\/2897367","type":"journal-article","created":{"date-parts":[[2016,4,25]],"date-time":"2016-04-25T19:51:13Z","timestamp":1461613873000},"page":"1-20","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["PPLib"],"prefix":"10.1145","volume":"7","author":[{"given":"Patrick M. De","family":"Boer","sequence":"first","affiliation":[{"name":"University of Zurich, Binzm\u00fchlestrasse, Z\u00fcrich"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Abraham","family":"Bernstein","sequence":"additional","affiliation":[{"name":"University of Zurich, Binzm\u00fchlestrasse, Z\u00fcrich"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,4,25]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047203"},{"key":"#cr-split#-e_1_2_1_2_1.1","doi-asserted-by":"crossref","unstructured":"J\u00fcrgen Angele Michael Kifer and Georg Lausen. 2009. Ontologies in f-logic. In Handbook on Ontologies. 45--70. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-540-92673-3_2 10.1007\/978-3-540-92673-3_2","DOI":"10.1007\/978-3-540-92673-3_2"},{"key":"#cr-split#-e_1_2_1_2_1.2","doi-asserted-by":"crossref","unstructured":"J\u00fcrgen Angele Michael Kifer and Georg Lausen. 2009. Ontologies in f-logic. In Handbook on Ontologies. 45--70. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-540-92673-3_2","DOI":"10.1007\/978-3-540-92673-3_2"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2384616.2384663"},{"key":"e_1_2_1_4_1","volume-title":"Semantic web services language (SWSL). W3C Member Submission 9","author":"Battle Steve","year":"2005","unstructured":"Steve Battle , Abraham Bernstein , Harold Boley , Benjamin Grosof , Michael Gruninger , Richard Hull , Michael Kifer , David Martin , Sheila McIlraith , Deborah McGuinness , Su Jianwen , and Said Tabet . 2005. Semantic web services language (SWSL). W3C Member Submission 9 ( 2005 ), 1--61. Steve Battle, Abraham Bernstein, Harold Boley, Benjamin Grosof, Michael Gruninger, Richard Hull, Michael Kifer, David Martin, Sheila McIlraith, Deborah McGuinness, Su Jianwen, and Said Tabet. 2005. Semantic web services language (SWSL). W3C Member Submission 9 (2005), 1--61."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/358916.358999"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2160718.2160731"},{"volume-title":"Proceedings of the 20th International Conference on Information (ICIS\u201999)","author":"Bernstein Abraham","key":"e_1_2_1_7_1","unstructured":"Abraham Bernstein , Mark Klein , and Thomas W. Malone . 1999. The process recombinator: A tool for generating new business process ideas . In Proceedings of the 20th International Conference on Information (ICIS\u201999) . 178--192. Abraham Bernstein, Mark Klein, and Thomas W. Malone. 1999. The process recombinator: A tool for generating new business process ideas. In Proceedings of the 20th International Conference on Information (ICIS\u201999). 178--192."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2005.67"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_10_1","volume-title":"Aspects of the theory of syntax","author":"Chomsky Noam","year":"1965","unstructured":"Noam Chomsky . 1965. Aspects of the theory of syntax . MIT Press ( 1965 ). DOI:http:\/\/dx.doi.org\/10.1016\/0732-118X(86)90008-5 10.1016\/0732-118X(86)90008-5 Noam Chomsky. 1965. Aspects of the theory of syntax. MIT Press (1965). DOI:http:\/\/dx.doi.org\/10.1016\/0732-118X(86)90008-5"},{"key":"e_1_2_1_11_1","unstructured":"Martin Fowler. 2004. Inversion of control containers and the dependency injection pattern. (2004). http:\/\/martinfowler.com\/articles\/injection.html.  Martin Fowler. 2004. Inversion of control containers and the dependency injection pattern. (2004). http:\/\/martinfowler.com\/articles\/injection.html."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989331"},{"key":"e_1_2_1_13_1","volume-title":"theory and practice","author":"Ghallab Malik","year":"2004","unstructured":"Malik Ghallab , Dana Nau , and Paolo Traverso . 2004. Automated Planning , theory and practice . Elsevier ( 2004 ). Malik Ghallab, Dana Nau, and Paolo Traverso. 2004. Automated Planning, theory and practice. Elsevier (2004)."},{"key":"e_1_2_1_14_1","volume-title":"Stochastic Dilemmas: Foundations and Applications","author":"Goschin Sergiu","year":"2014","unstructured":"Sergiu Goschin . 2014 . Stochastic Dilemmas: Foundations and Applications . Rutgers University-Graduate School-New Brunswick . Sergiu Goschin. 2014. Stochastic Dilemmas: Foundations and Applications. Rutgers University-Graduate School-New Brunswick."},{"key":"e_1_2_1_15_1","volume-title":"Reengineering the Corporation","author":"Hammer Michael","year":"2006","unstructured":"Michael Hammer and James Champy . 1993. Reengineering the corporation . In Reengineering the Corporation . HarperBusiness ; Rev Upd edition ( October 10, 2006 ). Michael Hammer and James Champy. 1993. Reengineering the corporation. In Reengineering the Corporation. HarperBusiness; Rev Upd edition (October 10, 2006)."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/359863.359886"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-11915-1_31"},{"key":"e_1_2_1_18_1","volume-title":"Richard Rogers, and Brad Chissom.","author":"Kincaid Peter","year":"1975","unstructured":"Peter Kincaid , Robert Fishburne Jr , Richard Rogers, and Brad Chissom. 1975 . Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Naval Technical Training Command Millington TN Research Branch ( 1975). Peter Kincaid, Robert Fishburne Jr, Richard Rogers, and Brad Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Naval Technical Training Command Millington TN Research Branch (1975)."},{"key":"e_1_2_1_19_1","volume-title":"N.Y.) 324","author":"King Ross D.","year":"2009","unstructured":"Ross D. King , Jem Rowland , Stephen G. Oliver , Michael Young , Wayne Aubrey , Emma Byrne , Maria Liakata , Magdalena Markham , Pinar Pir , Larisa N. Soldatova , Andrew Sparkes , Kenneth E. Whelan , and Amanda Clare . 2009. The automation of science. Science (New York , N.Y.) 324 ( 2009 ), 85--89. DOI:http:\/\/dx.doi.org\/10.1126\/science.1165620 10.1126\/science.1165620 Ross D. King, Jem Rowland, Stephen G. Oliver, Michael Young, Wayne Aubrey, Emma Byrne, Maria Liakata, Magdalena Markham, Pinar Pir, Larisa N. Soldatova, Andrew Sparkes, Kenneth E. Whelan, and Amanda Clare. 2009. The automation of science. Science (New York, N.Y.) 324 (2009), 85--89. DOI:http:\/\/dx.doi.org\/10.1126\/science.1165620"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145354"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/2017399.2017406"},{"key":"e_1_2_1_23_1","unstructured":"Beatrice Liem and Yiling Chen. 2011. An iterative dual pathway structure for speech-to-text transcription. In Human Computation. 37--42.   Beatrice Liem and Yiling Chen. 2011. An iterative dual pathway structure for speech-to-text transcription. In Human Computation. 37--42."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866040"},{"key":"e_1_2_1_25_1","volume-title":"Conference on Human Computation and Crowdsourcing. 159--170","author":"Livshits Benjamin","year":"2014","unstructured":"Benjamin Livshits and Todd Mytkowicz . 2014 . Saving money while polling with interpoll using power analysis . In Conference on Human Computation and Crowdsourcing. 159--170 . Benjamin Livshits and Todd Mytkowicz. 2014. Saving money while polling with interpoll using power analysis. In Conference on Human Computation and Crowdsourcing. 159--170."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/174666.174668"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.5555\/2909990.2909999"},{"volume-title":"Organizing Business Knowledge: The MIT Process Handbook","author":"Malone Thomas W.","key":"e_1_2_1_28_1","unstructured":"Thomas W. Malone , Kevin Crowston , Jintae Lee , Brian Pentland , Chrysanthos Dellarocas , George Wyner , John Quimby , Charley Osborn , Abraham Bernstein , George Herman , Mark Klein , and Elissa O\u2019Donnell . 2003. Tools for inventing organizations: Toward a handbook of organizational processes . In Thomas W. Malone, Kevin Crowston, and George Herman (Eds.). Organizing Business Knowledge: The MIT Process Handbook . Cambridge, MA : MIT Press . Thomas W. Malone, Kevin Crowston, Jintae Lee, Brian Pentland, Chrysanthos Dellarocas, George Wyner, John Quimby, Charley Osborn, Abraham Bernstein, George Herman, Mark Klein, and Elissa O\u2019Donnell. 2003. Tools for inventing organizations: Toward a handbook of organizational processes. In Thomas W. Malone, Kevin Crowston, and George Herman (Eds.). Organizing Business Knowledge: The MIT Process Handbook. Cambridge, MA: MIT Press."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/EMR.2010.5559142"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35386-4_10"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35386-4_10"},{"volume-title":"Human Problem Solving","author":"Newell Allen","key":"e_1_2_1_32_1","unstructured":"Allen Newell and Herbert Alexander Simon . 1972. Human Problem Solving . Englewood Cliffs, NJ : Prentice-Hall . Allen Newell and Herbert Alexander Simon. 1972. Human Problem Solving. Englewood Cliffs, NJ: Prentice-Hall."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2307636.2307668"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139173438"},{"key":"e_1_2_1_35_1","series-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 313--320. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-642-17749-1_22","volume-title":"Auto-experimentation of KDD workflows based on ontological planning","author":"Serban Floarea","unstructured":"Floarea Serban . 2010. Auto-experimentation of KDD workflows based on ontological planning . In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 313--320. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-642-17749-1_22 10.1007\/978-3-642-17749-1_22 Floarea Serban. 2010. Auto-experimentation of KDD workflows based on ontological planning. In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 313--320. DOI:http:\/\/dx.doi.org\/10.1007\/978-3-642-17749-1_22"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2480741.2480748"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"volume-title":"The New Science of Management Decision Rev","author":"Simon Herbert Alexander","key":"e_1_2_1_38_1","unstructured":"Herbert Alexander Simon . 1977. The New Science of Management Decision Rev . Englewood Cliffs, NJ : Prentice-Hall . Herbert Alexander Simon. 1977. The New Science of Management Decision Rev. Englewood Cliffs, NJ: Prentice-Hall."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2746353"},{"key":"e_1_2_1_40_1","volume-title":"Eppinger","author":"Ulrich Karl T.","year":"1995","unstructured":"Karl T. Ulrich and Steven D . Eppinger . 1995 . Product design and development. Tata McGraw-Hill Education ( 1995). Karl T. Ulrich and Steven D. Eppinger. 1995. Product design and development. Tata McGraw-Hill Education (1995)."},{"key":"e_1_2_1_41_1","volume-title":"HCOMP","author":"Verroios Vasilis","year":"2014","unstructured":"Vasilis Verroios and Michael S. Bernstein . 2014. Context trees: Crowdsourcing global understanding from local views . In HCOMP 2014 . Stanford InfoLab. Vasilis Verroios and Michael S. Bernstein. 2014. Context trees: Crowdsourcing global understanding from local views. In HCOMP 2014. Stanford InfoLab."},{"volume-title":"Understanding Computers and Cognition: A New Foundation for Design","author":"Winograd Terry","key":"e_1_2_1_42_1","unstructured":"Terry Winograd and Fernando Flores . 1986. Understanding Computers and Cognition: A New Foundation for Design . Norwood, NJ : Ablex Pub. Corp . Terry Winograd and Fernando Flores. 1986. Understanding Computers and Cognition: A New Foundation for Design. Norwood, NJ: Ablex Pub. Corp."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/362575.362577"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1016091900743"},{"key":"e_1_2_1_45_1","volume-title":"Office automation: Revolution or evolution&quest","author":"Zisman Michael D.","year":"1978","unstructured":"Michael D. Zisman . 1978. Office automation: Revolution or evolution&quest ; Sloan Management Review 19, 3 ( 1978 ), 1--16. Michael D. Zisman. 1978. Office automation: Revolution or evolution&quest; Sloan Management Review 19, 3 (1978), 1--16."}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2897367","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2897367","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:39:02Z","timestamp":1750221542000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2897367"}},"subtitle":["Toward the Automated Generation of Crowd Computing Programs Using Process Recombination and Auto-Experimentation"],"short-title":[],"issued":{"date-parts":[[2016,4,25]]},"references-count":46,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,7,14]]}},"alternative-id":["10.1145\/2897367"],"URL":"https:\/\/doi.org\/10.1145\/2897367","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"type":"print","value":"2157-6904"},{"type":"electronic","value":"2157-6912"}],"subject":[],"published":{"date-parts":[[2016,4,25]]},"assertion":[{"value":"2015-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-04-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}