{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,19]],"date-time":"2026-02-19T02:56:39Z","timestamp":1771469799650,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":53,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,6,26]],"date-time":"2022-06-26T00:00:00Z","timestamp":1656201600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"US Department of Defense","award":["Laboratory for Analytic Sciences at North Carolina State University"],"award-info":[{"award-number":["Laboratory for Analytic Sciences at North Carolina State University"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,6,26]]},"DOI":"10.1145\/3501247.3531555","type":"proceedings-article","created":{"date-parts":[[2022,6,24]],"date-time":"2022-06-24T22:41:09Z","timestamp":1656110469000},"page":"66-74","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Enhancing Creativity as Innovation via Asynchronous Crowdwork"],"prefix":"10.1145","author":[{"given":"Pradeep Kumar","family":"Murukannaiah","sequence":"first","affiliation":[{"name":"Intelligent Systems, Delft University of Technology, Netherlands"}]},{"given":"Nirav","family":"Ajmeri","sequence":"additional","affiliation":[{"name":"Computer Science, University of Bristol, United Kingdom"}]},{"given":"Munindar P.","family":"Singh","sequence":"additional","affiliation":[{"name":"Computer Science, North Carolina State University, USA"}]}],"member":"320","published-online":{"date-parts":[[2022,6,26]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"123test BV. 2018. DISC Personality Test. https:\/\/www.123test.com\/disc-personality-test\/.  123test BV. 2018. DISC Personality Test. https:\/\/www.123test.com\/disc-personality-test\/."},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557158"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1002\/jocb.002"},{"key":"e_1_3_2_2_4_1","volume-title":"Weaving the Web: The Original Design and Ultimate Destiny of the World Wide Web","author":"Berners-Lee Tim","unstructured":"Tim Berners-Lee . 1999. Weaving the Web: The Original Design and Ultimate Destiny of the World Wide Web . Harper Business , New York . Tim Berners-Lee. 1999. Weaving the Web: The Original Design and Ultimate Destiny of the World Wide Web. Harper Business, New York."},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2014.6912266"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(98)00055-1"},{"key":"e_1_3_2_2_7_1","volume-title":"Steven Mithen (Ed.)","author":"Boden A.","unstructured":"Margaret\u00a0 A. Boden . 2005. What is creativity?In Creativity in human evolution and prehistory , Steven Mithen (Ed.) . Routledge , London , 27\u201355. Margaret\u00a0A. Boden. 2005. What is creativity?In Creativity in human evolution and prehistory, Steven Mithen (Ed.). Routledge, London, 27\u201355."},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2016.36"},{"key":"e_1_3_2_2_9_1","volume-title":"Proceedings of the 25th International World Wide Web Conference. ACM, Montr\u00e9al, 903\u2013914","author":"K.","unstructured":"Amit\u00a0 K. Chopra and Munindar\u00a0P. Singh. 2016. From Social Machines to Social Protocols: Software Engineering Foundations for Sociotechnical Systems . In Proceedings of the 25th International World Wide Web Conference. ACM, Montr\u00e9al, 903\u2013914 . https:\/\/doi.org\/10.1145\/2872427.2883018 Amit\u00a0K. Chopra and Munindar\u00a0P. Singh. 2016. From Social Machines to Social Protocols: Software Engineering Foundations for Sociotechnical Systems. In Proceedings of the 25th International World Wide Web Conference. ACM, Montr\u00e9al, 903\u2013914. https:\/\/doi.org\/10.1145\/2872427.2883018"},{"key":"e_1_3_2_2_10_1","volume-title":"Ordinal Methods for Behavioral Data Analysis","author":"Cliff Norman","unstructured":"Norman Cliff . 2014. Ordinal Methods for Behavioral Data Analysis . Psychology Press , New York . Norman Cliff. 2014. Ordinal Methods for Behavioral Data Analysis. Psychology Press, New York."},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.53.3.497"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1080\/00401706.1964.10490181"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447535.3462482"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196709.3196732"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.20982\/tqmp.08.1.p023"},{"key":"e_1_3_2_2_16_1","article-title":"Understanding Crowdsourcing Systems from a Multiagent Perspective and Approach","volume":"13","author":"Jiang Jiuchuan","year":"2018","unstructured":"Jiuchuan Jiang , Bo An , Yichuan Jiang , Donghui Lin , Zhan Bu , Jie Cao , and Zhifeng Hao . 2018 . Understanding Crowdsourcing Systems from a Multiagent Perspective and Approach . ACM Transactions on Autonomous and Adaptive Systems (TAAS) 13 , 2(2018), 8:1\u20138:32. https:\/\/doi.org\/10.1145\/3226028 Jiuchuan Jiang, Bo An, Yichuan Jiang, Donghui Lin, Zhan Bu, Jie Cao, and Zhifeng Hao. 2018. Understanding Crowdsourcing Systems from a Multiagent Perspective and Approach. ACM Transactions on Autonomous and Adaptive Systems (TAAS) 13, 2(2018), 8:1\u20138:32. https:\/\/doi.org\/10.1145\/3226028","journal-title":"ACM Transactions on Autonomous and Adaptive Systems (TAAS)"},{"key":"e_1_3_2_2_17_1","volume-title":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence(IJCAI\u201916)","author":"Kamar Ece","year":"2016","unstructured":"Ece Kamar . 2016 . Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence . In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence(IJCAI\u201916) . AAAI Press, New York, 4070\u20134073. Ece Kamar. 2016. Directions in Hybrid Intelligence: Complementing AI Systems with Human Intelligence. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence(IJCAI\u201916). AAAI Press, New York, 4070\u20134073."},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2017.83"},{"key":"e_1_3_2_2_19_1","volume-title":"Proceedings of the Seventeenth Americas Conference on Information Systems. Association for Information Systems, Detroit, 1\u201311","author":"Kaufmann Nicolas","year":"2011","unstructured":"Nicolas Kaufmann , Thimo Schulze , and Daniel Veit . 2011 . More than fun and money. Worker motivation in crowdsourcing: A study on Mechanical Turk . In Proceedings of the Seventeenth Americas Conference on Information Systems. Association for Information Systems, Detroit, 1\u201311 . Nicolas Kaufmann, Thimo Schulze, and Daniel Veit. 2011. More than fun and money. Worker motivation in crowdsourcing: A study on Mechanical Turk. In Proceedings of the Seventeenth Americas Conference on Information Systems. Association for Information Systems, Detroit, 1\u201311."},{"key":"e_1_3_2_2_20_1","volume-title":"Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP). AAAI","author":"Kulkarni Anand","year":"2014","unstructured":"Anand Kulkarni , Prayag Narula , David Rolnitzky , and Nathan Kontny . 2014 . Wish: Amplifying Creative Ability with Expert Crowds . In Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP). AAAI , Pittsburgh, 112\u2013120. http:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP14\/paper\/view\/8977 Anand Kulkarni, Prayag Narula, David Rolnitzky, and Nathan Kontny. 2014. Wish: Amplifying Creative Ability with Expert Crowds. In Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP). AAAI, Pittsburgh, 112\u2013120. http:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP14\/paper\/view\/8977"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1002\/ejsp.2420030402"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/3463952.3464048"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2819979"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998322"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2631912"},{"key":"e_1_3_2_2_26_1","volume-title":"Emotions of Normal People","author":"Marston M.","unstructured":"William\u00a0 M. Marston . 2013. Emotions of Normal People . Taylor & Francis , London . William\u00a0M. Marston. 2013. Emotions of Normal People. Taylor & Francis, London."},{"key":"e_1_3_2_2_27_1","volume-title":"Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS). IFAAMAS, Auckland, 1706\u20131710","author":"Murukannaiah K.","year":"2020","unstructured":"Pradeep\u00a0 K. Murukannaiah , Nirav Ajmeri , Catholijn\u00a0 M. Jonker , and Munindar\u00a0 P. Singh . 2020 . New Foundations of Ethical Multiagent Systems . In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS). IFAAMAS, Auckland, 1706\u20131710 . https:\/\/doi.org\/10.5555\/3398761.3398958 Blue Sky Ideas Track. Pradeep\u00a0K. Murukannaiah, Nirav Ajmeri, Catholijn\u00a0M. Jonker, and Munindar\u00a0P. Singh. 2020. New Foundations of Ethical Multiagent Systems. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS). IFAAMAS, Auckland, 1706\u20131710. https:\/\/doi.org\/10.5555\/3398761.3398958 Blue Sky Ideas Track."},{"key":"e_1_3_2_2_28_1","unstructured":"Pradeep\u00a0K. Murukannaiah Nirav Ajmeri Muhammad\u00a0Fazalul Rahman and Munindar\u00a0P. Singh. 2020. Enhancing Crowd Creativity as Innovation via Teamwork: Dataset. https:\/\/doi.org\/10.5281\/zenodo.4308066. https:\/\/doi.org\/10.5281\/zenodo.4308066  Pradeep\u00a0K. Murukannaiah Nirav Ajmeri Muhammad\u00a0Fazalul Rahman and Munindar\u00a0P. Singh. 2020. Enhancing Crowd Creativity as Innovation via Teamwork: Dataset. https:\/\/doi.org\/10.5281\/zenodo.4308066. https:\/\/doi.org\/10.5281\/zenodo.4308066"},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2016.68"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2017.74"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376677"},{"key":"e_1_3_2_2_32_1","volume-title":"Applied Imagination: Principles and Procedures of Creative Thinking.","author":"Osborn F.","year":"1957","unstructured":"Alex\u00a0 F. Osborn . 1957 . Applied Imagination: Principles and Procedures of Creative Thinking. Scribners, New York . Alex\u00a0F. Osborn. 1957. Applied Imagination: Principles and Procedures of Creative Thinking. Scribners, New York."},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1111\/1464-0597.00013"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1177\/0146167293191009"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1006\/obhd.2000.2888"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-0663.98.3.583"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647409"},{"key":"e_1_3_2_2_38_1","volume-title":"Proceedings of the International AAAI Conference on Web and Social Media. AAAI Press, Barcelona, 321\u2013328","author":"Rogstadius Jakob","year":"2011","unstructured":"Jakob Rogstadius , Vassilis Kostakos , Aniket Kittur , Boris Smus , Jim Laredo , and Maja Vukovic . 2011 . An Assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets . In Proceedings of the International AAAI Conference on Web and Social Media. AAAI Press, Barcelona, 321\u2013328 . Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An Assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In Proceedings of the International AAAI Conference on Web and Social Media. AAAI Press, Barcelona, 321\u2013328."},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1080\/10400419.2012.650092"},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998300"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1111\/isj.12227"},{"key":"e_1_3_2_2_42_1","volume-title":"Everything DiSC Manual","author":"Scullard Mark","unstructured":"Mark Scullard and Dabney Baum . 2015. Everything DiSC Manual . John Wiley & Sons, Hoboken , NJ. Mark Scullard and Dabney Baum. 2015. Everything DiSC Manual. John Wiley & Sons, Hoboken, NJ."},{"key":"e_1_3_2_2_43_1","volume-title":"Creativity: Cognitive, personal, developmental, and social aspects.American Psychologist 55, 1","author":"Simonton Dean\u00a0Keith","year":"2000","unstructured":"Dean\u00a0Keith Simonton . 2000 . Creativity: Cognitive, personal, developmental, and social aspects.American Psychologist 55, 1 (2000), 151\u2013158. https:\/\/doi.org\/10.1037\/0003-066X.55.1.151 Dean\u00a0Keith Simonton. 2000. Creativity: Cognitive, personal, developmental, and social aspects.American Psychologist 55, 1 (2000), 151\u2013158. https:\/\/doi.org\/10.1037\/0003-066X.55.1.151"},{"key":"e_1_3_2_2_44_1","first-page":"3","article-title":"The Pragmatic Web","volume":"6","author":"Singh P.","year":"2002","unstructured":"Munindar\u00a0 P. Singh . 2002 . The Pragmatic Web . IEEE Internet Computing (IC) 6 , 3 (May 2002), 4\u20135. https:\/\/doi.org\/10.1109\/MIC.2002.1003124 Instance of the column Being Interactive. Munindar\u00a0P. Singh. 2002. The Pragmatic Web. IEEE Internet Computing (IC) 6, 3 (May 2002), 4\u20135. https:\/\/doi.org\/10.1109\/MIC.2002.1003124 Instance of the column Being Interactive.","journal-title":"IEEE Internet Computing (IC)"},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2542182.2542203"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i11.21494"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.2307\/2393872"},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2015.7320409"},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.5555\/3398761.3398923"},{"key":"e_1_3_2_2_50_1","volume-title":"Proceedings of the First International Conference on Hybrid Human-Artificial Intelligence(HHAI). IOS Press","author":"van\u00a0der Meer Michiel","year":"2022","unstructured":"Michiel van\u00a0der Meer , Enrico Liscio , Catholijn\u00a0 M. Jonker , Aske Plaat , Piek Vossen , and Pradeep\u00a0 K. Murukannaiah . 2022 . HyEnA: A Hybrid Method for Extracting Arguments from Opinions . In Proceedings of the First International Conference on Hybrid Human-Artificial Intelligence(HHAI). IOS Press , Amsterdam, 1\u201316. To appear. Michiel van\u00a0der Meer, Enrico Liscio, Catholijn\u00a0M. Jonker, Aske Plaat, Piek Vossen, and Pradeep\u00a0K. Murukannaiah. 2022. HyEnA: A Hybrid Method for Extracting Arguments from Opinions. In Proceedings of the First International Conference on Hybrid Human-Artificial Intelligence(HHAI). IOS Press, Amsterdam, 1\u201316. To appear."},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.5555\/3535850.3536118"},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979147"},{"key":"e_1_3_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-021-01084-x"}],"event":{"name":"WebSci '22: 14th ACM Web Science Conference 2022","location":"Barcelona Spain","acronym":"WebSci '22","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web"]},"container-title":["14th ACM Web Science Conference 2022"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3501247.3531555","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3501247.3531555","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3501247.3531555","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:12:18Z","timestamp":1750191138000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3501247.3531555"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,26]]},"references-count":53,"alternative-id":["10.1145\/3501247.3531555","10.1145\/3501247"],"URL":"https:\/\/doi.org\/10.1145\/3501247.3531555","relation":{},"subject":[],"published":{"date-parts":[[2022,6,26]]},"assertion":[{"value":"2022-06-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}