{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,2]],"date-time":"2026-05-02T16:57:20Z","timestamp":1777741040185,"version":"3.51.4"},"reference-count":151,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2023,2,8]],"date-time":"2023-02-08T00:00:00Z","timestamp":1675814400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001871","name":"National Funds through FLAD\u2014Luso-American Development Foundation","doi-asserted-by":"publisher","award":["SFRH\/BD\/136211\/2018"],"award-info":[{"award-number":["SFRH\/BD\/136211\/2018"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001871","name":"National Funds through FLAD\u2014Luso-American Development Foundation","doi-asserted-by":"publisher","award":["SFRH\/BD\/136211\/2018"],"award-info":[{"award-number":["SFRH\/BD\/136211\/2018"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001871","name":"FCT\u2014Portuguese Foundation for Science and Technology","doi-asserted-by":"publisher","award":["SFRH\/BD\/136211\/2018"],"award-info":[{"award-number":["SFRH\/BD\/136211\/2018"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001871","name":"FCT\u2014Portuguese Foundation for Science and Technology","doi-asserted-by":"publisher","award":["SFRH\/BD\/136211\/2018"],"award-info":[{"award-number":["SFRH\/BD\/136211\/2018"]}],"id":[{"id":"10.13039\/501100001871","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Applied Sciences"],"abstract":"<jats:p>With the widespread availability and pervasiveness of artificial intelligence (AI) in many application areas across the globe, the role of crowdsourcing has seen an upsurge in terms of importance for scaling up data-driven algorithms in rapid cycles through a relatively low-cost distributed workforce or even on a volunteer basis. However, there is a lack of systematic and empirical examination of the interplay among the processes and activities combining crowd-machine hybrid interaction. To uncover the enduring aspects characterizing the human-centered AI design space when involving ensembles of crowds and algorithms and their symbiotic relations and requirements, a Computer-Supported Cooperative Work (CSCW) lens strongly rooted in the taxonomic tradition of conceptual scheme development is taken with the aim of aggregating and characterizing some of the main component entities in the burgeoning domain of hybrid crowd-AI centered systems. The goal of this article is thus to propose a theoretically grounded and empirically validated analytical framework for the study of crowd-machine interaction and its environment. Based on a scoping review and several cross-sectional analyses of research studies comprising hybrid forms of human interaction with AI systems and applications at a crowd scale, the available literature was distilled and incorporated into a unifying framework comprised of taxonomic units distributed across integration dimensions that range from the original time and space axes in which every collaborative activity take place to the main attributes that constitute a hybrid intelligence architecture. The upshot is that when turning to the challenges that are inherent in tasks requiring massive participation, novel properties can be obtained for a set of potential scenarios that go beyond the single experience of a human interacting with the technology to comprise a vast set of massive machine-crowd interactions.<\/jats:p>","DOI":"10.3390\/app13042198","type":"journal-article","created":{"date-parts":[[2023,2,9]],"date-time":"2023-02-09T01:37:07Z","timestamp":1675906627000},"page":"2198","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":26,"title":["Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine Interaction"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2736-3835","authenticated-orcid":false,"given":"Ant\u00f3nio","family":"Correia","sequence":"first","affiliation":[{"name":"INESC TEC, University of Tr\u00e1s-os-Montes e Alto Douro, 5000-801 Vila Real, Portugal"},{"name":"College of Information Science & Technology, University of Nebraska at Omaha, Omaha, NE 68182, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4082-4138","authenticated-orcid":false,"given":"Andrea","family":"Grover","sequence":"additional","affiliation":[{"name":"College of Information Science & Technology, University of Nebraska at Omaha, Omaha, NE 68182, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2987-4732","authenticated-orcid":false,"given":"Daniel","family":"Schneider","sequence":"additional","affiliation":[{"name":"Postgraduate Program in Informatics, PPGI\/UFRJ, Rio de Janeiro 21941-916, Brazil"},{"name":"T\u00e9rcio Pacitti Institute of Computer Applications and Research (NCE), Federal University of Rio de Janeiro, Rio de Janeiro 21941-916, Brazil"}]},{"given":"Ana Paula","family":"Pimentel","sequence":"additional","affiliation":[{"name":"Postgraduate Program in Informatics, PPGI\/UFRJ, Rio de Janeiro 21941-916, Brazil"}]},{"given":"Ramon","family":"Chaves","sequence":"additional","affiliation":[{"name":"Systems Engineering and Computer Science Program (PESC\/COPPE\/UFRJ), Rio de Janeiro 21941-972, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8900-3464","authenticated-orcid":false,"given":"Marcos Antonio","family":"de Almeida","sequence":"additional","affiliation":[{"name":"Systems Engineering and Computer Science Program (PESC\/COPPE\/UFRJ), Rio de Janeiro 21941-972, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0850-9755","authenticated-orcid":false,"given":"Benjamim","family":"Fonseca","sequence":"additional","affiliation":[{"name":"INESC TEC, University of Tr\u00e1s-os-Montes e Alto Douro, 5000-801 Vila Real, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2023,2,8]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Lofi, C., and El Maarry, K. (2014, January 14\u201317). Design patterns for hybrid algorithmic-crowdsourcing workflows. Proceedings of the IEEE 16th Conference on Business Informatics, Geneva, Switzerland.","DOI":"10.1109\/CBI.2014.16"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"034002","DOI":"10.1117\/1.JMI.5.3.034002","article-title":"Large-scale medical image annotation with crowd-powered algorithms","volume":"5","author":"Heim","year":"2018","journal-title":"J. Med. Imaging"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Vargas-Santiago, M., Monroy, R., Ramirez-Marquez, J.E., Zhang, C., Leon-Velasco, D.A., and Zhu, H. (2020). Complementing solutions to optimization problems via crowdsourcing on video game plays. Appl. Sci., 10.","DOI":"10.3390\/app10238410"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3479196","article-title":"Flud: A hybrid crowd\u2013algorithm approach for visualizing biological networks","volume":"29","author":"Bharadwaj","year":"2022","journal-title":"ACM Trans. Comput. Interact."},{"key":"ref_5","first-page":"1323","article-title":"Taxonomy and theory in computer supported cooperative work","volume":"2","author":"Grudin","year":"2012","journal-title":"Oxf. Handb. Organ. Psychol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1057\/ejis.2012.26","article-title":"A method for taxonomy development and its application in information systems","volume":"22","author":"Nickerson","year":"2013","journal-title":"Eur. J. Inf. Syst."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3359250","article-title":"Joining together online: The trajectory of CSCW scholarship on group formation","volume":"3","author":"Harris","year":"2019","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_8","unstructured":"McGrath, J.E. (1984). Groups: Interaction and Performance, Prentice-Hall."},{"key":"ref_9","first-page":"8","article-title":"Scaling group tasks: A method for dimensional analysis","volume":"3","author":"Shaw","year":"1973","journal-title":"JSAS Cat. Sel. Doc. Psychol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"102135","DOI":"10.1016\/j.ipm.2019.102135","article-title":"Information technology (IT) enabled crowdsourcing: A conceptual framework","volume":"57","author":"Modaresnezhad","year":"2020","journal-title":"Inf. Process. Manag."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"110611","DOI":"10.1016\/j.jss.2020.110611","article-title":"General framework, opportunities and challenges for crowdsourcing techniques: A comprehensive survey","volume":"167","author":"Bhatti","year":"2020","journal-title":"J. Syst. Softw."},{"key":"ref_12","unstructured":"Johansen, R. (1988). Groupware: Computer Support for Business Teams, The Free Press."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Lee, C.P., and Paine, D. (2015, January 14\u201318). From the matrix to a model of coordinated action (MoCA): A conceptual framework of and for CSCW. Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, Vancouver, BC, Canada.","DOI":"10.1145\/2675133.2675161"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"517","DOI":"10.1007\/s10606-022-09433-8","article-title":"Uncovering the complexity of care networks\u2014Towards a taxonomy of collaboration complexity in homecare","volume":"31","author":"Renyi","year":"2022","journal-title":"Comput. Support. Cooperative Work. (CSCW)"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3274442","article-title":"Transforming taxonomic interfaces: \u201cArm\u2019s length\u201d cooperative work and the maintenance of a long-lived classification system","volume":"2","author":"Thomer","year":"2018","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/MC.2020.2996587","article-title":"A research agenda for hybrid intelligence: Augmenting human intellect with collaborative, adaptive, responsible, and explainable artificial intelligence","volume":"53","author":"Akata","year":"2020","journal-title":"Computer"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"633","DOI":"10.3390\/forecast3030039","article-title":"A brief taxonomy of hybrid intelligence","volume":"3","author":"Pescetelli","year":"2021","journal-title":"Forecasting"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Dellermann, D., Calma, A., Lipusch, N., Weber, T., Weigel, S., and Ebel, P. (2019, January 8\u201311). The future of human-AI collaboration: A taxonomy of design knowledge for hybrid intelligence systems. Proceedings of the 52nd Hawaii International Conference on System Sciences, Maui, HI, USA.","DOI":"10.24251\/HICSS.2019.034"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Dubey, A., Abhinav, K., Jain, S., Arora, V., and Puttaveerana, A. (2020, January 27\u201329). HACO: A framework for developing human-AI teaming. Proceedings of the 13th Innovations in Software Engineering Conference, Jabalpur, India.","DOI":"10.1145\/3385032.3385044"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1016\/j.endeavour.2014.03.002","article-title":"Crowdsourcing, the great meteor storm of 1833, and the founding of meteor science","volume":"38","author":"Littmann","year":"2014","journal-title":"Endeavour"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"294","DOI":"10.1504\/IJISD.2009.033083","article-title":"Outsourcing labour to the cloud","volume":"4","author":"Corney","year":"2009","journal-title":"Int. J. Innovation Sustain. Dev."},{"key":"ref_22","unstructured":"Rouse, A.C. (2010, January 1\u20133). A preliminary taxonomy of crowdsourcing. Proceedings of the Australasian Conference on Information Systems, Brisbane, Australia."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1109\/EMR.2010.5559142","article-title":"The collective intelligence genome","volume":"38","author":"Malone","year":"2010","journal-title":"IEEE Eng. Manag. Rev."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"11","DOI":"10.2753\/JEC1086-4415150101","article-title":"Co-creation: Toward a taxonomy and an integrated research perspective","volume":"15","author":"Zwass","year":"2010","journal-title":"Int. J. Electron. Commer."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1145\/1924421.1924442","article-title":"Crowdsourcing systems on the world-wide web","volume":"54","author":"Doan","year":"2011","journal-title":"Commun. ACM"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1080\/10580530.2013.739883","article-title":"Rules of crowdsourcing: Models, issues, and systems of control","volume":"30","author":"Saxton","year":"2013","journal-title":"Inf. Syst. Management"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"823","DOI":"10.1177\/0165551514550140","article-title":"A taxonomy of crowdsourcing based on task complexity","volume":"40","author":"Nakatsu","year":"2014","journal-title":"J. Inf. Sci."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Gadiraju, U., Kawase, R., and Dietze, S. (2014, January 1\u20134). A taxonomy of microtasks on the web. Proceedings of the 25th ACM Conference on Hypertext and Social Media, Santiago, Chile.","DOI":"10.1145\/2631775.2631819"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1016\/j.cosrev.2015.05.001","article-title":"Crowdsourcing: A taxonomy and systematic mapping study","volume":"17","author":"Hosseini","year":"2015","journal-title":"Comput. Sci. Rev."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Alabduljabbar, R., and Al-Dossari, H. (2017, January 22\u201323). Towards a classification model for tasks in crowdsourcing. Proceedings of the Second International Conference on Internet of Things and Cloud Computing, Cambridge, UK.","DOI":"10.1145\/3018896.3018916"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1108\/JKM-10-2021-0769","article-title":"Exploring the effects of problem- and solution-related knowledge sharing in internal crowdsourcing","volume":"26","author":"Chen","year":"2022","journal-title":"J. Knowl. Manag."},{"key":"ref_32","unstructured":"Chilton, L.B., Little, G., Edge, D., Weld, D.S., and Landay, J.A. (May, January 27). Cascade: Crowdsourcing taxonomy creation. Proceedings of the CHI Conference on Human Factors in Computing Systems, Paris, France."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Sharif, A., Gopal, P., Saugstad, M., Bhatt, S., Fok, R., Weld, G., Dey, K.A.M., and Froehlich, J.E. (2021, January 18\u201322). Experimental crowd+AI approaches to track accessibility features in sidewalk intersections over time. Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility, Virtual Event.","DOI":"10.1145\/3441852.3476549"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zhang, D.Y., Huang, Y., Zhang, Y., and Wang, D. (2020, January 7\u201312). Crowd-assisted disaster scene assessment with human-AI interactive attention. Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i03.5658"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Kaspar, A., Patterson, G., Kim, C., Aksoy, Y., Matusik, W., and Elgharib, M. (2018, January 21\u201326). Crowd-guided ensembles: How can we choreograph crowd workers for video segmentation?. Proceedings of the CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.","DOI":"10.1145\/3173574.3173685"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1515","DOI":"10.1109\/TCSS.2021.3109143","article-title":"CollabLearn: An uncertainty-aware crowd-AI collaboration system for cultural heritage damage assessment","volume":"9","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Comput. Soc. Syst."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Maier-Hein, L., Ross, T., Gr\u00f6hl, J., Glocker, B., Bodenstedt, S., Stock, C., Heim, E., G\u00f6tz, M., Wirkert, S.J., and Kenngott, H. (2016, January 17\u201321). Crowd-algorithm collaboration for large-scale endoscopic image annotation with confidence. Proceedings of the 19th International Conference on Medical Image Computing and Computer-Assisted Intervention, Athens, Greece.","DOI":"10.1007\/978-3-319-46723-8_71"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Mohanty, V., Thames, D., Mehta, S., and Luther, K. (2019, January 17\u201320). Photo Sleuth: Combining human expertise and face recognition to identify historical portraits. Proceedings of the 24th International Conference on Intelligent User Interfaces, Marina del Ray, CA, USA.","DOI":"10.1145\/3301275.3302301"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Huang, T.H., Chang, J.C., and Bigham, J.P. (2018, January 21\u201326). Evorus: A crowd-powered conversational assistant built to automate itself over time. Proceedings of the CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.","DOI":"10.1145\/3173574.3173869"},{"key":"ref_40","first-page":"1","article-title":"Crowd-AI camera sensing in the real world","volume":"2","author":"Guo","year":"2018","journal-title":"Proc. ACM Interactive, Mobile, Wearable Ubiquitous Technol."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Correia, A., Paredes, H., Schneider, D., Jameel, S., and Fonseca, B. (2019, January 6\u20139). Towards hybrid crowd-AI centered systems: Developing an integrated framework from an empirical perspective. Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Bari, Italy.","DOI":"10.1109\/SMC.2019.8914075"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"494","DOI":"10.1080\/10447318.2022.2041900","article-title":"Transitioning to human interaction with AI systems: New challenges and opportunities for HCI professionals to enable human-centered AI","volume":"39","author":"Xu","year":"2022","journal-title":"Int. J. Human\u2013Computer Interact."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"A77","DOI":"10.1051\/0004-6361\/202243428","article-title":"Zero-phase angle asteroid taxonomy classification using unsupervised machine learning algorithms","volume":"666","author":"Colazo","year":"2022","journal-title":"Astron. Astrophys."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"e2122636119","DOI":"10.1073\/pnas.2122636119","article-title":"Taxonomic classification of DNA sequences beyond sequence similarity using deep neural networks","volume":"119","author":"Mock","year":"2022","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1111\/j.1547-5069.1987.tb00613.x","article-title":"The nature of taxonomy","volume":"19","author":"Rasch","year":"1987","journal-title":"Image J. Nurs. Scholarsh."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Tricco, A.C., Lillie, E., Zarin, W., O\u2019Brien, K., Colquhoun, H., Kastner, M., Levac, D., Ng, C., Sharpe, J.P., and Wilson, K. (2016). A scoping review on the conduct and reporting of scoping reviews. BMC Med. Res. Methodol., 16.","DOI":"10.1186\/s12874-016-0116-4"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1146\/annurev.es.17.110186.002231","article-title":"Phenetic taxonomy: Theory and methods","volume":"17","author":"Sokal","year":"1986","journal-title":"Annu. Rev. Ecol. Syst."},{"key":"ref_48","unstructured":"Oberl\u00e4nder, A.M., L\u00f6sser, B., and Rau, D. (2019, January 8\u201314). Taxonomy research in information systems: A systematic assessment. Proceedings of the 27th European Conference on Information Systems, Stockholm and Uppsala, Sweden."},{"key":"ref_49","unstructured":"Gerber, A. (2018, January 16\u201318). Computational ontologies as classification artifacts in IS research. Proceedings of the 24th Americas Conference on Information Systems, New Orleans, LA, USA."},{"key":"ref_50","first-page":"2","article-title":"Analyzing the past to prepare for the future: Writing a literature review","volume":"26","author":"Webster","year":"2002","journal-title":"MIS Q."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Schmidt-Kraepelin, M., Thiebes, S., Tran, M.C., and Sunyaev, A. (2018, January 3\u20136). What\u2019s in the game? Developing a taxonomy of gamification concepts for health apps. Proceedings of the 51st Hawaii International Conference on System Sciences, Hilton Waikoloa Village, HI, USA.","DOI":"10.24251\/HICSS.2018.150"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"102584","DOI":"10.1016\/j.ipm.2021.102584","article-title":"Taxonomy of centralization in public blockchain systems: A systematic literature review","volume":"58","author":"Sai","year":"2021","journal-title":"Inf. Process. Manag."},{"key":"ref_53","unstructured":"Andraschko, L., Wunderlich, P., Veit, D., and Sarker, S. (2021, January 12\u201315). Towards a taxonomy of smart home technology: A preliminary understanding. Proceedings of the 42nd International Conference on Information Systems, Austin, TX, USA."},{"key":"ref_54","first-page":"15","article-title":"Understanding the elephant: The discourse approach to boundary identification and corpus construction for theory review articles","volume":"20","author":"Larsen","year":"2019","journal-title":"J. Assoc. Inf. Syst."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Elliott, J.H., Turner, T., Clavisi, O., Thomas, J., Higgins, J.P.T., Mavergames, C., and Gruen, R.L. (2014). Living systematic reviews: An emerging opportunity to narrow the evidence-practice gap. PLoS Med., 11.","DOI":"10.1371\/journal.pmed.1001603"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"5113","DOI":"10.1007\/s11192-021-03948-5","article-title":"The journal coverage of Web of Science, Scopus and Dimensions: A comparative analysis","volume":"126","author":"Singh","year":"2021","journal-title":"Scientometrics"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Kittur, A., Nickerson, J.V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., and Horton, J.J. (2013, January 23\u201327). The future of crowd work. Proceedings of the ACM Conference on Computer Supported Cooperative Work, San Antonio, TX, USA.","DOI":"10.1145\/2441776.2441923"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Zhang, D., Zhang, Y., Li, Q., Plummer, T., and Wang, D. (2019, January 7\u201310). CrowdLearn: A crowd-AI hybrid system for deep learning-based damage assessment applications. Proceedings of the 39th IEEE International Conference on Distributed Computing Systems, Dallas, TX, USA.","DOI":"10.1109\/ICDCS.2019.00123"},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Landolt, S., Wambsganss, T., and S\u00f6llner, M. (2021, January 5). A taxonomy for deep learning in natural language processing. Proceedings of the 54th Hawaii International Conference on System Sciences, Kauai, HI, USA.","DOI":"10.24251\/HICSS.2021.129"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1177\/104649649903000202","article-title":"Testing a typology of tasks: An empirical validation of McGrath\u2019s (1984) group task circumplex","volume":"30","author":"Straus","year":"1999","journal-title":"Small Group Research"},{"key":"ref_61","unstructured":"Chesbrough, H.W. (2003). Open Innovation: The New Imperative for Creating and Profiting from Technology, Harvard Business Press."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1111\/caim.12454","article-title":"Understanding crowdsourcing projects: A review on the key design elements of a crowdsourcing initiative","volume":"30","author":"Karachiwalla","year":"2021","journal-title":"Creativity Innov. Manag."},{"key":"ref_63","unstructured":"Hemmer, P., Schemmer, M., V\u00f6ssing, M., and K\u00fchl, N. (2021, January 12\u201314). Human-AI complementarity in hybrid intelligence systems: A structured literature review. Proceedings of the 25th Pacific Asia Conference on Information Systems, Virtual Event, Dubai, United Arab Emirates."},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Holstein, K., Aleven, V., and Rummel, N. (2020, January 6\u201310). A conceptual framework for human-AI hybrid adaptivity in education. Proceedings of the 21st International Conference on Artificial Intelligence in Education, Ifrane, Morocco.","DOI":"10.1007\/978-3-030-52237-7_20"},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"871","DOI":"10.1007\/s10726-022-09792-z","article-title":"Elaborating team roles for artificial intelligence-based teammates in human-AI collaboration","volume":"31","author":"Siemon","year":"2022","journal-title":"Group Decis. Negot."},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Weber, E., Marzo, N., Papadopoulos, D.P., Biswas, A., Lapedriza, A., Ofli, F., Imran, M., and Torralba, A. (2020, January 23\u201328). Detecting natural disasters, damage, and incidents in the wild. Proceedings of the 16th European Conference on Computer Vision, Glasgow, UK.","DOI":"10.1007\/978-3-030-58529-7_20"},{"key":"ref_67","first-page":"7026","article-title":"Making better use of the crowd: How crowdsourcing can advance machine learning research","volume":"18","author":"Vaughan","year":"2017","journal-title":"J. Mach. Learn. Res."},{"key":"ref_68","doi-asserted-by":"crossref","unstructured":"Hamadi, R., Ghazzai, H., and Massoud, Y. (2022). A generative adversarial network for financial advisor recruitment in smart crowdsourcing platforms. Appl. Sci., 12.","DOI":"10.3390\/app12199830"},{"key":"ref_69","first-page":"2","article-title":"Work system theory: Overview of core concepts, extensions, and challenges for the future","volume":"14","author":"Alter","year":"2013","journal-title":"J. Assoc. Inf. Syst."},{"key":"ref_70","doi-asserted-by":"crossref","unstructured":"Venumuddala, V.R., and Kamath, R. (2022). Work systems in the Indian information technology (IT) industry delivering artificial intelligence (AI) solutions and the challenges of work from home. Inf. Syst. Front., 1\u201325.","DOI":"10.1007\/s10796-022-10259-4"},{"key":"ref_71","doi-asserted-by":"crossref","unstructured":"Nardi, B. (1996). Context and Consciousness: Activity Theory and Human-Computer Interaction, MIT Press.","DOI":"10.7551\/mitpress\/2137.001.0001"},{"key":"ref_72","doi-asserted-by":"crossref","unstructured":"Neale, D.C., Carroll, J.M., and Rosson, M.B. (2004, January 6\u201310). Evaluating computer-supported cooperative work: Models and frameworks. Proceedings of the ACM Conference on Computer Supported Cooperative Work, Chicago, IL, USA.","DOI":"10.1145\/1031607.1031626"},{"key":"ref_73","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3274373","article-title":"Exploring real-time collaboration in crowd-powered systems through a UI design tool","volume":"2","author":"Lee","year":"2018","journal-title":"Proc. ACM Human-Computer Interact."},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"2902","DOI":"10.1109\/TVT.2016.2592685","article-title":"A picture is worth a thousand words: Share your real-time view on the road","volume":"66","author":"Wang","year":"2016","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"Agapie, E., Teevan, J., and Monroy-Hern\u00e1ndez, A. (2015, January 8\u201311). Crowdsourcing in the field: A case study using local crowds for event reporting. Proceedings of the Third AAAI Conference on Human Computation and Crowdsourcing, San Diego, CA, USA.","DOI":"10.1609\/hcomp.v3i1.13235"},{"key":"ref_76","doi-asserted-by":"crossref","unstructured":"Lafreniere, B.J., Grossman, T., Anderson, F., Matejka, J., Kerrick, H., Nagy, D., Vasey, L., Atherton, E., Beirne, N., and Coelho, M.H. (2016, January 16\u201319). Crowdsourced fabrication. Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, Japan.","DOI":"10.1145\/2984511.2984553"},{"key":"ref_77","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1016\/j.chb.2017.04.044","article-title":"Profiles of engagement in online communities of citizen science participation","volume":"74","author":"Aristeidou","year":"2017","journal-title":"Comput. Hum. Behav."},{"key":"ref_78","unstructured":"Bouwer, A. (2022). Marketing and Smart Technologies, Springer."},{"key":"ref_79","unstructured":"Lubars, B., and Tan, C. (2019, January 8\u201314). Ask not what AI can do, but what AI should do: Towards a framework of task delegability. Proceedings of the Annual Conference on Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_80","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3492854","article-title":"Investigating crowdworkers\u2019 identify, perception and practices in micro-task crowdsourcing","volume":"6","author":"Sun","year":"2022","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_81","doi-asserted-by":"crossref","unstructured":"Khan, V.J., Papangelis, K., Lykourentzou, I., and Markopoulos, P. (2019). Macrotask Crowdsourcing\u2014Engaging the Crowds to Address Complex Problems, Springer.","DOI":"10.1007\/978-3-030-12334-5"},{"key":"ref_82","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1145\/3019600","article-title":"The future of microwork","volume":"23","author":"Teevan","year":"2016","journal-title":"XRDS Crossroads ACM Mag. Stud."},{"key":"ref_83","doi-asserted-by":"crossref","first-page":"24721","DOI":"10.1109\/ACCESS.2022.3148400","article-title":"Microtasking activities in crowdsourced software development: A systematic literature review","volume":"10","author":"Zulfiqar","year":"2022","journal-title":"IEEE Access"},{"key":"ref_84","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s00778-018-0516-7","article-title":"Optimized group formation for solving collaborative tasks","volume":"28","author":"Rahman","year":"2018","journal-title":"VLDB J."},{"key":"ref_85","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3140459","article-title":"Online sequencing of non-decomposable macrotasks in expert crowdsourcing","volume":"1","author":"Schmitz","year":"2018","journal-title":"ACM Trans. Soc. Comput."},{"key":"ref_86","doi-asserted-by":"crossref","first-page":"103351","DOI":"10.1016\/j.artint.2020.103351","article-title":"A technical survey on statistical modelling and design methods for crowdsourcing quality control","volume":"287","author":"Jin","year":"2020","journal-title":"Artif. Intell."},{"key":"ref_87","doi-asserted-by":"crossref","first-page":"113404","DOI":"10.1016\/j.dss.2020.113404","article-title":"Optimizing microtask assignment on crowdsourcing platforms using Markov chain Monte Carlo","volume":"139","author":"Moayedikia","year":"2020","journal-title":"Decis. Support Syst."},{"key":"ref_88","doi-asserted-by":"crossref","unstructured":"Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S.T., Bennett, P.N., and Inkpen, K. (2019, January 4\u20139). Guidelines for human-AI interaction. Proceedings of the CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland, UK.","DOI":"10.1145\/3290605.3300233"},{"key":"ref_89","doi-asserted-by":"crossref","first-page":"66","DOI":"10.15346\/hc.v9i1.133","article-title":"Mapping citizen science through the lens of human-centered AI","volume":"9","author":"Rafner","year":"2022","journal-title":"Hum. Comput."},{"key":"ref_90","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3419764","article-title":"Bridging the gap between ethics and practice: Guidelines for reliable, safe, and trustworthy human-centered AI systems","volume":"10","author":"Shneiderman","year":"2020","journal-title":"ACM Trans. Interact. Intell. Syst."},{"key":"ref_91","first-page":"1","article-title":"On the state of reporting in crowdsourcing experiments and a checklist to aid current practices","volume":"5","author":"Sayin","year":"2021","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_92","doi-asserted-by":"crossref","unstructured":"Robert, L., and Romero, D.M. (2015, January 18\u201323). Crowd size, diversity and performance. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Republic of Korea.","DOI":"10.1145\/2702123.2702469"},{"key":"ref_93","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1111\/1468-0394.00151","article-title":"Intelligent interaction design: The role of human-computer interaction research in the design of intelligent systems","volume":"18","author":"Blandford","year":"2001","journal-title":"Expert Syst."},{"key":"ref_94","first-page":"1","article-title":"Being a solo endeavor or team worker in crowdsourcing contests? It is a long-term decision you need to make","volume":"6","author":"Huang","year":"2022","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_95","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3359209","article-title":"GroundTruth: Augmenting expert image geolocation with crowdsourcing and shared representations","volume":"3","author":"Venkatagiri","year":"2019","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_96","doi-asserted-by":"crossref","unstructured":"Zhou, S., Valentine, M., and Bernstein, M.S. (2018, January 21\u201326). In search of the dream team: Temporally constrained multi-armed bandits for identifying effective team structures. Proceedings of the CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.","DOI":"10.1145\/3173574.3173682"},{"key":"ref_97","unstructured":"Gray, M.L., Suri, S., Ali, S.S., and Kulkarni, D. (March, January 27). The crowd is a collaborative network. Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, San Francisco, CA, USA."},{"key":"ref_98","doi-asserted-by":"crossref","first-page":"103027","DOI":"10.1016\/j.ipm.2022.103027","article-title":"Imbalanced volunteer engagement in cultural heritage crowdsourcing: A task-related exploration based on causal inference","volume":"59","author":"Zhang","year":"2022","journal-title":"Inf. Process. Manag."},{"key":"ref_99","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1177\/15553434211017354","article-title":"Team situation awareness and conflict: A study of human\u2013machine teaming","volume":"15","author":"McNeese","year":"2021","journal-title":"J. Cogn. Eng. Decis. Mak."},{"key":"ref_100","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1038\/d41586-021-01170-0","article-title":"Cooperative AI: Machines must learn to find common ground","volume":"593","author":"Dafoe","year":"2021","journal-title":"Nature"},{"key":"ref_101","unstructured":"Alorwu, A., Savage, S., van Berkel, N., Ustalov, D., Drutsa, A., Oppenlaender, J., Bates, O., Hettiachchi, D., Gadiraju, U., and Gon\u00e7alves, J. (May, January 29). REGROW: Reimagining global crowdsourcing for better human-AI collaboration. Proceedings of the CHI Conference on Human Factors in Computing Systems, Extended Abstracts, New Orleans, LA, USA."},{"key":"ref_102","doi-asserted-by":"crossref","unstructured":"Santos, C.A., Baldi, A.M., de Assis Neto, F.R., and Barcellos, M.P. (2022). Essential elements, conceptual foundations and workflow design in crowd-powered projects. J. Inf. Sci.","DOI":"10.1177\/01655515211062466"},{"key":"ref_103","doi-asserted-by":"crossref","unstructured":"Valentine, M.A., Retelny, D., To, A., Rahmati, N., Doshi, T., and Bernstein, M.S. (2017, January 6\u201311). Flash organizations: Crowdsourcing complex work by structuring crowds as organizations. Proceedings of the CHI Conference on Human Factors in Computing Systems, Denver, CO, USA.","DOI":"10.1145\/3025453.3025811"},{"key":"ref_104","unstructured":"Kamar, E. (2016, January 9\u201315). Directions in hybrid intelligence: Complementing AI systems with human intelligence. Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, New York, NY, USA."},{"key":"ref_105","doi-asserted-by":"crossref","first-page":"826499","DOI":"10.3389\/frai.2022.826499","article-title":"EXP-Crowd: A gamified crowdsourcing framework for explainability","volume":"5","author":"Tocchetti","year":"2022","journal-title":"Front. Artif. Intell."},{"key":"ref_106","doi-asserted-by":"crossref","unstructured":"Barbosa, N.M., and Chen, M. (2019, January 4\u20139). Rehumanized crowdsourcing: A labeling framework addressing bias and ethics in machine learning. Proceedings of the CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland, UK.","DOI":"10.1145\/3290605.3300773"},{"key":"ref_107","first-page":"1003056","article-title":"Artificial intelligence and human learning: Improving analytic reasoning via crowdsourcing and structured analytic techniques","volume":"3","author":"Basker","year":"2022","journal-title":"Comput. Educ."},{"key":"ref_108","first-page":"38","article-title":"Ethics and AI in information systems research","volume":"50","author":"Mirbabaie","year":"2022","journal-title":"Commun. Assoc. Inf. Syst."},{"key":"ref_109","first-page":"74","article-title":"Rise of machine agency: A framework for studying the psychology of human\u2013AI interaction (HAII)","volume":"25","author":"Sundar","year":"2020","journal-title":"J. Comput. Commun."},{"key":"ref_110","first-page":"384","article-title":"In AI we trust? Effects of agency locus and transparency on uncertainty reduction in human\u2013AI interaction","volume":"26","author":"Liu","year":"2021","journal-title":"J. Comput. Commun."},{"key":"ref_111","first-page":"zmac014","article-title":"AI agency vs. human agency: Understanding human\u2013AI interactions on TikTok and their implications for user engagement","volume":"27","author":"Kang","year":"2022","journal-title":"J. Comput. Commun."},{"key":"ref_112","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3148148","article-title":"Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions","volume":"51","author":"Daniel","year":"2018","journal-title":"ACM Comput. Surv."},{"key":"ref_113","doi-asserted-by":"crossref","unstructured":"Pedersen, J., Kocsis, D., Tripathi, A., Tarrell, A., Weerakoon, A., Tahmasbi, N., Xiong, J., Deng, W., Oh, O., and de Vreede, G.-J. (2013, January 7\u201310). Conceptual foundations of crowdsourcing: A review of IS research. Proceedings of the 46th Hawaii International Conference on System Sciences, Wailea, HI, USA.","DOI":"10.1109\/HICSS.2013.143"},{"key":"ref_114","doi-asserted-by":"crossref","first-page":"791","DOI":"10.1007\/s10606-018-9343-z","article-title":"Crowd dynamics: Conflicts, contradictions, and community in crowdsourcing","volume":"28","author":"Hansson","year":"2019","journal-title":"Comput. Support. Coop. Work."},{"key":"ref_115","doi-asserted-by":"crossref","unstructured":"Gimpel, H., Graf-Seyfried, V., Laubacher, R., and Meindl, O. (2023). Towards artificial intelligence augmenting facilitation: AI affordances in macro-task crowdsourcing. Group Decis. Negot., 1\u201350.","DOI":"10.1007\/s10726-022-09801-1"},{"key":"ref_116","unstructured":"Wu, T., Terry, M., and Cai, C.J. (May, January 29). AI chains: Transparent and controllable human-AI interaction by chaining large language model prompts. Proceedings of the CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA."},{"key":"ref_117","doi-asserted-by":"crossref","unstructured":"Kobayashi, M., Wakabayashi, K., and Morishima, A. (2021, January 14\u201318). Human+AI crowd task assignment considering result quality requirements. Proceedings of the Ninth AAAI Conference on Human Computation and Crowdsourcing, Virtual.","DOI":"10.1609\/hcomp.v9i1.18943"},{"key":"ref_118","doi-asserted-by":"crossref","first-page":"685","DOI":"10.1007\/s40685-020-00108-y","article-title":"Frontiers of business intelligence and analytics 3.0: A taxonomy-based literature review and research agenda","volume":"13","author":"Eggert","year":"2020","journal-title":"Bus. Res."},{"key":"ref_119","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3274300","article-title":"SOLVENT: A mixed initiative system for finding analogies between research papers","volume":"2","author":"Chan","year":"2018","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_120","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Shang, L., Zong, R., Wang, Z., Kou, Z., and Wang, D. (2021, January 14\u201318). StreamCollab: A streaming crowd-AI collaborative system to smart urban infrastructure monitoring in social sensing. Proceedings of the Ninth AAAI Conference on Human Computation and Crowdsourcing, Virtual.","DOI":"10.1609\/hcomp.v9i1.18950"},{"key":"ref_121","doi-asserted-by":"crossref","unstructured":"Yang, J., Smirnova, A., Yang, D., Demartini, G., Lu, Y., and Cudr\u00e9-Mauroux, P. (2019, January 13\u201317). Scalpel-CD: Leveraging crowdsourcing and deep probabilistic modeling for debugging noisy training data. Proceedings of the World Wide Web Conference, San Francisco, CA, USA.","DOI":"10.1145\/3308558.3313599"},{"key":"ref_122","doi-asserted-by":"crossref","first-page":"811","DOI":"10.1111\/isj.12227","article-title":"Ethical norms and issues in crowdsourcing practices: A Habermasian analysis","volume":"29","author":"Schlagwein","year":"2018","journal-title":"Inf. Syst. J."},{"key":"ref_123","doi-asserted-by":"crossref","first-page":"815","DOI":"10.1007\/s10606-018-9336-y","article-title":"Crowd anatomy beyond the good and bad: Behavioral traces for crowd worker modeling and pre-selection","volume":"28","author":"Gadiraju","year":"2018","journal-title":"Comput. Support. Cooperative Work."},{"key":"ref_124","doi-asserted-by":"crossref","first-page":"54","DOI":"10.15346\/hc.v8i2.123","article-title":"Citizen science, computing, and conservation: How can \u201ccrowd AI\u201d change the way we tackle large-scale ecological challenges?","volume":"8","author":"Palmer","year":"2021","journal-title":"Hum. Comput."},{"key":"ref_125","first-page":"61","article-title":"Governance, risk, and artificial intelligence","volume":"41","author":"Mannes","year":"2020","journal-title":"AI Mag."},{"key":"ref_126","doi-asserted-by":"crossref","unstructured":"Choung, H., David, P., and Ross, A. (2022). Trust and ethics in AI. AI Soc., 1\u201313.","DOI":"10.1080\/10447318.2022.2050543"},{"key":"ref_127","unstructured":"Zheng, Q., Tang, Y., Liu, Y., Liu, W., and Huang, Y. (May, January 29). UX research on conversational human-AI interaction: A literature review of the ACM Digital Library. Proceedings of the CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA."},{"key":"ref_128","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1023\/A:1021247413718","article-title":"Configuring awareness","volume":"11","author":"Heath","year":"2002","journal-title":"Comput. Support. Coop. Work."},{"key":"ref_129","doi-asserted-by":"crossref","unstructured":"Park, J., Krishna, R., Khadpe, P., Fei-Fei, L., and Bernstein, M. (2019, January 28\u201330). AI-based request augmentation to increase crowdsourcing participation. Proceedings of the Seventh AAAI Conference on Human Computation and Crowdsourcing, Stevenson, WA, USA.","DOI":"10.1609\/hcomp.v7i1.5282"},{"key":"ref_130","doi-asserted-by":"crossref","unstructured":"Star, S.L., and Ruhleder, K. (1994, January 22\u201326). Steps towards an ecology of infrastructure: Complex problems in design and access for large-scale collaborative systems. Proceedings of the ACM Conference on Computer Supported Cooperative Work, Chapel Hill, NC, USA.","DOI":"10.1145\/192844.193021"},{"key":"ref_131","doi-asserted-by":"crossref","first-page":"959","DOI":"10.1007\/s10606-017-9291-z","article-title":"From Facebook to the neighbourhood: Infrastructuring of hybrid community engagement","volume":"26","author":"Mosconi","year":"2017","journal-title":"Comput. Support. Coop. Work (CSCW)"},{"key":"ref_132","doi-asserted-by":"crossref","unstructured":"Ehsan, U., Liao, Q.V., Muller, M., Riedl, M.O., and Weisz, J.D. (2021, January 8\u201313). Expanding explainability: Towards social transparency in AI systems. Proceedings of the CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.","DOI":"10.1145\/3411764.3445188"},{"key":"ref_133","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1145\/3411286","article-title":"Interpretability as a dynamic of human-AI interaction","volume":"27","author":"Thieme","year":"2020","journal-title":"Interactions"},{"key":"ref_134","unstructured":"Walzner, D.D., Fuegener, A., and Gupta, A. (2022, January 9\u201314). Managing AI advice in crowd decision-making. Proceedings of the International Conference on Information Systems, Copenhagen, Denmark."},{"key":"ref_135","doi-asserted-by":"crossref","first-page":"76","DOI":"10.15346\/hc.v8i2.121","article-title":"Exploring the use of deep learning with crowdsourcing to annotate images","volume":"8","author":"Anjum","year":"2021","journal-title":"Hum. Comput."},{"key":"ref_136","doi-asserted-by":"crossref","first-page":"1902","DOI":"10.1073\/pnas.1807190116","article-title":"Citizen science frontiers: Efficiency, engagement, and serendipitous discovery with human-machine systems","volume":"116","author":"Trouille","year":"2019","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_137","unstructured":"Zhou, Z., and Yatani, K. (November, January 29). Gesture-aware interactive machine teaching with in-situ object annotations. Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, Bend, OR, USA."},{"key":"ref_138","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3449289","article-title":"Two cases for traces: A theoretical framing of mediated joint activity","volume":"5","author":"Avdic","year":"2021","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_139","doi-asserted-by":"crossref","unstructured":"Tchernavskij, P., and B\u00f8dker, S. (2022, January 8\u201312). Entangled artifacts: The meeting between a volunteer-run citizen science project and a biodiversity data platform. Proceedings of the Nordic Human-Computer Interaction Conference, Aarhus, Denmark.","DOI":"10.1145\/3546155.3546682"},{"key":"ref_140","doi-asserted-by":"crossref","unstructured":"Rzeszotarski, J.M., and Kittur, A. (2011, January 16\u201319). Instrumenting the crowd: Using implicit behavioral measures to predict task performance. Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, Santa Barbara, CA, USA.","DOI":"10.1145\/2047196.2047199"},{"key":"ref_141","doi-asserted-by":"crossref","unstructured":"Newman, A., McNamara, B., Fosco, C., Zhang, Y.B., Sukhum, P., Tancik, M., Kim, N.W., and Bylinskii, Z. (2020, January 25\u201330). TurkEyes: A web-based toolbox for crowdsourcing attention data. Proceedings of the CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.","DOI":"10.1145\/3313831.3376799"},{"key":"ref_142","doi-asserted-by":"crossref","unstructured":"Goyal, T., McDonnell, T., Kutlu, M., Elsayed, T., and Lease, M. (2018, January 5\u20138). Your behavior signals your reliability: Modeling crowd behavioral traces to ensure quality relevance annotations. Proceedings of the Sixth AAAI Conference on Human Computation and Crowdsourcing, Z\u00fcrich, Switzerland.","DOI":"10.1609\/hcomp.v6i1.13331"},{"key":"ref_143","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3415181","article-title":"CrowdCog: A cognitive skill based system for heterogeneous task assignment and recommendation in crowdsourcing","volume":"4","author":"Hettiachchi","year":"2020","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_144","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1145\/3436954","article-title":"UX designers pushing AI in the enterprise: A case for adaptive UIs","volume":"28","author":"Zimmerman","year":"2020","journal-title":"Interactions"},{"key":"ref_145","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3494522","article-title":"A survey on task assignment in crowdsourcing","volume":"55","author":"Hettiachchi","year":"2022","journal-title":"ACM Comput. Surv."},{"key":"ref_146","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3479586","article-title":"Quality control in crowdsourcing based on fine-grained behavioral features","volume":"5","author":"Pei","year":"2021","journal-title":"Proc. ACM Hum.-Comput. Interact."},{"key":"ref_147","doi-asserted-by":"crossref","first-page":"102121","DOI":"10.1016\/j.ijinfomgt.2020.102121","article-title":"Comparison of crowdsourcing platforms from social-psychological and motivational perspectives","volume":"54","author":"Bakici","year":"2020","journal-title":"Int. J. Inf. Manag."},{"key":"ref_148","doi-asserted-by":"crossref","unstructured":"Truong, N.V.-Q., Dinh, L.C., Stein, S., Tran-Thanh, L., and Jennings, N.R. (2022). Efficient and adaptive incentive selection for crowdsourcing contests. Appl. Intell., 1\u201331.","DOI":"10.1007\/s10489-022-03593-2"},{"key":"ref_149","unstructured":"Correia, A., Jameel, S., Paredes, H., Fonseca, B., and Schneider, D. (2019). Macrotask Crowdsourcing\u2014Engaging the Crowds to Address Complex Problems, Springer."},{"key":"ref_150","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1177\/0950017019886511","article-title":"Work precarity and gig literacies in online freelancing","volume":"34","author":"Sutherland","year":"2019","journal-title":"Work Employ. Soc."},{"key":"ref_151","doi-asserted-by":"crossref","unstructured":"Salminen, J., Kamel, A.M.S., Jung, S.-G., Mustak, M., and Jansen, B.J. (2022). Fair compensation of crowdsourcing work: The problem of flat rates. Behav. Inf. Technol., 1\u201322.","DOI":"10.1080\/0144929X.2022.2150564"}],"container-title":["Applied Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-3417\/13\/4\/2198\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:27:43Z","timestamp":1760120863000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-3417\/13\/4\/2198"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,8]]},"references-count":151,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["app13042198"],"URL":"https:\/\/doi.org\/10.3390\/app13042198","relation":{},"ISSN":["2076-3417"],"issn-type":[{"value":"2076-3417","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,8]]}}}