{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,22]],"date-time":"2026-01-22T11:44:36Z","timestamp":1769082276715,"version":"3.49.0"},"reference-count":81,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2022,3,26]],"date-time":"2022-03-26T00:00:00Z","timestamp":1648252800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,3,26]],"date-time":"2022-03-26T00:00:00Z","timestamp":1648252800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001858","name":"VINNOVA","doi-asserted-by":"publisher","award":["DNR 2017-03527 f"],"award-info":[{"award-number":["DNR 2017-03527 f"]}],"id":[{"id":"10.13039\/501100001858","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100011898","name":"Marianne and Marcus Wallenberg Foundation","doi-asserted-by":"publisher","award":["MMW2018-0036"],"award-info":[{"award-number":["MMW2018-0036"]}],"id":[{"id":"10.13039\/501100011898","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005760","name":"University of Gothenburg","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005760","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["AI &amp; Soc"],"published-print":{"date-parts":[[2024,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Citizen science (CS) projects have started to utilize Machine Learning (ML) to sort through large datasets generated in fields like astronomy, ecology and biodiversity, biology, and neuroimaging. Human\u2013machine systems have been created to take advantage of the complementary strengths of humans and machines and have been optimized for efficiency and speed. We conducted qualitative content analysis on meta-summaries of documents reporting the results of 12 citizen science projects that used machine learning to optimize classification tasks. We examined the distribution of tasks between citizen scientists, experts, and algorithms, and how epistemic agency was enacted in terms of whose knowledge shapes the distribution of tasks, who decides what knowledge is relevant to the classification, and who validates it. In our descriptive results, we found that experts, who include professional scientists and algorithm developers, are involved in every aspect of a project, from annotating or labelling data to giving data to algorithms to train them to make decisions from predictions. Experts also test and validate models to improve their accuracy by scoring their outputs when algorithms fail to make correct decisions. Experts are mostly the humans involved in a loop, but when algorithms encounter problems, citizens are also involved at several stages. In this paper, we present three main examples of citizens-in-the-loop: (a) when algorithms provide incorrect suggestions; (b) when algorithms fail to know how to perform classification; and (c) when algorithms pose queries. We consider the implications of the emphasis on optimization on the ideal of science and the role of citizen scientists from a perspective informed by Science and Technology Studies (STS) and Information Systems (IS). Based on our findings, we conclude that ML in CS classification projects, far from being deterministic in its nature and effects, may be open to question. There is no guarantee that these technologies can replace citizen scientists, nor any guarantee that they can provide citizens with opportunities for more interesting tasks.<\/jats:p>","DOI":"10.1007\/s00146-022-01428-9","type":"journal-article","created":{"date-parts":[[2022,3,28]],"date-time":"2022-03-28T21:06:32Z","timestamp":1648501592000},"page":"523-540","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Narratives of epistemic agency in citizen science classification projects: ideals of science and roles of citizens"],"prefix":"10.1007","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4708-4048","authenticated-orcid":false,"given":"Marisa","family":"Ponti","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4539-2055","authenticated-orcid":false,"given":"Dick","family":"Kasperowski","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6170-2653","authenticated-orcid":false,"given":"Anna Jia","family":"Gander","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,3,26]]},"reference":[{"key":"1428_CR1","doi-asserted-by":"publisher","first-page":"276","DOI":"10.1111\/phis.12014","volume":"23","author":"K Ahlstrom-Vij","year":"2013","unstructured":"Ahlstrom-Vij K (2013) Why we cannot rely on ourselves for epistemic improvement. Philos Issues 23:276\u2013296","journal-title":"Philos Issues"},{"key":"1428_CR3","volume-title":"Narratology: introduction to the theory of narrative","author":"M Bal","year":"2009","unstructured":"Bal M (2009) Narratology: introduction to the theory of narrative. University of Toronto Press, Toronto"},{"key":"1428_CR4","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1016\/j.npls.2016.01.001","volume":"2","author":"M Bengtsson","year":"2016","unstructured":"Bengtsson M (2016) How to plan and perform a qualitative study using content analysis. NursingPlus Open 2:8\u201314. https:\/\/doi.org\/10.1016\/j.npls.2016.01.001","journal-title":"NursingPlus Open"},{"issue":"1","key":"1428_CR5","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1007\/BF01905886","volume":"1","author":"M Boden","year":"1987","unstructured":"Boden M (1987) Artificial intelligence: cannibal or missionary. AI & Soc 1(1):17\u201323","journal-title":"AI & Soc"},{"issue":"4","key":"1428_CR01","first-page":"7","volume":"15","author":"R Bonney","year":"1996","unstructured":"Bonney R (1996) Citizen science: a lab tradition. Living Bird 15(4):7\u201315","journal-title":"Living Bird"},{"issue":"11","key":"1428_CR6","doi-asserted-by":"publisher","first-page":"977","DOI":"10.1525\/bio.2009.59.11.9","volume":"59","author":"R Bonney","year":"2009","unstructured":"Bonney R, Cooper CB, Dickinson J, Kelling S, Phillips T, Rosenberg KV, Shirk J (2009) Citizen science: a developing tool for expanding science knowledge and scientific literacy. Bioscience 59(11):977\u2013984. https:\/\/doi.org\/10.1525\/bio.2009.59.11.9","journal-title":"Bioscience"},{"issue":"2","key":"1428_CR7","doi-asserted-by":"publisher","first-page":"27","DOI":"10.3316\/QRJ0902027","volume":"9","author":"GA Bowen","year":"2009","unstructured":"Bowen GA (2009) Document analysis as a qualitative research method. Qual Res J 9(2):27\u201340. https:\/\/doi.org\/10.3316\/QRJ0902027","journal-title":"Qual Res J"},{"key":"1428_CR8","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9780262013246.001.0001","volume-title":"Science in democracy: expertise, institutions, and representation","author":"MB Brown","year":"2009","unstructured":"Brown MB (2009) Science in democracy: expertise, institutions, and representation. The MIT Press, New York"},{"key":"1428_CR9","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1257\/pandp.20181019","volume":"108","author":"E Brynjolfsson","year":"2018","unstructured":"Brynjolfsson E, Mitchell T, Rock D (2018) Economic consequences of Artificial Intelligence and robotics: what can machines learn and what does it mean for occupations and the economy? AEA Papers Proc 108:43\u201347. https:\/\/doi.org\/10.1257\/pandp.20181019","journal-title":"AEA Papers Proc"},{"key":"1428_CR10","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1007\/978-1-349-07408-2_2","volume-title":"Mapping the dynamics of science and technology: sociology of science in the real world","author":"M Callon","year":"1986","unstructured":"Callon M (1986) The sociology of an actor-network: the case of the electric vehicle. In: Callon M, Law J, Rip A (eds) Mapping the dynamics of science and technology: sociology of science in the real world. Palgrave Macmillan, Basingstoke, pp 19\u201334"},{"issue":"1","key":"1428_CR11","doi-asserted-by":"publisher","first-page":"29","DOI":"10.5334\/cstp.241","volume":"4","author":"L Ceccaroni","year":"2019","unstructured":"Ceccaroni L, Bibby J, Roger E et al (2019) Opportunities and risks for citizen science in the age of artificial intelligence. Citizen Sci Theory Pract 4(1):29. https:\/\/doi.org\/10.5334\/cstp.241","journal-title":"Citizen Sci Theory Pract"},{"key":"1428_CR12","doi-asserted-by":"publisher","DOI":"10.1007\/s00146-021-01259-0","author":"J Chubb","year":"2021","unstructured":"Chubb J, Cowling P, Reed D (2021) Speeding up to keep up: exploring the use of AI in the research process. AI & Soc. https:\/\/doi.org\/10.1007\/s00146-021-01259-0","journal-title":"AI & Soc"},{"key":"1428_CR13","unstructured":"Clemente JC (2020) Can a bot named Sam help citizen scientists save our oceans? IBM Blogs, 2020. https:\/\/www.ibm.com\/blogs\/industries\/unep-ai-marine-pollution-sam-virtual-human-citizen-science\/. Accessed 18 Oct 2021"},{"key":"1428_CR14","doi-asserted-by":"publisher","DOI":"10.7208\/chicago\/9780226113623.001.0001","volume-title":"Rethinking expertise","author":"H Collins","year":"2007","unstructured":"Collins H, Evans R (2007) Rethinking expertise. The University of Chicago Press, Chicago"},{"key":"1428_CR17","doi-asserted-by":"publisher","DOI":"10.1177\/0963662511434894","author":"AW Crall","year":"2012","unstructured":"Crall AW et al (2012) The impacts of an invasive species citizen science training program on participant attitudes, behaviour, and science literacy. Public Underst Sci. https:\/\/doi.org\/10.1177\/0963662511434894","journal-title":"Public Underst Sci"},{"issue":"1","key":"1428_CR18","doi-asserted-by":"publisher","first-page":"16","DOI":"10.5334\/cstp.166","volume":"4","author":"K Crowston","year":"2019","unstructured":"Crowston K, Mitchell E, \u00d8sterlund C (2019) Coordinating advanced crowd work: extending citizen science. Citizen Sci Theory Pract 4(1):16. https:\/\/doi.org\/10.5334\/cstp.166","journal-title":"Citizen Sci Theory Pract"},{"key":"1428_CR19","volume-title":"Essays on actions and events","author":"D Davidson","year":"1980","unstructured":"Davidson D (1980) Essays on actions and events. Oxford University Press, Oxford"},{"issue":"1","key":"1428_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.5334\/cstp.96","volume":"2","author":"MV Eitzel","year":"2017","unstructured":"Eitzel MV, Cappadonna JL, Santos-Lang C et al (2017) Citizen science terminology matters: exploring key terms. Citizen Sci Theory Pract 2(1):1. https:\/\/doi.org\/10.5334\/cstp.96","journal-title":"Citizen Sci Theory Pract"},{"issue":"4","key":"1428_CR22","doi-asserted-by":"publisher","first-page":"408","DOI":"10.1177\/016224399502000402","volume":"20","author":"S Epstein","year":"1995","unstructured":"Epstein S (1995) The construction of lay expertise: AIDS activism and the forging of credibility in the reform of clinical trials. Sci Technol Human Values 20(4):408\u2013437. https:\/\/doi.org\/10.1177\/016224399502000402","journal-title":"Sci Technol Human Values"},{"issue":"1","key":"1428_CR23","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1016\/j.infoandorg.2018.02.005","volume":"28","author":"S Faraj","year":"2018","unstructured":"Faraj S, Pachidi S, Sayegh K (2018) Working and organizing in the age of the learning algorithm. Inf Organ 28(1):62\u201370. https:\/\/doi.org\/10.1016\/j.infoandorg.2018.02.005","journal-title":"Inf Organ"},{"key":"1428_CR24","doi-asserted-by":"publisher","DOI":"10.4324\/9780203849088","volume-title":"Actor-network theory in education","author":"T Fenwick","year":"2010","unstructured":"Fenwick T, Edwards R (2010) Actor-network theory in education. Routledge, Abingdon"},{"key":"1428_CR25","volume-title":"Human engineering for an effective air-navigation and traffic-control system","author":"PM Fitts","year":"1951","unstructured":"Fitts PM (1951) Human engineering for an effective air-navigation and traffic-control system. Division of National Research Council, Oxford"},{"key":"1428_CR26","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1007\/978-3-030-58278-4_10","volume-title":"The science of citizen science","author":"M Franzen","year":"2021","unstructured":"Franzen M, Kloetzer L, Ponti M, Trojan J, Vicens J (2021) Machine learning in citizen science: promises and implications. In: Vohland K, Land-Zandstra A, Ceccaroni L, Lemmens R, Perell\u00f3 J, Ponti M et al (eds) The science of citizen science. Springer, Cham, pp 183\u2013198. https:\/\/doi.org\/10.1007\/978-3-030-58278-4_10"},{"issue":"1","key":"1428_CR27","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1073\/pnas.1408907112","volume":"43","author":"C Franzoni","year":"2014","unstructured":"Franzoni C, Sauermann H (2014) Crowd science: the organization of scientific research in open collaborative projects. Res Policy 43(1):1\u201320. https:\/\/doi.org\/10.1073\/pnas.1408907112","journal-title":"Res Policy"},{"key":"1428_CR28","doi-asserted-by":"publisher","DOI":"10.1080\/13662716.2021.1976627","author":"C Franzoni","year":"2021","unstructured":"Franzoni C, Poetz M, Sauermann H (2021) Crowds, citizens, and science: a multi-dimensional framework and agenda for future research. Ind Innov. https:\/\/doi.org\/10.1080\/13662716.2021.1976627","journal-title":"Ind Innov"},{"key":"1428_CR29","doi-asserted-by":"publisher","first-page":"922","DOI":"10.1038\/s41893-019-0390-3","volume":"2","author":"S Fritz","year":"2019","unstructured":"Fritz S, See L, Carlson T et al (2019) Citizen science and the United Nations Sustainable Development Goals. Nat Sustain 2:922\u2013930. https:\/\/doi.org\/10.1038\/s41893-019-0390-3","journal-title":"Nat Sustain"},{"key":"1428_CR30","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-009-0621-1","volume-title":"Uncertainty and quality in science for policy","author":"SO Funtowicz","year":"1990","unstructured":"Funtowicz SO, Ravetz J (1990) Uncertainty and quality in science for policy. Kluwer Academic Publishers, Dordrecht"},{"key":"1428_CR31","unstructured":"Geoghegan H, Dyke A, Pateman R, West S, Everett G (2016) Understanding Motivations for Citizen Science. Final report on behalf of UKEOF. University of Reading, Stockholm Environment Institute (University of York) and University of the West of England, UK"},{"key":"1428_CR32","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1177\/26317877211004609","volume":"2","author":"VL Glaser","year":"2021","unstructured":"Glaser VL, Pollock N, D\u2019Adderio L (2021) The biography of an algorithm: performing algorithmic technologies in organizations. Organization Theory 2:1\u201327. https:\/\/doi.org\/10.1177\/26317877211004609","journal-title":"Organization Theory"},{"issue":"1","key":"1428_CR33","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1007\/s11024-016-9308-7","volume":"5","author":"R Grundmann","year":"2017","unstructured":"Grundmann R (2017) The problem of expertise in knowledge societies. Minerva 5(1):25\u201348. https:\/\/doi.org\/10.1007\/s11024-016-9308-7","journal-title":"Minerva"},{"key":"1428_CR34","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1016\/0001-6918(69)90073-0","volume":"31","author":"JR Hackman","year":"1969","unstructured":"Hackman JR (1969) Toward understanding the role of tasks in behavioral research. Acta Physiol (oxf) 31:97\u2013128. https:\/\/doi.org\/10.1016\/0001-6918(69)90073-0","journal-title":"Acta Physiol (oxf)"},{"issue":"6","key":"1428_CR35","doi-asserted-by":"publisher","first-page":"364","DOI":"10.1016\/j.jnc.2012.08.002","volume":"20","author":"SJ Hobbs","year":"2012","unstructured":"Hobbs SJ, White PCL (2012) Motivations and barriers in relation to community participation in biodiversity recording. J Nat Conserv 20(6):364\u2013373. https:\/\/doi.org\/10.1016\/j.jnc.2012.08.002","journal-title":"J Nat Conserv"},{"issue":"2","key":"1428_CR36","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1016\/j.tree.2011.11.006","volume":"27","author":"WM Hochachka","year":"2012","unstructured":"Hochachka WM, Fink D, Hutchinson RA, Sheldon D, Wong WK, Kelling S (2012) Data-intensive science applied to broad-scale citizen science. Trends Ecol Evol 27(2):130\u2013137. https:\/\/doi.org\/10.1016\/j.tree.2011.11.006","journal-title":"Trends Ecol Evol"},{"issue":"9","key":"1428_CR37","doi-asserted-by":"publisher","first-page":"1277","DOI":"10.1177\/1049732305276687","volume":"15","author":"H-F Hsieh","year":"2005","unstructured":"Hsieh H-F, Shannon SE (2005) Three approaches to qualitative content analysis. Qual Health Res 15(9):1277\u20131288. https:\/\/doi.org\/10.1177\/1049732305276687","journal-title":"Qual Health Res"},{"key":"1428_CR38","volume-title":"Citizen science: a study of people, expertise and sustainable development","author":"A Irwin","year":"1995","unstructured":"Irwin A (1995) Citizen science: a study of people, expertise and sustainable development. Routledge, London"},{"key":"1428_CR39","unstructured":"Ischell (2019) Artificial intelligence meets citizen science. Discover. https:\/\/www.discovermagazine.com\/technology\/artificial-intelligence-meets-citizen-science"},{"issue":"4","key":"1428_CR40","doi-asserted-by":"publisher","first-page":"564","DOI":"10.1177\/0306312718778806","volume":"48","author":"D Kasperowski","year":"2018","unstructured":"Kasperowski D, Hillman T (2018) The epistemic culture in an online citizen science project: programs, antiprograms and epistemic subjects. Soc Stud Sci 48(4):564\u2013588. https:\/\/doi.org\/10.1177\/0306312718778806","journal-title":"Soc Stud Sci"},{"key":"1428_CR41","doi-asserted-by":"publisher","first-page":"158","DOI":"10.4324\/9780429489839-10","volume-title":"Designs for experimentation and inquiry: approaching learning and knowing in digital transformation","author":"D Kasperowski","year":"2019","unstructured":"Kasperowski D, Kullenberg C, Rohden F (2019) The epistemology of mobilising citizens in the sciences: tensions in epistemic cultures of contribution and ideals of science. In: M\u00e4kitalo \u00c5, Nicewonger TE, Elam M (eds) Designs for experimentation and inquiry: approaching learning and knowing in digital transformation, 1st edn. Routledge, Oxon, pp 158\u2013177","edition":"1"},{"key":"1428_CR42","unstructured":"Kasperowski D, Hagen N (2022) Ethical boundary work in citizen science: themes of insufficiency. Nordic J Sci Technol Stud 9(2):13\u201324. https:\/\/www.ntnu.no\/ojs\/index.php\/njsts\/article\/view\/4318"},{"issue":"1","key":"1428_CR43","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1609\/aimag.v34i1.2431","volume":"34","author":"S Kelling","year":"2012","unstructured":"Kelling S, Gerbracht J, Fink D et al (2012) A human\/computer learning network to improve biodiversity conservation and research. AI Mag 34(1):10\u201310. https:\/\/doi.org\/10.1609\/aimag.v34i1.2431","journal-title":"AI Mag"},{"key":"1428_CR44","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3389\/fphy.2021.650720","volume":"9","author":"I Khairunnisa","year":"2021","unstructured":"Khairunnisa I, Khodursky S, Yasseri T (2021) Gender imbalance and spatiotemporal patterns of contributions to citizen science projects: the case of Zooniverse. Front Phys 9:1\u201312. https:\/\/doi.org\/10.3389\/fphy.2021.650720","journal-title":"Front Phys"},{"key":"1428_CR45","doi-asserted-by":"publisher","DOI":"10.4159\/9780674039681","volume-title":"Epistemic cultures. How the sciences make knowledge","author":"K Knorr Cetina","year":"1999","unstructured":"Knorr Cetina K (1999) Epistemic cultures. How the sciences make knowledge. Harvard University Press, Cambridge"},{"issue":"4","key":"1428_CR46","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1179\/030801807X163571","volume":"32","author":"K Knorr-Cetina","year":"2007","unstructured":"Knorr-Cetina K (2007) Culture in global knowledge societies: knowledge cultures and epistemic cultures. Interdisc Sci Rev 32(4):361\u2013375. https:\/\/doi.org\/10.1179\/030801807X163571","journal-title":"Interdisc Sci Rev"},{"key":"1428_CR47","volume-title":"Positivist philosophy. From Hume to the Vienna Circle","author":"L Kolakowski","year":"1972","unstructured":"Kolakowski L (1972) Positivist philosophy. From Hume to the Vienna Circle. Pelican Books, Harmondsworth"},{"key":"1428_CR48","first-page":"19","volume-title":"Representation in scientific practice","author":"B Latour","year":"1990","unstructured":"Latour B (1990) Drawing things together. In: Lynch M, Woolgar S (eds) Representation in scientific practice. MIT Press, Cambridge, pp 19\u201368"},{"key":"1428_CR50","unstructured":"Latour B (1992) Where are the missing masses, sociology of a few mundane artefacts. In: Bijker W, Law J (eds) Shaping Technology-Building Society. Studies in Sociotechnical Change. MIT Press, Cambridge, pp. 225\u2013259. [New expanded and revised version of article (35). Republication in the reader Johnson Deborah J, Jameson M W (eds) Technology and Society, Building Our Sociotechnical Future. MIT Press, Cambridge Mass, 2008 pp. 151\u2013180]"},{"key":"1428_CR49","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780199256044.001.0001","volume-title":"Reassembling the social: an introduction to actor-network theory","author":"B Latour","year":"2005","unstructured":"Latour B (2005) Reassembling the social: an introduction to actor-network theory. Oxford University Press, Oxford"},{"key":"1428_CR51","doi-asserted-by":"publisher","DOI":"10.7249\/RR4401","volume-title":"Emerging developments in citizen science: reflecting on areas of innovation","author":"B Leach","year":"2020","unstructured":"Leach B, Parkinson S, Lichten CA et al (2020) Emerging developments in citizen science: reflecting on areas of innovation. RAND Corporation, Santa Monica. https:\/\/doi.org\/10.7249\/RR4401"},{"issue":"3","key":"1428_CR52","doi-asserted-by":"publisher","first-page":"1501","DOI":"10.25300\/MISQ\/2021\/16564","volume":"45","author":"S Lebovitz","year":"2021","unstructured":"Lebovitz S, Levina N, Lifshitz-Assaf H (2021) Is AI ground truth really true? The dangers of training and evaluating AI tools based on experts\u2019 know-what. MIS Quart 45(3):1501\u20131525. https:\/\/doi.org\/10.25300\/MISQ\/2021\/16564","journal-title":"MIS Quart"},{"key":"1428_CR54","doi-asserted-by":"publisher","first-page":"8087","DOI":"10.3390\/su13148087","volume":"13","author":"M Lotfian","year":"2021","unstructured":"Lotfian M, Ingensand J, Brovelli MA (2021) The partnership of citizen science and machine learning: benefits, risks, and future challenges for engagement, data collection, and data quality. Sustainability 13:8087. https:\/\/doi.org\/10.3390\/su13148087","journal-title":"Sustainability"},{"issue":"10","key":"1428_CR56","doi-asserted-by":"publisher","first-page":"1250","DOI":"10.15252\/embr.201541001","volume":"16","author":"F Mazzocchi","year":"2015","unstructured":"Mazzocchi F (2015) Could Big Data be the end of theory in science? A few remarks on the epistemology of data-driven science. EMBO Rep 16(10):1250\u20131255. https:\/\/doi.org\/10.15252\/embr.201541001","journal-title":"EMBO Rep"},{"issue":"7","key":"1428_CR57","doi-asserted-by":"publisher","first-page":"100109","DOI":"10.1016\/j.patter.2020.100109","volume":"1","author":"EC McClure","year":"2020","unstructured":"McClure EC, Sievers M, Brown CJ, Buelow CA, Ditria EM, Hayes MA et al (2020) Artificial intelligence meets citizen science to supercharge ecological monitoring. Patterns (NY) 1(7):100109. https:\/\/doi.org\/10.1016\/j.patter.2020.100109","journal-title":"Patterns (NY)"},{"issue":"2","key":"1428_CR58","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1177\/1049731517745600","volume":"30","author":"ME Mor Barak","year":"2020","unstructured":"Mor Barak ME (2020) The Practice and Science of Social Good: Emerging Paths to Positive Social Impact. Res Soc Work Pract 30(2):139\u2013150. https:\/\/doi.org\/10.1177\/1049731517745600","journal-title":"Res Soc Work Pract"},{"issue":"2","key":"1428_CR60","doi-asserted-by":"publisher","first-page":"244","DOI":"10.1177\/0162243909337121","volume":"35","author":"G Ottinger","year":"2010","unstructured":"Ottinger G (2010) Buckets of resistance: standards and the effectiveness of citizen science. Sci Technol Human Values 35(2):244\u2013270. https:\/\/doi.org\/10.1177\/0162243909337121","journal-title":"Sci Technol Human Values"},{"issue":"1","key":"1428_CR61","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1287\/orsc.2020.1377","volume":"32","author":"S Pachidi","year":"2021","unstructured":"Pachidi S, Berends H, Faraj S, Huysman M (2021) Make way for the algorithms: symbolic actions and change in a regime of knowing. Organ Sci 32(1):18\u201341. https:\/\/doi.org\/10.1287\/orsc.2020.1377","journal-title":"Organ Sci"},{"issue":"1","key":"1428_CR62","doi-asserted-by":"publisher","first-page":"9","DOI":"10.5334\/cstp.369","volume":"6","author":"R Pateman","year":"2021","unstructured":"Pateman R, Dyke A, West S (2021) The diversity of participants in environmental citizen science. Citizen Sci Theory Pract 6(1):9. https:\/\/doi.org\/10.5334\/cstp.369","journal-title":"Citizen Sci Theory Pract"},{"key":"1428_CR63","volume-title":"Qualitative research and evaluation methods","author":"MQ Patton","year":"2002","unstructured":"Patton MQ (2002) Qualitative research and evaluation methods, 3rd edn. Sage Publications, Thousand Oaks","edition":"3"},{"key":"1428_CR64","volume-title":"Software and Organisations: the biography of the enterprise- wide system or how SAP conquered the world","author":"N Pollock","year":"2009","unstructured":"Pollock N, Williams R (2009) Software and Organisations: the biography of the enterprise- wide system or how SAP conquered the world. Routledge, London"},{"key":"1428_CR2","doi-asserted-by":"publisher","unstructured":"Ponti M, Seredko A (2022) Human-machine-learning integration and task allocation in citizen science. Humanit Soc Sci Commun 9(48). https:\/\/doi.org\/10.1057\/s41599-022-01049-z","DOI":"10.1057\/s41599-022-01049-z"},{"issue":"2","key":"1428_CR65","doi-asserted-by":"publisher","first-page":"5","DOI":"10.15346\/hc.v8i2.128","volume":"8","author":"M Ponti","year":"2021","unstructured":"Ponti M, Kloetzer L, Ostermann FO, Miller G, Schade S (2021) Can\u2019t we all just get along? Citizen scientists interacting with algorithms. Hum Comput 8(2):5\u201314. https:\/\/doi.org\/10.15346\/hc.v8i2.128","journal-title":"Hum Comput"},{"key":"1428_CR66","doi-asserted-by":"publisher","DOI":"10.1186\/s41039-017-0062-8","author":"SAD Popenici","year":"2017","unstructured":"Popenici SAD, Kerr S (2017) Exploring the impact of artificial intelligence on teaching and learning in higher education. Res Pract Technol Enhanced Learn. https:\/\/doi.org\/10.1186\/s41039-017-0062-8","journal-title":"Res Pract Technol Enhanced Learn"},{"key":"1428_CR67","volume-title":"The logic of scientific discovery","author":"K Popper","year":"1934","unstructured":"Popper K (1934) The logic of scientific discovery. Julius Springer, Vienna"},{"key":"1428_CR68","doi-asserted-by":"publisher","first-page":"191","DOI":"10.7591\/cornell\/9780801449116.003.0014","volume-title":"Citizen science: public participation in environmental research","author":"K Purcell","year":"2012","unstructured":"Purcell K, Garibay C, Dickinson JL (2012) A gateway to science for all: celebrate urban birds. In: Dickinson JL, Louv R, Bonney R (eds) Citizen science: public participation in environmental research. Comstock Publication Associates, Ithaca, pp 191\u2013200"},{"key":"1428_CR69","unstructured":"QSR International (2020) NVivo Qualitative Data Analysis Software [Software]"},{"key":"1428_CR70","unstructured":"Raddick MJ, Bracey G, Carney C, Gyuk G et al (2009) Citizen science: status and research directions for the coming decade. Astro2010: The Astronomy and Astrophysics Decadal Survey, Position Papers, no. 46"},{"issue":"5","key":"1428_CR71","doi-asserted-by":"publisher","first-page":"736","DOI":"10.1177\/0306312714539058","volume":"44","author":"L Reyes-Galindo","year":"2014","unstructured":"Reyes-Galindo L (2014) Linking the subcultures of physics: virtual empiricism and the bonding role of trust. Soc Stud Sci 44(5):736\u2013757. https:\/\/doi.org\/10.1177\/0306312714539058","journal-title":"Soc Stud Sci"},{"issue":"4","key":"1428_CR72","doi-asserted-by":"publisher","first-page":"226","DOI":"10.1097\/00006199-200307000-00004","volume":"52","author":"M Sandelowski","year":"2003","unstructured":"Sandelowski M, Barroso M (2003) Creating metasummaries of qualitative studies. Nurs Res 52(4):226\u2013233. https:\/\/doi.org\/10.1097\/00006199-200307000-00004","journal-title":"Nurs Res"},{"issue":"3","key":"1428_CR73","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1073\/pnas.1408907112","volume":"112","author":"H Sauermann","year":"2015","unstructured":"Sauermann H, Franzoni C (2015) Crowd science user contribution patterns and their implications. PNAS 112(3):679\u2013684. https:\/\/doi.org\/10.1073\/pnas.1408907112","journal-title":"PNAS"},{"issue":"5","key":"1428_CR74","doi-asserted-by":"publisher","DOI":"10.1016\/j.respol.2020.103978","volume":"49","author":"H Sauermann","year":"2020","unstructured":"Sauermann H, Vohland K, Antoniou V, Balazs B, Goebel C, Karatzas K, Mooney P, Perello J, Ponti M et al (2020) Citizen science and sustainability transitions. Res Policy 49(5):103978. https:\/\/doi.org\/10.1016\/j.respol.2020.103978","journal-title":"Res Policy"},{"key":"1428_CR75","doi-asserted-by":"publisher","first-page":"412","DOI":"10.2307\/j.ctvc77mp9.30","volume-title":"Digitalsts: a field guide for science & technology studies","author":"N Seaver","year":"2019","unstructured":"Seaver N (2019) Knowing algorithms. In: Vertesi J, Ribes D (eds) Digitalsts: a field guide for science & technology studies. Princeton University Press, Princeton, pp 412\u2013422"},{"issue":"1","key":"1428_CR76","doi-asserted-by":"publisher","first-page":"5","DOI":"10.5334\/cstp.66","volume":"2","author":"V Seymour","year":"2017","unstructured":"Seymour V, Haklay M (2017) Exploring engagement characteristics and behaviours of environmental volunteers. Citizen Sci Theory Pract 2(1):5. https:\/\/doi.org\/10.5334\/cstp.66","journal-title":"Citizen Sci Theory Pract"},{"key":"1428_CR77","unstructured":"Shanley LA, Fortson L, Berger-Wolf T, Crowston K, Michelucci P (2021) Imagine all the people: citizen science, artificial intelligence, and computational research. Computing Community Consortium (CCC), Washington D.C."},{"key":"1428_CR78","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1007\/978-3-319-90403-0_3","volume-title":"Human and ML. Visible, explainable, trustworthy and transparent","author":"PC Shih","year":"2018","unstructured":"Shih PC (2018) Beyond human-in-the-loop: Empowering end users with transparent ML. In: Zhou J, Chen F (eds) Human and ML. Visible, explainable, trustworthy and transparent. Springer, Cham, pp 37\u201354. https:\/\/doi.org\/10.1007\/978-3-319-90403-0_3"},{"key":"1428_CR80","unstructured":"Sloane M, Moss E, Awomolo O, Forlano L (2020) Participation is not a design fix for machine learning. arXiv:2007.02423v3\u00a0[cs.CY]"},{"key":"1428_CR81","unstructured":"Sloane M (2020) Participation-washing could be the next dangerous fad in machine learning. MIT Technology Review. https:\/\/www.technologyreview.com\/2020\/08\/25\/1007589\/participation-washing-ai-trends-opinion-machine-learning\/. Accessed 18 Oct 2021"},{"key":"1428_CR82","unstructured":"Strasser B, Haklay ME (2018) Citizen Science: Expertise, Democracy, and Public Participation. SSC Policy Analysis 1\/2018: 1\u201392. Swiss Science Council, Bern, Switzerland. https:\/\/discovery.ucl.ac.uk\/id\/eprint\/10062223. Accessed 10 Oct 2021"},{"key":"1428_CR84","volume-title":"Thinking machines: machine learning and its hardware implementation","author":"S Takano","year":"2021","unstructured":"Takano S (2021) Thinking machines: machine learning and its hardware implementation. Academic Press, London"},{"key":"1428_CR85","doi-asserted-by":"publisher","DOI":"10.1007\/s10111-020-00656-7","author":"A Tausch","year":"2020","unstructured":"Tausch A, Kluge A (2020) The best task allocation process is to decide on one\u2019s own: effects of the allocation agent in human\u2013robot interaction on perceived work characteristics and satisfaction. Cogn Tech Work. https:\/\/doi.org\/10.1007\/s10111-020-00656-7","journal-title":"Cogn Tech Work"},{"key":"1428_CR87","volume-title":"Cybernetics; or Control and Communication in the Animal and the Machine","author":"N Wiener","year":"1948","unstructured":"Wiener N (1948) Cybernetics; or Control and Communication in the Animal and the Machine. MA, MIT Press, Cambridge"},{"key":"1428_CR90","doi-asserted-by":"publisher","first-page":"1013","DOI":"10.1080\/08941920.2015.1054976","volume":"28","author":"DR Wright","year":"2015","unstructured":"Wright DR, Underhill LG, Keenec M, Knight AT (2015) Understanding the motivations and satisfactions of volunteers to improve the effectiveness of citizen science programs. Soc Nat Resour 28:1013\u20131029. https:\/\/doi.org\/10.1080\/08941920.2015.1054976","journal-title":"Soc Nat Resour"},{"issue":"3","key":"1428_CR91","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3362741","volume":"2","author":"DE Wright","year":"2019","unstructured":"Wright DE, Fortson L, Lintott CJ et al (2019) Help me to help you: machine augmented citizen science. ACM Trans Soc Comput 2(3):1\u201320. https:\/\/doi.org\/10.1145\/3362741","journal-title":"ACM Trans Soc Comput"}],"container-title":["AI &amp; SOCIETY"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00146-022-01428-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00146-022-01428-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00146-022-01428-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,10]],"date-time":"2024-04-10T12:02:21Z","timestamp":1712750541000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00146-022-01428-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,26]]},"references-count":81,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,4]]}},"alternative-id":["1428"],"URL":"https:\/\/doi.org\/10.1007\/s00146-022-01428-9","relation":{},"ISSN":["0951-5666","1435-5655"],"issn-type":[{"value":"0951-5666","type":"print"},{"value":"1435-5655","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,26]]},"assertion":[{"value":"7 November 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 March 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 March 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflict of interest to declare.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}