{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,6]],"date-time":"2025-10-06T18:28:26Z","timestamp":1759775306880,"version":"3.41.0"},"reference-count":83,"publisher":"Association for Computing Machinery (ACM)","issue":"1-4","license":[{"start":{"date-parts":[[2024,9,24]],"date-time":"2024-09-24T00:00:00Z","timestamp":1727136000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000780","name":"European Union","doi-asserted-by":"crossref","award":["G53D23002800006"],"award-info":[{"award-number":["G53D23002800006"]}],"id":[{"id":"10.13039\/501100000780","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Trans. Soc. Comput."],"published-print":{"date-parts":[[2024,12,31]]},"abstract":"<jats:p>\n            Crowdsourcing tasks have been widely used to collect a large number of human labels at scale. While some of these tasks are deployed by requesters and performed only once by crowd workers, others require the same worker to perform the same task or a variant of it more than once, thus participating in a so-called\n            <jats:italic>longitudinal study<\/jats:italic>\n            . Despite the prevalence of longitudinal studies in crowdsourcing, there is a limited understanding of factors that influence worker participation in them across different crowdsourcing marketplaces. We present results from a large-scale survey of 300 workers on 3 different micro-task crowdsourcing platforms: Amazon Mechanical Turk, Prolific, and Toloka. The aim is to understand how longitudinal studies are performed using crowdsourcing. We collect answers about 547 experiences and we analyze them both quantitatively and qualitatively. We synthesize 17 take-home messages about longitudinal studies together with 8 recommendations for task requesters and 5 best practices for crowdsourcing platforms to adequately conduct and support such kinds of studies. We release the survey and the data at: https:\/\/osf.io\/h4du9\/.\n          <\/jats:p>","DOI":"10.1145\/3674884","type":"journal-article","created":{"date-parts":[[2024,7,11]],"date-time":"2024-07-11T11:27:46Z","timestamp":1720697266000},"page":"1-49","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Longitudinal Loyalty: Understanding The Barriers To Running Longitudinal Studies On Crowdsourcing Platforms"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7337-7592","authenticated-orcid":false,"given":"Michael","family":"Soprano","sequence":"first","affiliation":[{"name":"Department of Mathematics, Computer Science and Physics, University of Udine, Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9191-3280","authenticated-orcid":false,"given":"Kevin","family":"Roitero","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Computer Science and Physics, University of Udine, Udine Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6189-6539","authenticated-orcid":false,"given":"Ujwal","family":"Gadiraju","sequence":"additional","affiliation":[{"name":"Delft University of Technology, Delft, Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5423-8669","authenticated-orcid":false,"given":"Eddy","family":"Maddalena","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Computer Science and Physics, University of Udine, Udine Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7311-3693","authenticated-orcid":false,"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[{"name":"The University of Queensland, Saint Lucia, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,9,24]]},"reference":[{"key":"e_1_3_4_2_2","first-page":"2","volume-title":"Proceedings of the 10th AAAI Conference on Human Computation and Crowdsourcing","volume":"10","author":"Abbas Tahir","year":"2022","unstructured":"Tahir Abbas and Ujwal Gadiraju. 2022. Goal-setting behavior of workers on crowdsourcing platforms: An exploratory study on MTurk and Prolific. In Proceedings of the 10th AAAI Conference on Human Computation and Crowdsourcing, Jane Hsu and Ming Yin (Eds.), Vol. 10. AAAI Press, Washington, DC, 2\u201313. DOI:10.1609\/hcomp.v10i1.21983"},{"issue":"1","key":"e_1_3_4_3_2","first-page":"1","article-title":"Tukey\u2019s honestly significant difference (HSD) test","volume":"3","author":"Abdi Herv\u00e9","year":"2010","unstructured":"Herv\u00e9 Abdi and Lynne J. Williams. 2010. Tukey\u2019s honestly significant difference (HSD) test. Encyclopedia of Research Design 3, 1 (2010), 1\u20135. https:\/\/personal.utdallas.edu\/Herve\/abdi-HSD2010-pretty.pdf","journal-title":"Encyclopedia of Research Design"},{"key":"e_1_3_4_4_2","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1145\/3468920.3468942","volume-title":"Proceedings of the 2021 3rd International Conference on Big Data Engineering (BDE\u201921)","author":"Aljohani Asmaa","year":"2021","unstructured":"Asmaa Aljohani and James Jones. 2021. Conducting malicious cybersecurity experiments on crowdsourcing platforms. In Proceedings of the 2021 3rd International Conference on Big Data Engineering (BDE\u201921) (Shanghai, China). ACM, New York,, 150\u2013161. DOI:10.1145\/3468920.3468942"},{"key":"e_1_3_4_5_2","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1007\/978-3-031-05563-8_20","volume-title":"HCI for Cybersecurity, Privacy and Trust","author":"Aljohani Asmaa","year":"2022","unstructured":"Asmaa Aljohani and James Jones. 2022. The pitfalls of evaluating cyber defense techniques by an anonymous population. In HCI for Cybersecurity, Privacy and Trust. Springer International Publishing, Cham, 307\u2013325."},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/0749-5978(85)90049-4"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0245460"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1002\/ir.102"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10676-023-09708-8"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2022.107605"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691610393980"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1093\/icc\/dtab022"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.3978\/j.issn.2072-1439.2015.10.63"},{"key":"e_1_3_4_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3274306"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1080\/17469899.2023.2200935"},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbusres.2015.05.001"},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.2196\/37004"},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.puhe.2022.08.003"},{"key":"e_1_3_4_19_2","volume-title":"Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI \u201924)","author":"Groot Esra Cemre Su de","year":"2024","unstructured":"Esra Cemre Su de Groot and Ujwal Gadiraju. 2024. \u201cAre we all in the same boat?\u201d Customizable and evolving avatars to improve worker engagement and foster a sense of community in online crowd work. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI \u201924) (Honolulu, HI, USA). ACM, New York, Article 640, 26 pages. DOI:10.1145\/3613904.3642429"},{"key":"e_1_3_4_20_2","first-page":"50","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"2","author":"Difallah Djellel","year":"2014","unstructured":"Djellel Difallah, Michele Catasta, Gianluca Demartini, and Philippe Cudr\u00e9-Mauroux. 2014. Scaling-up the crowd: Micro-task pricing schemes for worker retention and latency improvement. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 2. AAAI Press, Washington, DC, 50\u201358. DOI:10.1609\/hcomp.v2i1.13154"},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v9i1.18939"},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1080\/07421222.2019.1705506"},{"key":"e_1_3_4_23_2","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1145\/3447535.3462482","volume-title":"Proceedings of the 13th ACM Web Science Conference 2021 (WebSci \u201921)","author":"Edixhoven Tom","year":"2021","unstructured":"Tom Edixhoven, Sihang Qiu, Lucie Kuiper, Olivier Dikken, Gwennan Smitskamp, and Ujwal Gadiraju. 2021. Improving reactions to rejection in crowdsourcing through self-reflection. In Proceedings of the 13th ACM Web Science Conference 2021 (WebSci \u201921) (Virtual Event, United Kingdom). ACM, New York, 74\u201383. DOI:10.1145\/3447535.3462482"},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3415203"},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","unstructured":"Zachary Fulker and Christoph Riedl. 2023. Cooperation in Crowd Work: Attitude and Perception of Freelancers on a Knowledge Work Platform. DOI:10.48550\/arXiv.2301.08808","DOI":"10.48550\/arXiv.2301.08808"},{"issue":"3","key":"e_1_3_4_26_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3130914","article-title":"Modus operandi of crowd workers: The invisible role of microtask work environments","volume":"1","author":"Gadiraju Ujwal","year":"2017","unstructured":"Ujwal Gadiraju, Alessandro Checco, Neha Gupta, and Gianluca Demartini. 2017. Modus operandi of crowd workers: The invisible role of microtask work environments. In Proceedings of the ACM Conference on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1\u201329.","journal-title":"Proceedings of the ACM Conference on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"e_1_3_4_27_2","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1145\/3342220.3343644","volume-title":"Proceedings of the 30th ACM Conference on Hypertext and Social Media (HT \u201919)","author":"Gadiraju Ujwal","year":"2019","unstructured":"Ujwal Gadiraju and Gianluca Demartini. 2019. Understanding worker moods and reactions to rejection in crowdsourcing. In Proceedings of the 30th ACM Conference on Hypertext and Social Media (HT \u201919) (Hof, Germany). ACM, New York, 211\u2013220. DOI:10.1145\/3342220.3343644"},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2015.66"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-66435-4_2"},{"key":"e_1_3_4_30_2","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1145\/2984511.2984542","volume-title":"Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST \u201916)","author":"Gaikwad Snehalkumar (Neil) S.","year":"2016","unstructured":"Snehalkumar (Neil) S. Gaikwad, Durim Morina, Adam Ginzberg, Catherine Mullings, Shirish Goyal, Dilrukshi Gamage, Christopher Diemert, Mathias Burton, Sharon Zhou, Mark Whiting, Karolina Ziulkoski, Alipta Ballav, Aaron Gilbee, Senadhipathige S. Niranga, Vibhor Sehgal, Jasmine Lin, Leonardy Kristianto, Angela Richmond-Fuller, Jeff Regino, Nalin Chhibber, Dinesh Majeti, Sachin Sharma, Kamila Mananova, Dinesh Dhakal, William Dai, Victoria Purynova, Samarth Sandeep, Varshine Chandrakanthan, Tejas Sarma, Sekandar Matin, Ahmed Nasser, Rohit Nistala, Alexander Stolzoff, Kristy Milland, Vinayak Mathur, Rajan Vaish, and Michael S. Bernstein. 2016. Boomerang: Rebounding the consequences of reputation feedback on crowdsourcing platforms. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST \u201916) (Tokyo, Japan). Association for Computing Machinery, New York, NY, USA, 625\u2013637. DOI:10.1145\/2984511.2984542"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.josat.2023.209011"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijresmar.2022.10.002"},{"key":"e_1_3_4_33_2","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1145\/3573051.3593390","volume-title":"Proceedings of the 10th ACM Conference on Learning @ Scale","author":"Gurung Ashish","year":"2023","unstructured":"Ashish Gurung, Sami Baral, Morgan P. Lee, Adam C. Sales, Aaron Haim, Kirk P. Vanacore, Andrew A. McReynolds, Hilary Kreisberg, Cristina Heffernan, and Neil T. Heffernan. 2023. How common are common wrong answers? Crowdsourcing remediation at scale. In Proceedings of the 10th ACM Conference on Learning @ Scale (Copenhagen, Denmark) (L@S \u201923). ACM, New York., 70\u201380. DOI:10.1145\/3573051.3593390"},{"key":"e_1_3_4_34_2","first-page":"321","volume-title":"Proceedings of the 12th ACM International Conference on Web Search and Data Mining (WSDM\u201919)","author":"Han Lei","year":"2019","unstructured":"Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini. 2019. All those wasted hours: On task abandonment in crowdsourcing. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining (WSDM\u201919) (Melbourne VIC, Australia). ACM, New York, 321\u2013329. DOI:10.1145\/3289600.3291035"},{"key":"e_1_3_4_35_2","first-page":"1","volume-title":"Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI \u201918)","author":"Hara Kotaro","year":"2018","unstructured":"Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P. Bigham. 2018. A data-driven analysis of workers\u2019 earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI \u201918) (Montreal QC, Canada). ACM, New York, 1\u201314. DOI:10.1145\/3173574.3174023"},{"key":"e_1_3_4_36_2","doi-asserted-by":"crossref","first-page":"889","DOI":"10.1145\/2998181.2998248","volume-title":"Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing","author":"Hata Kenji","year":"2017","unstructured":"Kenji Hata, Ranjay Krishna, Li Fei-Fei, and Michael S. Bernstein. 2017. A glimpse far into the future: Understanding long-term crowd worker quality. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, New York, 889\u2013901. DOI:10.1145\/2998181.2998248"},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3494522"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2013.02.020"},{"issue":"6","key":"e_1_3_4_39_2","first-page":"1","article-title":"The rise of crowdsourcing","volume":"14","author":"Howe Jeff","year":"2006","unstructured":"Jeff Howe. 2006. The rise of crowdsourcing. Wired Magazine 14, 6 (2006), 1\u20134. https:\/\/www.wired.com\/2006\/06\/crowds\/","journal-title":"Wired Magazine"},{"key":"e_1_3_4_40_2","doi-asserted-by":"publisher","DOI":"10.1177\/1049732305276687"},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.3390\/su12083091"},{"key":"e_1_3_4_42_2","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1145\/2470654.2470742","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI \u201913)","author":"Irani Lilly C.","year":"2013","unstructured":"Lilly C. Irani and M. Six Silberman. 2013. Turkopticon: Interrupting worker invisibility in Amazon Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI \u201913) (Paris, France). ACM, 611\u2013620. DOI:10.1145\/2470654.2470742"},{"key":"e_1_3_4_43_2","volume-title":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI \u201922)","author":"Jung Ji-Youn","year":"2022","unstructured":"Ji-Youn Jung, Sihang Qiu, Alessandro Bozzon, and Ujwal Gadiraju. 2022. Great chain of agents: The role of metaphorical representation of agents in conversational crowdsourcing. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI \u201922) (New Orleans, LA,). ACM, New York, Article 57, 22 pages. DOI:10.1145\/3491102.3517653"},{"key":"e_1_3_4_44_2","doi-asserted-by":"publisher","DOI":"10.1017\/S026021050900850X"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.obhdp.2014.01.001"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.4018\/JGIM.20211101.oa13"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.2196\/40765"},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-016-0727-z"},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359174"},{"key":"e_1_3_4_50_2","first-page":"72","volume-title":"Adjunct Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization (UMAP \u201923 Adjunct)","author":"Mensio Martino","year":"2023","unstructured":"Martino Mensio, Gregoire Burel, Tracie Farrell, and Harith Alani. 2023. MisinfoMe: A tool for longitudinal assessment of Twitter accounts\u2019 sharing of misinformation. In Adjunct Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization (UMAP \u201923 Adjunct) (Limassol, Cyprus). ACM, New York, 72\u201375. DOI:10.1145\/3563359.3597396"},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1080\/14459795.2017.1284250"},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpain.2023.02.302"},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpain.2022.02.005"},{"key":"e_1_3_4_54_2","first-page":"165","volume-title":"Proceedings of the 32nd ACM Conference on Hypertext and Social Media (HT \u201921)","author":"Nouri Zahra","year":"2021","unstructured":"Zahra Nouri, Ujwal Gadiraju, Gregor Engels, and Henning Wachsmuth. 2021. What is unclear? Computational assessment of task clarity in crowdsourcing. In Proceedings of the 32nd ACM Conference on Hypertext and Social Media (HT \u201921) (Virtual Event, USA). ACM, New York, 165\u2013175. DOI:10.1145\/3465336.3475109"},{"key":"e_1_3_4_55_2","first-page":"737","volume-title":"Proceedings of the 28th International Conference on Intelligent User Interfaces (IUI \u201923)","author":"Nouri Zahra","year":"2023","unstructured":"Zahra Nouri, Nikhil Prakash, Ujwal Gadiraju, and Henning Wachsmuth. 2023. Supporting requesters in writing clear crowdsourcing task descriptions through computational flaw assessment. In Proceedings of the 28th International Conference on Intelligent User Interfaces (IUI \u201923) (Sydney, NSW, Australia). ACM, New York, 737\u2013749. DOI:10.1145\/3581641.3584039"},{"key":"e_1_3_4_56_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0284101"},{"key":"e_1_3_4_57_2","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.8.4.434"},{"issue":"1","key":"e_1_3_4_58_2","first-page":"1","article-title":"The state of pilot study reporting in crowdsourcing: A reflection on best practices and guidelines","volume":"8","author":"Oppenlaender Jonas","year":"2024","unstructured":"Jonas Oppenlaender, Tahir Abbas, and Ujwal Gadiraju. 2024. The state of pilot study reporting in crowdsourcing: A reflection on best practices and guidelines. In Proceedings of the ACM Conference on Human-Computer Interaction 8, CSCW1 (2024), 1\u201345.","journal-title":"Proceedings of the ACM Conference on Human-Computer Interaction"},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbef.2017.12.004"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1017\/S1930297500002205"},{"key":"e_1_3_4_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2017.01.006"},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-021-01694-3"},{"key":"e_1_3_4_63_2","doi-asserted-by":"publisher","DOI":"10.1177\/1035304620959750"},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10869-011-9209-6"},{"key":"e_1_3_4_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3476063"},{"key":"e_1_3_4_66_2","first-page":"1","volume-title":"Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI \u201920)","author":"Qiu Sihang","year":"2020","unstructured":"Sihang Qiu, Ujwal Gadiraju, and Alessandro Bozzon. 2020. Improving worker engagement through conversational microtask crowdsourcing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI \u201920) (Honolulu, HI, USA). ACM, New York, 1\u201312. DOI:10.1145\/3313831.3376403"},{"key":"e_1_3_4_67_2","first-page":"69","volume-title":"Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval (ICTIR \u201920)","author":"Qiu Sihang","year":"2020","unstructured":"Sihang Qiu, Ujwal Gadiraju, and Alessandro Bozzon. 2020. Towards memorable information retrieval. In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval (ICTIR \u201920) (Virtual Event, Norway). ACM, New York, 69\u201376. DOI:10.1145\/3409256.3409830"},{"key":"e_1_3_4_68_2","doi-asserted-by":"publisher","DOI":"10.1055\/s-0043-1767684"},{"key":"e_1_3_4_69_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00779-021-01604-6"},{"key":"e_1_3_4_70_2","first-page":"1305","volume-title":"Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM \u201920).","author":"Roitero Kevin","year":"2020","unstructured":"Kevin Roitero, Michael Soprano, Beatrice Portelli, Damiano Spina, Vincenzo Della Mea, Giuseppe Serra, Stefano Mizzaro, and Gianluca Demartini. 2020. The COVID-19 infodemic: Can the crowd judge recent misinformation objectively?. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (CIKM \u201920). (Virtual Event, Ireland) ACM, New York, 1305\u20131314. DOI:10.1145\/3340531.3412048"},{"key":"e_1_3_4_71_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-9863-7_223"},{"key":"e_1_3_4_72_2","first-page":"1621","volume-title":"Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI \u201915)","author":"Salehi Niloufar","year":"2015","unstructured":"Niloufar Salehi, Lilly C. Irani, Michael S. Bernstein, Ali Alkhatib, Eva Ogbe, Kristy Milland, and Clickhappier. 2015. We are dynamo: Overcoming stalling and friction in collective action for crowd workers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI \u201915) (Seoul, Republic of Korea). ACM, New York. , 1621\u20131630. DOI:10.1145\/2702123.2702508"},{"key":"e_1_3_4_73_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.waojou.2022.100718"},{"key":"e_1_3_4_74_2","doi-asserted-by":"publisher","DOI":"10.1177\/2167702612469015"},{"key":"e_1_3_4_75_2","first-page":"1605","volume-title":"Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM \u201922)","author":"Soprano Michael","year":"2022","unstructured":"Michael Soprano, Kevin Roitero, Francesco Bombassei De Bona, and Stefano Mizzaro. 2022. Crowd_Frame: A simple and complete framework to deploy complex crowdsourcing tasks off-the-shelf. In Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM \u201922) (Virtual Event, AZ, USA). ACM, New York, 1605\u20131608. DOI:10.1145\/3488560.3502182"},{"key":"e_1_3_4_76_2","doi-asserted-by":"publisher","DOI":"10.1002\/jeab.445"},{"key":"e_1_3_4_77_2","doi-asserted-by":"publisher","DOI":"10.1037\/pha0000235"},{"key":"e_1_3_4_78_2","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1145\/3450613.3456817","volume-title":"Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization (UMAP \u201921)","author":"Tolmeijer Suzanne","year":"2021","unstructured":"Suzanne Tolmeijer, Ujwal Gadiraju, Ramya Ghantasala, Akshit Gupta, and Abraham Bernstein. 2021. Second chance for a first impression? Trust development in intelligent system interaction. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization (UMAP \u201921) (Utrecht, Netherlands).. ACM, New York, 77\u201387. DOI:10.1145\/3450613.3456817"},{"key":"e_1_3_4_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3476060"},{"key":"e_1_3_4_80_2","volume-title":"Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI \u201922)","author":"Varanasi Rama Adithya","year":"2022","unstructured":"Rama Adithya Varanasi, Divya Siddarth, Vivek Seshadri, Kalika Bali, and Aditya Vashistha. 2022. Feeling proud, feeling embarrassed: Experiences of low-income women with crowd work. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI \u201922) (New Orleans, LA, USA). ACM, New York, Article 298, 18 pages. DOI:10.1145\/3491102.3501834"},{"key":"e_1_3_4_81_2","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2020.1724010"},{"key":"e_1_3_4_82_2","first-page":"197","volume-title":"Proceedings of the 7th AAAI Conference on Human Computation and Crowdsourcing","author":"Whiting Mark E.","year":"2019","unstructured":"Mark E. Whiting, Grant Hugh, and Michael S. Bernstein. 2019. Fair work: Crowd work minimum wage with one line of code. In Proceedings of the 7th AAAI Conference on Human Computation and Crowdsourcing, Edith Law and Jennifer Wortman Vaughan (Eds.). AAAI Press, Washington, DC, 197\u2013206. DOI:10.1609\/hcomp.v7i1.5283"},{"key":"e_1_3_4_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359126"},{"key":"e_1_3_4_84_2","first-page":"206","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"5","author":"Wu Meng-Han","year":"2017","unstructured":"Meng-Han Wu and Alexander Quinn. 2017. Confusing the crowd: Task instruction quality on Amazon mechanical turk. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 5. AAAI, USA, 206\u2013215. DOI:10.1609\/hcomp.v5i1.13317"}],"container-title":["ACM Transactions on Social Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3674884","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3674884","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:05:56Z","timestamp":1750291556000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3674884"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,24]]},"references-count":83,"journal-issue":{"issue":"1-4","published-print":{"date-parts":[[2024,12,31]]}},"alternative-id":["10.1145\/3674884"],"URL":"https:\/\/doi.org\/10.1145\/3674884","relation":{},"ISSN":["2469-7818","2469-7826"],"issn-type":[{"type":"print","value":"2469-7818"},{"type":"electronic","value":"2469-7826"}],"subject":[],"published":{"date-parts":[[2024,9,24]]},"assertion":[{"value":"2023-09-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-10","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-09-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}