{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T14:48:09Z","timestamp":1782830889640,"version":"3.54.5"},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2022,11,7]],"date-time":"2022-11-07T00:00:00Z","timestamp":1667779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["2009003"],"award-info":[{"award-number":["2009003"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,11,7]]},"abstract":"<jats:p>Misinformation runs rampant on social media and has been tied to adverse health behaviors such as vaccine hesitancy. Crowdsourcing can be a means to detect and impede the spread of misinformation online. However, past studies have not deeply examined the individual characteristics - such as cognitive factors and biases - that predict crowdworker accuracy at identifying misinformation. In our study (n = 265), Amazon Mechanical Turk (MTurk) workers and university students assessed the truthfulness and sentiment of COVID-19 related tweets as well as answered several surveys on personal characteristics. Results support the viability of crowdsourcing for assessing misinformation and content stance (i.e., sentiment) related to ongoing and politically-charged topics like the COVID-19 pandemic, however, alignment with experts depends on who is in the crowd. Specifically, we find that respondents with high Cognitive Reflection Test (CRT) scores, conscientiousness, and trust in medical scientists are more aligned with experts while respondents with high Need for Cognitive Closure (NFCC) and those who lean politically conservative are less aligned with experts. We see differences between recruitment platforms as well, as our data shows university students are on average more aligned with experts than MTurk workers, most likely due to overall differences in participant characteristics on each platform. Results offer transparency into how crowd composition affects misinformation and stance assessment and have implications on future crowd recruitment and filtering practices.<\/jats:p>","DOI":"10.1145\/3555611","type":"journal-article","created":{"date-parts":[[2022,11,11]],"date-time":"2022-11-11T22:58:54Z","timestamp":1668207534000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["Who's in the Crowd Matters: Cognitive Factors and Beliefs Predict Misinformation Assessment Accuracy"],"prefix":"10.1145","volume":"6","author":[{"given":"Robert A.","family":"Kaufman","sequence":"first","affiliation":[{"name":"University of California, San Diego, La Jolla, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael Robert","family":"Haupt","sequence":"additional","affiliation":[{"name":"University of California, San Diego, La Jolla, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steven P.","family":"Dow","sequence":"additional","affiliation":[{"name":"University of California, San Diego, La Jolla, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,11,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.31620\/JCCC.12.20\/06"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3297722"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0118093"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415164"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1618923114"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313721"},{"key":"e_1_2_1_8_1","volume-title":"Dominance analysis: a new approach to the problem of relative importance of predictors in multiple regression. Psychological bulletin 114, 3","author":"Budescu David V","year":"1993","unstructured":"David V Budescu . 1993. Dominance analysis: a new approach to the problem of relative importance of predictors in multiple regression. Psychological bulletin 114, 3 ( 1993 ), 542. David V Budescu. 1993. Dominance analysis: a new approach to the problem of relative importance of predictors in multiple regression. Psychological bulletin 114, 3 (1993), 542."},{"key":"e_1_2_1_9_1","volume-title":"Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality data?","author":"Buhrmester Michael","year":"2016","unstructured":"Michael Buhrmester , Tracy Kwang , and Samuel D Gosling . 2016. Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality data? ( 2016 ). Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. 2016. Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality data? (2016)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1177\/1948550619875149"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2021.106716"},{"key":"e_1_2_1_12_1","volume-title":"International Conference on Artificial Intelligence and Statistics. PMLR, 3457--3465","author":"Covert Ian","year":"2021","unstructured":"Ian Covert and Su-In Lee . 2021 . Improving KernelSHAP: Practical Shapley value estimation using linear regression . In International Conference on Artificial Intelligence and Statistics. PMLR, 3457--3465 . Ian Covert and Su-In Lee. 2021. Improving KernelSHAP: Practical Shapley value estimation using linear regression. In International Conference on Artificial Intelligence and Statistics. PMLR, 3457--3465."},{"key":"e_1_2_1_13_1","volume-title":"Carolina Robledo-Vega, Maria Fernanda P\u00e9rez-Zepeda, and Elena Soto-Vega.","author":"Cuan-Baltazar Jose Yunam","year":"2020","unstructured":"Jose Yunam Cuan-Baltazar , Maria Jos\u00e9 Mu\u00f1oz-Perez , Carolina Robledo-Vega, Maria Fernanda P\u00e9rez-Zepeda, and Elena Soto-Vega. 2020 . Misinformation of COVID-19 on the internet: infodemiology study. JMIR public health and surveillance 6, 2 (2020), e18444. Jose Yunam Cuan-Baltazar, Maria Jos\u00e9 Mu\u00f1oz-Perez, Carolina Robledo-Vega, Maria Fernanda P\u00e9rez-Zepeda, and Elena Soto-Vega. 2020. Misinformation of COVID-19 on the internet: infodemiology study. JMIR public health and surveillance 6, 2 (2020), e18444."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2011.61"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1177\/0146167219853844"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-923X.12892"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v9i1.18939"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-28577-7_25"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173950"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1257\/089533005775196732"},{"key":"e_1_2_1_21_1","volume-title":"Trust and mistrust in Americans' views of scientific experts","author":"Funk Cary","year":"2019","unstructured":"Cary Funk , Meg Hefferon , Brian Kennedy , and Courtney Johnson . 2019. Trust and mistrust in Americans' views of scientific experts . Pew Research Center 2 ( 2019 ). Cary Funk, Meg Hefferon, Brian Kennedy, and Courtney Johnson. 2019. Trust and mistrust in Americans' views of scientific experts. Pew Research Center 2 (2019)."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2896377.2901462"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa2012410"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1002\/bdm.1753"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1177\/1075547020950735"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1177\/20539517211013843"},{"key":"e_1_2_1_28_1","volume-title":"It's a conspiracy: Covid-19 conspiracies link to psychopathy, Machiavellianism and collective narcissism. Personality and individual differences 171","author":"Hughes Sara","year":"2021","unstructured":"Sara Hughes and Laura Machan . 2021. It's a conspiracy: Covid-19 conspiracies link to psychopathy, Machiavellianism and collective narcissism. Personality and individual differences 171 ( 2021 ), 110559. Sara Hughes and Laura Machan. 2021. It's a conspiracy: Covid-19 conspiracies link to psychopathy, Machiavellianism and collective narcissism. Personality and individual differences 171 (2021), 110559."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10888-006-9036-6"},{"key":"e_1_2_1_30_1","volume-title":"echnical Report].","author":"John Oliver P","year":"1991","unstructured":"Oliver P John , EM Donahue , and R_ L Kentle . 1991. The big five inventory: Versions 4a and 54 [ T echnical Report]. Berkeley : University of California, Institute of Personality and Social Research ( 1991 ). Oliver P John, EM Donahue, and R_L Kentle. 1991. The big five inventory: Versions 4a and 54 [Technical Report]. Berkeley: University of California, Institute of Personality and Social Research (1991)."},{"key":"e_1_2_1_31_1","unstructured":"Oliver P John Laura P Naumann and Christopher J Soto. 2008. Paradigm shift to the integrative Big Five trait taxonomy: History measurement and conceptual issues. (2008).  Oliver P John Laura P Naumann and Christopher J Soto. 2008. Paradigm shift to the integrative Big Five trait taxonomy: History measurement and conceptual issues. (2008)."},{"key":"e_1_2_1_32_1","volume-title":"Pylons ablaze: Examining the role of 5G COVID-19 conspiracy beliefs and support for violence. British journal of social psychology 59, 3","author":"Jolley Daniel","year":"2020","unstructured":"Daniel Jolley and Jenny L Paterson . 2020. Pylons ablaze: Examining the role of 5G COVID-19 conspiracy beliefs and support for violence. British journal of social psychology 59, 3 ( 2020 ), 628--640. Daniel Jolley and Jenny L Paterson. 2020. Pylons ablaze: Examining the role of 5G COVID-19 conspiracy beliefs and support for violence. British journal of social psychology 59, 3 (2020), 628--640."},{"key":"e_1_2_1_33_1","volume-title":"Web 2.0, and the postmodern paradigm--An overview of tactics and tropes used online by the anti-vaccination movement. Vaccine 30, 25","author":"Kata Anna","year":"2012","unstructured":"Anna Kata . 2012. Anti-vaccine activists , Web 2.0, and the postmodern paradigm--An overview of tactics and tropes used online by the anti-vaccination movement. Vaccine 30, 25 ( 2012 ), 3778--3789. Anna Kata. 2012. Anti-vaccine activists, Web 2.0, and the postmodern paradigm--An overview of tactics and tropes used online by the anti-vaccination movement. Vaccine 30, 25 (2012), 3778--3789."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291009"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/SocialInformatics.2012.33"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531677"},{"key":"e_1_2_1_37_1","first-page":"207","article-title":"Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias","volume":"12036","author":"Barbera David La","year":"2020","unstructured":"David La Barbera , Kevin Roitero , Gianluca Demartini , Stefano Mizzaro , and Damiano Spina . 2020 . Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias . Advances in Information Retrieval 12036 (2020), 207 . David La Barbera, Kevin Roitero, Gianluca Demartini, Stefano Mizzaro, and Damiano Spina. 2020. Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias. Advances in Information Retrieval 12036 (2020), 207.","journal-title":"Advances in Information Retrieval"},{"key":"e_1_2_1_38_1","volume-title":"Of pandemics, politics, and personality: The role of conscientiousness and political ideology in the sharing of fake news. Journal of Experimental Psychology: General","author":"Asher Lawson M","year":"2021","unstructured":"M Asher Lawson and Hemant Kakkar . 2021. Of pandemics, politics, and personality: The role of conscientiousness and political ideology in the sharing of fake news. Journal of Experimental Psychology: General ( 2021 ). M Asher Lawson and Hemant Kakkar. 2021. Of pandemics, politics, and personality: The role of conscientiousness and political ideology in the sharing of fake news. Journal of Experimental Psychology: General (2021)."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1002\/asmb.446"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/S2589-7500(20)30318-6"},{"key":"e_1_2_1_41_1","volume-title":"CIKM Workshops.","author":"Maddalena Eddy","year":"2018","unstructured":"Eddy Maddalena , Davide Ceolin , and Stefano Mizzaro . 2018 . Multidimensional News Quality: A Comparison of Crowdsourcing and Nichesourcing .. In CIKM Workshops. Eddy Maddalena, Davide Ceolin, and Stefano Mizzaro. 2018. Multidimensional News Quality: A Comparison of Crowdsourcing and Nichesourcing.. In CIKM Workshops."},{"key":"e_1_2_1_42_1","first-page":"84","article-title":"Conspiracy theories and flat-earth videos on YouTube","volume":"8","author":"Mohammed Shaheed N","year":"2019","unstructured":"Shaheed N Mohammed . 2019 . Conspiracy theories and flat-earth videos on YouTube . The Journal of Social Media in Society 8 , 2 (2019), 84 -- 102 . Shaheed N Mohammed. 2019. Conspiracy theories and flat-earth videos on YouTube. The Journal of Social Media in Society 8, 2 (2019), 84--102.","journal-title":"The Journal of Social Media in Society"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1111\/hir.12311"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR48806.2021.9412511"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1806781116"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1521\/soco.2014.32.4.337"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/CSCWD.2019.8791903"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jrp.2006.02.001"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.paid.2010.09.004"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401112"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412048"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.socscimed.2021.114480"},{"key":"e_1_2_1_53_1","volume-title":"Gabriel Recchia, Anne Marthe Van Der Bles, and Sander Van Der Linden.","author":"Roozenbeek Jon","year":"2020","unstructured":"Jon Roozenbeek , Claudia R Schneider , Sarah Dryhurst , John Kerr , Alexandra LJ Freeman , Gabriel Recchia, Anne Marthe Van Der Bles, and Sander Van Der Linden. 2020 . Susceptibility to misinformation about COVID-19 around the world. Royal Society open science 7, 10 (2020), 201199. Jon Roozenbeek, Claudia R Schneider, Sarah Dryhurst, John Kerr, Alexandra LJ Freeman, Gabriel Recchia, Anne Marthe Van Der Bles, and Sander Van Der Linden. 2020. Susceptibility to misinformation about COVID-19 around the world. Royal Society open science 7, 10 (2020), 201199."},{"key":"e_1_2_1_54_1","volume-title":"The Twitter pandemic: The critical role of Twitter in the dissemination of medical information and misinformation during the COVID-19 pandemic. Canadian journal of emergency medicine 22, 4","author":"Rosenberg Hans","year":"2020","unstructured":"Hans Rosenberg , Shahbaz Syed , and Salim Rezaie . 2020. The Twitter pandemic: The critical role of Twitter in the dissemination of medical information and misinformation during the COVID-19 pandemic. Canadian journal of emergency medicine 22, 4 ( 2020 ), 418--421. Hans Rosenberg, Shahbaz Syed, and Salim Rezaie. 2020. The Twitter pandemic: The critical role of Twitter in the dissemination of medical information and misinformation during the COVID-19 pandemic. Canadian journal of emergency medicine 22, 4 (2020), 418--421."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2021.114832"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2021.102710"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053168021993979"},{"key":"e_1_2_1_58_1","doi-asserted-by":"crossref","unstructured":"Anni Sternisko Aleksandra Cichocka Aleksandra Cislak and Jay J Van Bavel. 2020. Collective narcissism predicts the belief and dissemination of conspiracy theories during the COVID-19 pandemic. (2020).  Anni Sternisko Aleksandra Cichocka Aleksandra Cislak and Jay J Van Bavel. 2020. Collective narcissism predicts the belief and dissemination of conspiracy theories during the COVID-19 pandemic. (2020).","DOI":"10.31234\/osf.io\/4c6av"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1017\/bca.2020.12"},{"key":"e_1_2_1_60_1","volume-title":"Why education predicts decreased belief in conspiracy theories. Applied cognitive psychology 31, 1","author":"van Prooijen Jan-Willem","year":"2017","unstructured":"Jan-Willem van Prooijen . 2017. Why education predicts decreased belief in conspiracy theories. Applied cognitive psychology 31, 1 ( 2017 ), 50--58. Jan-Willem van Prooijen. 2017. Why education predicts decreased belief in conspiracy theories. Applied cognitive psychology 31, 1 (2017), 50--58."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359209"},{"key":"e_1_2_1_62_1","volume-title":"liar pants on fire\": A new benchmark dataset for fake news detection. arXiv preprint arXiv:1705.00648","author":"Wang William Yang","year":"2017","unstructured":"William Yang Wang . 2017. \" liar , liar pants on fire\": A new benchmark dataset for fake news detection. arXiv preprint arXiv:1705.00648 ( 2017 ). William Yang Wang. 2017. \" liar, liar pants on fire\": A new benchmark dataset for fake news detection. arXiv preprint arXiv:1705.00648 (2017)."},{"key":"e_1_2_1_63_1","volume-title":"Systematic literature review on the spread of health-related misinformation on social media. Social science & medicine 240","author":"Wang Yuxi","year":"2019","unstructured":"Yuxi Wang , Martin McKee , Aleksandra Torbica , and David Stuckler . 2019. Systematic literature review on the spread of health-related misinformation on social media. Social science & medicine 240 ( 2019 ), 112552. Yuxi Wang, Martin McKee, Aleksandra Torbica, and David Stuckler. 2019. Systematic literature review on the spread of health-related misinformation on social media. Social science & medicine 240 (2019), 112552."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.67.6.1049"},{"key":"e_1_2_1_65_1","unstructured":"World Health Organization (WHO). 2020. Managing the COVID-19 infodemic: Promoting healthy behaviours and mitigating the harm from misinformation and disinformation. https:\/\/www.who.int\/news\/item\/23-09--2020-managingthe- covid-19-infodemic-promoting-healthy-behaviours-and-mitigating-the-harm-from-misinformation-anddisinformation  World Health Organization (WHO). 2020. Managing the COVID-19 infodemic: Promoting healthy behaviours and mitigating the harm from misinformation and disinformation. https:\/\/www.who.int\/news\/item\/23-09--2020-managingthe- covid-19-infodemic-promoting-healthy-behaviours-and-mitigating-the-harm-from-misinformation-anddisinformation"},{"key":"e_1_2_1_66_1","volume-title":"Individual differences in false memory from misinformation: Personality characteristics and their interactions with cognitive abilities. Personality and Individual differences 48, 8","author":"Zhu Bi","year":"2010","unstructured":"Bi Zhu , Chuansheng Chen , Elizabeth F Loftus , Chongde Lin , Qinghua He , Chunhui Chen , He Li , Robert K Moyzis , Jared Lessard , and Qi Dong . 2010. Individual differences in false memory from misinformation: Personality characteristics and their interactions with cognitive abilities. Personality and Individual differences 48, 8 ( 2010 ), 889--894. Bi Zhu, Chuansheng Chen, Elizabeth F Loftus, Chongde Lin, Qinghua He, Chunhui Chen, He Li, Robert K Moyzis, Jared Lessard, and Qi Dong. 2010. Individual differences in false memory from misinformation: Personality characteristics and their interactions with cognitive abilities. Personality and Individual differences 48, 8 (2010), 889--894."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13278-014-0163-y"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555611","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3555611","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3555611","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:03Z","timestamp":1750182543000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555611"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,7]]},"references-count":65,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2022,11,7]]}},"alternative-id":["10.1145\/3555611"],"URL":"https:\/\/doi.org\/10.1145\/3555611","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,7]]},"assertion":[{"value":"2022-11-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}