{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T05:12:03Z","timestamp":1778044323415,"version":"3.51.4"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2020,10,14]],"date-time":"2020-10-14T00:00:00Z","timestamp":1602633600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["IIS-1755547"],"award-info":[{"award-number":["IIS-1755547"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2020,10,14]]},"abstract":"<jats:p>Misinformation about critical issues such as climate change and vaccine safety is oftentimes amplified on online social and search platforms. The crowdsourcing of content credibility assessment by laypeople has been proposed as one strategy to combat misinformation by attempting to replicate the assessments of experts at scale. In this work, we investigate news credibility assessments by crowds versus experts to understand when and how ratings between them differ. We gather a dataset of over 4,000 credibility assessments taken from 2 crowd groups---journalism students and Upwork workers---as well as 2 expert groups---journalists and scientists---on a varied set of 50 news articles related to climate science, a topic with widespread disconnect between public opinion and expert consensus. Examining the ratings, we find differences in performance due to the makeup of the crowd, such as rater demographics and political leaning, as well as the scope of the tasks that the crowd is assigned to rate, such as the genre of the article and partisanship of the publication. Finally, we find differences between expert assessments due to differing expert criteria that journalism versus science experts use---differences that may contribute to crowd discrepancies, but that also suggest a way to reduce the gap by designing crowd tasks tailored to specific expert criteria. From these findings, we outline future research directions to better design crowd processes that are tailored to specific crowds and types of content.<\/jats:p>","DOI":"10.1145\/3415164","type":"journal-article","created":{"date-parts":[[2020,10,15]],"date-time":"2020-10-15T22:28:12Z","timestamp":1602800892000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":63,"title":["Investigating Differences in Crowdsourced News Credibility Assessment"],"prefix":"10.1145","volume":"4","author":[{"given":"Md Momen","family":"Bhuiyan","sequence":"first","affiliation":[{"name":"Virginia Tech, Blacksburg, VA, USA"}]},{"given":"Amy X.","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]},{"given":"Connie Moon","family":"Sehat","sequence":"additional","affiliation":[{"name":"Hacks\/Hackers, Atlanta, GA, USA"}]},{"given":"Tanushree","family":"Mitra","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,10,15]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1003187107"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i1.2564"},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","unstructured":"Mahmoudreza Babaei Abhijnan Chakraborty Juhi Kulshrestha Elissa M Redmiles Meeyoung Cha and Krishna P Gummadi. 2019. Analyzing Biases in Perception of Truth in News Stories and Their Implications for Fact Checking.. In FAT. 139.  Mahmoudreza Babaei Abhijnan Chakraborty Juhi Kulshrestha Elissa M Redmiles Meeyoung Cha and Krishna P Gummadi. 2019. Analyzing Biases in Perception of Truth in News Stories and Their Implications for Fact Checking.. In FAT. 139.","DOI":"10.1145\/3287560.3287581"},{"key":"e_1_2_1_4_1","unstructured":"Mevan Babakar. 2018. Crowdsourced Factchecking.  Mevan Babakar. 2018. Crowdsourced Factchecking."},{"key":"e_1_2_1_5_1","volume-title":"Combining significance levels. The handbook of research synthesis","author":"Becker Betsy Jane","year":"1994","unstructured":"Betsy Jane Becker . 1994. Combining significance levels. The handbook of research synthesis ( 1994 ), 215--230. Betsy Jane Becker. 1994. Combining significance levels. The handbook of research synthesis (1994), 215--230."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1817195116"},{"key":"e_1_2_1_7_1","volume-title":"The problem with science journalism: we've forgotten that reality matters most. The Guardian (Dec","author":"Borel Brooke","year":"2015","unstructured":"Brooke Borel . 2015. The problem with science journalism: we've forgotten that reality matters most. The Guardian (Dec 2015 ). https:\/\/www.theguardian.com\/media\/2015\/dec\/30\/problem-with-science-journalism-2015-reality-kevin-folta Brooke Borel. 2015. The problem with science journalism: we've forgotten that reality matters most. The Guardian (Dec 2015). https:\/\/www.theguardian.com\/media\/2015\/dec\/30\/problem-with-science-journalism-2015-reality-kevin-folta"},{"key":"e_1_2_1_8_1","volume-title":"Influence of fake news in Twitter during the 2016 US presidential election. Nature communications","author":"Bovet Alexandre","year":"2019","unstructured":"Alexandre Bovet and Hern\u00e1n A Makse . 2019. Influence of fake news in Twitter during the 2016 US presidential election. Nature communications , Vol. 10 , 1 ( 2019 ), 7. Alexandre Bovet and Hern\u00e1n A Makse. 2019. Influence of fake news in Twitter during the 2016 US presidential election. Nature communications, Vol. 10, 1 (2019), 7."},{"key":"e_1_2_1_9_1","volume-title":"World","volume":"3","author":"Bujang Mohamad Adam","year":"2016","unstructured":"Mohamad Adam Bujang and Nurakmal Baharum . 2016 . Sample size guideline for correlation analysis . World , Vol. 3 , 1 (2016). Mohamad Adam Bujang and Nurakmal Baharum. 2016. Sample size guideline for correlation analysis. World, Vol. 3, 1 (2016)."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/SmartCloud.2017.40"},{"key":"e_1_2_1_11_1","unstructured":"Davide Ceolin. 2019. Conference Presentation: On the Quality of Crowdsourced Information Quality Assessments. https:\/\/drive.google.com\/a\/hackshackers.com\/file\/d\/1AJmFmRqEhdhSIZLwhXT_1bzStXfV-hVf\/view?usp=drive_open&usp=embed_facebook  Davide Ceolin. 2019. Conference Presentation: On the Quality of Crowdsourced Information Quality Assessments. https:\/\/drive.google.com\/a\/hackshackers.com\/file\/d\/1AJmFmRqEhdhSIZLwhXT_1bzStXfV-hVf\/view?usp=drive_open&usp=embed_facebook"},{"key":"e_1_2_1_12_1","volume-title":"Mass media and interpersonal channels: Competitive, convergent, or complementary. Inter\/media: Interpersonal communication in a media world","author":"Chaffee Steven H","year":"1982","unstructured":"Steven H Chaffee . 1982. Mass media and interpersonal channels: Competitive, convergent, or complementary. Inter\/media: Interpersonal communication in a media world , Vol. 57 ( 1982 ), 77. Steven H Chaffee. 1982. Mass media and interpersonal channels: Competitive, convergent, or complementary. Inter\/media: Interpersonal communication in a media world, Vol. 57 (1982), 77."},{"key":"e_1_2_1_13_1","volume-title":"Social influence: the ontario symposium","author":"Chaiken Shelly","unstructured":"Shelly Chaiken . 1987. The heuristic model of persuasion . In Social influence: the ontario symposium , Vol. 5 . Hillsdale, NJ : Lawrence Erlbaum , 3--39. Shelly Chaiken. 1987. The heuristic model of persuasion. In Social influence: the ontario symposium, Vol. 5. Hillsdale, NJ: Lawrence Erlbaum, 3--39."},{"key":"e_1_2_1_14_1","volume-title":"The mahalanobis distance. Chemometrics and intelligent laboratory systems","author":"Maesschalck Roy De","year":"2000","unstructured":"Roy De Maesschalck , Delphine Jouan-Rimbaud , and D\u00e9sir\u00e9 L Massart . 2000. The mahalanobis distance. Chemometrics and intelligent laboratory systems , Vol. 50 , 1 ( 2000 ), 1--18. Roy De Maesschalck, Delphine Jouan-Rimbaud, and D\u00e9sir\u00e9 L Massart. 2000. The mahalanobis distance. Chemometrics and intelligent laboratory systems, Vol. 50, 1 (2000), 1--18."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1080\/014492998119526"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376232"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.psych.59.103006.093629"},{"key":"e_1_2_1_18_1","unstructured":"Facebook. 2020. Fact-Checking on Facebook: What Publishers Should Know. https:\/\/www.facebook.com\/help\/publisher\/182222309230722. (Accessed on 01\/14\/2020).  Facebook. 2020. Fact-Checking on Facebook: What Publishers Should Know. https:\/\/www.facebook.com\/help\/publisher\/182222309230722. (Accessed on 01\/14\/2020)."},{"key":"e_1_2_1_19_1","volume-title":"FactCheckEU - 19 European media outlets are fact-checking the","author":"EU.","year":"2019","unstructured":"FactCheck EU. [n.d.]. FactCheckEU - 19 European media outlets are fact-checking the May 2019 European elections. https:\/\/www.factcheckeu.info\/en\/. (Accessed on 05\/14\/2020). FactCheckEU. [n.d.]. FactCheckEU - 19 European media outlets are fact-checking the May 2019 European elections. https:\/\/www.factcheckeu.info\/en\/. (Accessed on 05\/14\/2020)."},{"key":"e_1_2_1_20_1","volume-title":"Digital media and youth: Unparalleled opportunity and unprecedented responsibility. Digital media, youth, and credibility","author":"Flanagin Andrew J","year":"2008","unstructured":"Andrew J Flanagin and Miriam J Metzger . 2008. Digital media and youth: Unparalleled opportunity and unprecedented responsibility. Digital media, youth, and credibility ( 2008 ), 5--27. Andrew J Flanagin and Miriam J Metzger. 2008. Digital media and youth: Unparalleled opportunity and unprecedented responsibility. Digital media, youth, and credibility (2008), 5--27."},{"key":"e_1_2_1_21_1","unstructured":"Fabrice Florin. 2010. Crowdsourced Fact-Checking? What We Learned from Truthsquad. Mediashift (2010).  Fabrice Florin. 2010. Crowdsourced Fact-Checking? What We Learned from Truthsquad. Mediashift (2010)."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/765891.765951"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/302979.303001"},{"key":"e_1_2_1_24_1","volume-title":"Americans and the news media: What they do--and don't--understand about each other. The Media Insight Project","author":"American Press Institute & The AP-NORC Center for Public Affairs Research. 2018.","year":"2018","unstructured":"American Press Institute & The AP-NORC Center for Public Affairs Research. 2018. Americans and the news media: What they do--and don't--understand about each other. The Media Insight Project ( 2018 ). American Press Institute & The AP-NORC Center for Public Affairs Research. 2018. Americans and the news media: What they do--and don't--understand about each other. The Media Insight Project (2018)."},{"key":"e_1_2_1_25_1","volume-title":"Trust and Mistrust in Americans? Views of Scientific Experts","author":"Funk Cary","year":"2019","unstructured":"Cary Funk , Meg Hefferon , Brian Kennedy , and Courtney Johnson . 2019. Trust and Mistrust in Americans? Views of Scientific Experts . Pew Research Center . https:\/\/www. pewresearch. org\/science\/ 2019 \/08\/02\/trust-and-mistrust-inamericans-views-of-scientific-experts (2019). Cary Funk, Meg Hefferon, Brian Kennedy, and Courtney Johnson. 2019. Trust and Mistrust in Americans? Views of Scientific Experts. Pew Research Center. https:\/\/www. pewresearch. org\/science\/2019\/08\/02\/trust-and-mistrust-inamericans-views-of-scientific-experts (2019)."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1177\/107769908606300301"},{"key":"e_1_2_1_27_1","volume-title":"What Does a Sports Desk Do When Sports Are on Hold? The New York Times (Apr","author":"Grillo Emma","year":"2020","unstructured":"Emma Grillo . 2020. What Does a Sports Desk Do When Sports Are on Hold? The New York Times (Apr 2020 ). https:\/\/www.nytimes.com\/2020\/04\/05\/reader-center\/coronavirus-sports-reporting.html Emma Grillo. 2020. What Does a Sports Desk Do When Sports Are on Hold? The New York Times (Apr 2020). https:\/\/www.nytimes.com\/2020\/04\/05\/reader-center\/coronavirus-sports-reporting.html"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aau2706"},{"key":"e_1_2_1_29_1","volume-title":"Javier A Suarez Rivas, and Md Khadimul Islam","author":"Hassan Naeemul","year":"2017","unstructured":"Naeemul Hassan , Mohammad Yousuf , Mahfuzul Haque , Javier A Suarez Rivas, and Md Khadimul Islam . 2017 . Towards A Sustainable Model for Fact-checking Platforms: Examining the Roles of Automation, Crowds and Professionals . https:\/\/doi.org\/10.1145\/3308560.3316734 10.1145\/3308560.3316734 Naeemul Hassan, Mohammad Yousuf, Mahfuzul Haque, Javier A Suarez Rivas, and Md Khadimul Islam. 2017. Towards A Sustainable Model for Fact-checking Platforms: Examining the Roles of Automation, Crowds and Professionals. https:\/\/doi.org\/10.1145\/3308560.3316734"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2007.10.001"},{"key":"e_1_2_1_31_1","volume-title":"Horne and Sibel Adali","author":"Benjamin","year":"2017","unstructured":"Benjamin D. Horne and Sibel Adali . 2017 . This Just In: Fake News Packs a Lot in Title, Uses Simpler, Repetitive Content in Text Body, More Similar to Satire than Real News . (2017), 759--766. arxiv: 1703.09398 http:\/\/arxiv.org\/abs\/1703.09398 Benjamin D. Horne and Sibel Adali. 2017. This Just In: Fake News Packs a Lot in Title, Uses Simpler, Repetitive Content in Text Body, More Similar to Satire than Real News. (2017), 759--766. arxiv: 1703.09398 http:\/\/arxiv.org\/abs\/1703.09398"},{"key":"e_1_2_1_32_1","volume-title":"Irving Lester Janis, and Harold H Kelley","author":"Hovland Carl Iver","year":"1953","unstructured":"Carl Iver Hovland , Irving Lester Janis, and Harold H Kelley . 1953 . Communication and persuasion. (1953). Carl Iver Hovland, Irving Lester Janis, and Harold H Kelley. 1953. Communication and persuasion. (1953)."},{"key":"e_1_2_1_33_1","unstructured":"Rebecca Iannucci and Bill Adair. 2017. Reporters? Lab Study Results: Effective News Labeling and Media Literacy.  Rebecca Iannucci and Bill Adair. 2017. Reporters? Lab Study Results: Effective News Labeling and Media Literacy."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1093\/eurpub\/ckz004"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1017\/pan.2019.11"},{"key":"e_1_2_1_36_1","volume-title":"Public trust or mistrust? Perceptions of media credibility in the information age. Mass communication & society","author":"Kiousis Spiro","year":"2001","unstructured":"Spiro Kiousis . 2001. Public trust or mistrust? Perceptions of media credibility in the information age. Mass communication & society , Vol. 4 , 4 ( 2001 ), 381--403. Spiro Kiousis. 2001. Public trust or mistrust? Perceptions of media credibility in the information age. Mass communication & society, Vol. 4, 4 (2001), 381--403."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"key":"e_1_2_1_38_1","first-page":"4","article-title":"Science Journalism Faces Perilous Times","volume":"18","author":"Lucibella Michael","year":"2009","unstructured":"Michael Lucibella . 2009 . Science Journalism Faces Perilous Times . American Physical Society (APS) News , Vol. 18 , 4 (Apr 2009). http:\/\/www.aps.org\/publications\/apsnews\/200904\/journalism.cfm Michael Lucibella. 2009. Science Journalism Faces Perilous Times. American Physical Society (APS) News, Vol. 18, 4 (Apr 2009). http:\/\/www.aps.org\/publications\/apsnews\/200904\/journalism.cfm","journal-title":"American Physical Society (APS) News"},{"key":"e_1_2_1_39_1","volume-title":"The Wisdom of Select Crowds. Journal of personality and social psychology","author":"Mannes Albert","year":"2014","unstructured":"Albert Mannes , Jack Soll , and Richard Larrick . 2014. The Wisdom of Select Crowds. Journal of personality and social psychology ( 2014 ). Albert Mannes, Jack Soll, and Richard Larrick. 2014. The Wisdom of Select Crowds. Journal of personality and social psychology (2014)."},{"key":"e_1_2_1_40_1","volume-title":"Soll","author":"Mannes Albert E.","year":"2012","unstructured":"Albert E. Mannes , Richard P. Larrick , and Jack B . Soll . 2012 . The social psychology of the wisdom of crowds. Albert E. Mannes, Richard P. Larrick, and Jack B. Soll. 2012. The social psychology of the wisdom of crowds."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1088\/1748-9326\/8\/4\/044029"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/1315930.1315940"},{"key":"e_1_2_1_43_1","volume-title":"Cognitive dissonance or credibility? A comparison of two theoretical explanations for selective exposure to partisan news. Communication Research","author":"Metzger Miriam J","year":"2015","unstructured":"Miriam J Metzger , Ethan H Hartsell , and Andrew J Flanagin . 2015. Cognitive dissonance or credibility? A comparison of two theoretical explanations for selective exposure to partisan news. Communication Research ( 2015 ), 0093650215613136. Miriam J Metzger, Ethan H Hartsell, and Andrew J Flanagin. 2015. Cognitive dissonance or credibility? A comparison of two theoretical explanations for selective exposure to partisan news. Communication Research (2015), 0093650215613136."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1177\/107769908806500301"},{"key":"e_1_2_1_45_1","unstructured":"Amy Mitchell Jeffrey Gottfried Michael Barthel and Nami Sumida. 2018. Can Americans Tell Factual From Opinion Statements in the News?  Amy Mitchell Jeffrey Gottfried Michael Barthel and Nami Sumida. 2018. Can Americans Tell Factual From Opinion Statements in the News?"},{"key":"e_1_2_1_46_1","volume-title":"Proc. ICWSM'15","author":"Mitra Tanushree","year":"2015","unstructured":"Tanushree Mitra and Eric Gilbert . 2015 . CREDBANK: A Large-Scale Social Media Corpus with Associated Credibility Annotations . In Proc. ICWSM'15 . Tanushree Mitra and Eric Gilbert. 2015. CREDBANK: A Large-Scale Social Media Corpus with Associated Credibility Annotations. In Proc. ICWSM'15."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702553"},{"key":"e_1_2_1_48_1","unstructured":"Kevin Munger Mario Luca Jonathan Nagler and Joshua Tucker. 2019. Age matters: Sampling strategies for studying digital media effects.  Kevin Munger Mario Luca Jonathan Nagler and Joshua Tucker. 2019. Age matters: Sampling strategies for studying digital media effects."},{"key":"e_1_2_1_49_1","unstructured":"American Society of Newspaper Editors. 1975. ASNE Statement of Principles. https:\/\/members.newsleaders.org\/content.asp?pl=24&sl=171&contentid=171. (Accessed on 01\/14\/2020).  American Society of Newspaper Editors. 1975. ASNE Statement of Principles. https:\/\/members.newsleaders.org\/content.asp?pl=24&sl=171&contentid=171. (Accessed on 01\/14\/2020)."},{"key":"e_1_2_1_50_1","volume-title":"The International Encyclopedia of Communication","author":"O'Keefe Daniel J","year":"2008","unstructured":"Daniel J O'Keefe . 2008. Persuasion. The International Encyclopedia of Communication ( 2008 ). Daniel J O'Keefe. 2008. Persuasion. The International Encyclopedia of Communication (2008)."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306446.3340818"},{"key":"e_1_2_1_52_1","volume-title":"Rand","author":"Pennycook Gordon","year":"2018","unstructured":"Gordon Pennycook , Tyrone Cannon , and David G . Rand . 2018 . Prior Exposure Increases Perceived Accuracy of Fake News. Number ID 2958246. https:\/\/papers.ssrn.com\/abstract=2958246 Gordon Pennycook, Tyrone Cannon, and David G. Rand. 2018. Prior Exposure Increases Perceived Accuracy of Fake News. Number ID 2958246. https:\/\/papers.ssrn.com\/abstract=2958246"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1806781116"},{"key":"e_1_2_1_54_1","doi-asserted-by":"crossref","unstructured":"Gordon Pennycook and David G. Rand. 2019 b. Who Falls for Fake News? The Roles of Bullshit Receptivity Overclaiming Familiarity and Analytic Thinking. Number ID 3023545. https:\/\/papers.ssrn.com\/abstract=3023545  Gordon Pennycook and David G. Rand. 2019 b. Who Falls for Fake News? The Roles of Bullshit Receptivity Overclaiming Familiarity and Analytic Thinking. Number ID 3023545. https:\/\/papers.ssrn.com\/abstract=3023545","DOI":"10.1111\/jopy.12476"},{"key":"e_1_2_1_55_1","volume-title":"Communication and persuasion","author":"Petty Richard E","unstructured":"Richard E Petty and John T Cacioppo . 1986. The elaboration likelihood model of persuasion . In Communication and persuasion . Springer , 1--24. Richard E Petty and John T Cacioppo. 1986. The elaboration likelihood model of persuasion. In Communication and persuasion. Springer, 1--24."},{"key":"e_1_2_1_56_1","unstructured":"The Trust Project. 2017. Collaborator Materials. https:\/\/thetrustproject.org\/collaborator-materials\/. (Accessed on 01\/14\/2020).  The Trust Project. 2017. Collaborator Materials. https:\/\/thetrustproject.org\/collaborator-materials\/. (Accessed on 01\/14\/2020)."},{"key":"e_1_2_1_57_1","volume-title":"Credibility: A multidisciplinary framework. Annual review of information science and technology","author":"Rieh Soo Young","year":"2007","unstructured":"Soo Young Rieh and David R Danielson . 2007 . Credibility: A multidisciplinary framework. Annual review of information science and technology , Vol. 41 , 1 (2007), 307--364. Soo Young Rieh and David R Danielson. 2007. Credibility: A multidisciplinary framework. Annual review of information science and technology, Vol. 41, 1 (2007), 307--364."},{"key":"e_1_2_1_58_1","unstructured":"Robert M. Ross David G. Rand and Gordon Pennycook. 2019. Beyond 'fake news': The role of analytic thinking in the detection of inaccuracy and partisan bias in news headlines. (2019) 1--22.  Robert M. Ross David G. Rand and Gordon Pennycook. 2019. Beyond 'fake news': The role of analytic thinking in the detection of inaccuracy and partisan bias in news headlines. (2019) 1--22."},{"key":"e_1_2_1_59_1","volume-title":"Proceedings of the ASIS Annual Meeting","volume":"28","author":"Schamber Linda","year":"1991","unstructured":"Linda Schamber . 1991 . Users' Criteria for Evaluation in a Multimedia Environment .. In Proceedings of the ASIS Annual Meeting , Vol. 28 . ERIC, 126--33. Linda Schamber. 1991. Users' Criteria for Evaluation in a Multimedia Environment.. In Proceedings of the ASIS Annual Meeting, Vol. 28. ERIC, 126--33."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1805871115"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292522.3326055"},{"key":"e_1_2_1_62_1","volume-title":"Media Literacy: Keys to Interpreting Media Messages","author":"Silverblatt Art","year":"2014","unstructured":"Art Silverblatt , Donald C. Miller , Julie Smith , and Nikole Brown . 2014 . Media Literacy: Keys to Interpreting Media Messages , 4 th Edition: Keys to Interpreting Media Messages .ABC-CLIO. Art Silverblatt, Donald C. Miller, Julie Smith, and Nikole Brown. 2014. Media Literacy: Keys to Interpreting Media Messages, 4th Edition: Keys to Interpreting Media Messages .ABC-CLIO.","edition":"4"},{"key":"e_1_2_1_63_1","unstructured":"Henry Silverman. 2019. Helping Fact-Checkers Identify False Claims Faster - About Facebook. https:\/\/about.fb.com\/news\/2019\/12\/helping-fact-checkers\/. (Accessed on 01\/10\/2020).  Henry Silverman. 2019. Helping Fact-Checkers Identify False Claims Faster - About Facebook. https:\/\/about.fb.com\/news\/2019\/12\/helping-fact-checkers\/. (Accessed on 01\/10\/2020)."},{"key":"e_1_2_1_64_1","unstructured":"Julianne Stanford Ellen R Tauber BJ Fogg and Leslie Marable. 2002. Experts vs. online consumers: A comparative credibility study of health and finance Web sites .Consumer Web Watch.  Julianne Stanford Ellen R Tauber BJ Fogg and Leslie Marable. 2002. Experts vs. online consumers: A comparative credibility study of health and finance Web sites .Consumer Web Watch."},{"key":"e_1_2_1_65_1","volume-title":"Grounded theory methodology. Handbook of qualitative research","author":"Strauss Anselm","year":"1994","unstructured":"Anselm Strauss and Juliet Corbin . 1994. Grounded theory methodology. Handbook of qualitative research , Vol. 17 ( 1994 ), 273--85. Anselm Strauss and Juliet Corbin. 1994. Grounded theory methodology. Handbook of qualitative research, Vol. 17 (1994), 273--85."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1177\/107769909907600213"},{"key":"e_1_2_1_67_1","volume-title":"The MAIN model: A heuristic approach to understanding technology effects on credibility. Digital media, youth, and credibility","author":"Sundar S Shyam","year":"2008","unstructured":"S Shyam Sundar . 2008. The MAIN model: A heuristic approach to understanding technology effects on credibility. Digital media, youth, and credibility , Vol. 73100 ( 2008 ). S Shyam Sundar. 2008. The MAIN model: A heuristic approach to understanding technology effects on credibility. Digital media, youth, and credibility, Vol. 73100 (2008)."},{"key":"e_1_2_1_68_1","volume-title":"When Crowds Aren't Wise. Harvard Business Review (Sep","author":"Sunstein Cass R.","year":"2006","unstructured":"Cass R. Sunstein . 2006. When Crowds Aren't Wise. Harvard Business Review (Sep 2006 ). https:\/\/hbr.org\/2006\/09\/when-crowds-arent-wise Cass R. Sunstein. 2006. When Crowds Aren't Wise. Harvard Business Review (Sep 2006). https:\/\/hbr.org\/2006\/09\/when-crowds-arent-wise"},{"key":"e_1_2_1_69_1","volume-title":"Economies, Societies and Nations","author":"Surowiecki James","unstructured":"James Surowiecki . 2004. The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business , Economies, Societies and Nations . Doubleday . James Surowiecki. 2004. The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations .Doubleday."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1177\/1948550614567356"},{"key":"e_1_2_1_71_1","volume-title":"The Wisdom of Crowds: Impact of Collective Size and Expertise Transfer on Collective Performance. (Jan","author":"Wagner Christian","year":"2014","unstructured":"Christian Wagner and Ayoung Suh . 2014. The Wisdom of Crowds: Impact of Collective Size and Expertise Transfer on Collective Performance. (Jan 2014 ), 594--603. https:\/\/doi.org\/10.1109\/HICSS.2014.80 10.1109\/HICSS.2014.80 Christian Wagner and Ayoung Suh. 2014. The Wisdom of Crowds: Impact of Collective Size and Expertise Transfer on Collective Performance. (Jan 2014), 594--603. https:\/\/doi.org\/10.1109\/HICSS.2014.80"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.gloenvcha.2011.01.016"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.1193147"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3184558.3188731"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3415164","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3415164","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3415164","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:03:10Z","timestamp":1750197790000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3415164"}},"subtitle":["Raters, Tasks, and Expert Criteria"],"short-title":[],"issued":{"date-parts":[[2020,10,14]]},"references-count":74,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2020,10,14]]}},"alternative-id":["10.1145\/3415164"],"URL":"https:\/\/doi.org\/10.1145\/3415164","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,14]]},"assertion":[{"value":"2020-10-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}