{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T15:05:24Z","timestamp":1776092724622,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":56,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,7,25]],"date-time":"2020-07-25T00:00:00Z","timestamp":1595635200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Facebook Research","award":["The Online Safety Benchmark Research Award"],"award-info":[{"award-number":["The Online Safety Benchmark Research Award"]}]},{"name":"Australian Research Council","award":["DP190102141"],"award-info":[{"award-number":["DP190102141"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,7,25]]},"DOI":"10.1145\/3397271.3401112","type":"proceedings-article","created":{"date-parts":[[2020,7,25]],"date-time":"2020-07-25T07:50:08Z","timestamp":1595663408000},"page":"439-448","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["Can The Crowd Identify Misinformation Objectively?"],"prefix":"10.1145","author":[{"given":"Kevin","family":"Roitero","sequence":"first","affiliation":[{"name":"University of Udine, Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Soprano","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shaoyang","family":"Fan","sequence":"additional","affiliation":[{"name":"University of Queensland, Brisbane, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Damiano","family":"Spina","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefano","family":"Mizzaro","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[{"name":"University of Queensland, Brisbane, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,7,25]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"Analysis of Ordinal Categorical Data","author":"Agresti Alan","year":"2009","unstructured":"Alan Agresti . 2010. Analysis of Ordinal Categorical Data 2 nd ed.). Wiley . 2009 038760 Alan Agresti. 2010. Analysis of Ordinal Categorical Data 2nd ed.). Wiley. 2009038760","edition":"2"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23574"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-40802-1_31"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484081"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3297722"},{"key":"e_1_3_2_2_6_1","volume-title":"Proceedings of HCOMP. 11--20","author":"Checco Alessandro","year":"2017","unstructured":"Alessandro Checco , Kevin Roitero , Eddy Maddalena , Stefano Mizzaro , and Gianluca Demartini . 2017 . Let's Agree to Disagree: Fixing Agreement Measures for Crowdsourcing . In Proceedings of HCOMP. 11--20 . Alessandro Checco, Kevin Roitero, Eddy Maddalena, Stefano Mizzaro, and Gianluca Demartini. 2017. Let's Agree to Disagree: Fixing Agreement Measures for Crowdsourcing. In Proceedings of HCOMP. 11--20."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/2857070.2857152"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341545"},{"key":"e_1_3_2_2_9_1","volume-title":"Proceedings of the CIKM 2018 Workshops co-located with 27th ACM CIKM. CEUR Workshop Proceedings","volume":"2482","author":"Cuzzocrea Alfredo","year":"2019","unstructured":"Alfredo Cuzzocrea , Francesco Bonchi , and Dimitrios Gunopulos ( Eds .). 2019 . Proceedings of the CIKM 2018 Workshops co-located with 27th ACM CIKM. CEUR Workshop Proceedings , Vol. 2482 . CEUR-WS.org. Alfredo Cuzzocrea, Francesco Bonchi, and Dimitrios Gunopulos (Eds.). 2019. Proceedings of the CIKM 2018 Workshops co-located with 27th ACM CIKM. CEUR Workshop Proceedings, Vol. 2482. CEUR-WS.org."},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1080\/00401706.1964.10490181"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159654"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-28577-7_25"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1257\/089533005775196732"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190580.3190586"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.26615\/978-954-452-049-6_037"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S15-2080"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/N15-1124"},{"key":"e_1_3_2_2_18_1","volume-title":"201 The Impact of Task Abandonment in Crowdsourcing","author":"Han Lei","year":"2019","unstructured":"Lei Han , Kevin Roitero , Ujwal Gadiraju , Cristina Sarasua , Alessandro Checco , Eddy Maddalena , and Gianluca Demartini . 201 The Impact of Task Abandonment in Crowdsourcing . IEEE Transactions on Knowledge and Data Engineering ( 2019 ), 1--1. Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini. 201 The Impact of Task Abandonment in Crowdsourcing. IEEE Transactions on Knowledge and Data Engineering (2019), 1--1."},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3357988"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300637"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076063"},{"key":"e_1_3_2_2_22_1","volume-title":"Proceedings of the 7th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Clross-Lingual Information Access, NTCIR-7. NII.","author":"Ed Noriko Kando","year":"2008","unstructured":"Noriko Kando ( Ed .). 2008 . Proceedings of the 7th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Clross-Lingual Information Access, NTCIR-7. NII. Noriko Kando (Ed.). 2008. Proceedings of the 7th NTCIR Workshop Meeting on Evaluation of Information Access Technologies: Information Retrieval, Question Answering and Clross-Lingual Information Access, NTCIR-7. NII."},{"key":"e_1_3_2_2_23_1","volume-title":"Proceedings of SIGIR. 183--192","author":"Kelly Diane","year":"2015","unstructured":"Diane Kelly and Leif Azzopardi . 2015 . How Many Results per Page? A Study of SERP Size, Search Behavior and User Experience . In Proceedings of SIGIR. 183--192 . Diane Kelly and Leif Azzopardi. 2015. How Many Results per Page? A Study of SERP Size, Search Behavior and User Experience. In Proceedings of SIGIR. 183--192."},{"key":"e_1_3_2_2_24_1","unstructured":"Jooyeon Kim Dongkwan Kim and Alice Oh. 2019. Homogeneity-Based Transmissive Process to Model True and False News in Social Networks. 348--356.  Jooyeon Kim Dongkwan Kim and Alice Oh. 2019. Homogeneity-Based Transmissive Process to Model True and False News in Social Networks. 348--356."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531677"},{"key":"e_1_3_2_2_26_1","unstructured":"Klaus Krippendorff. 2011. Computing Krippendorff's Alpha-Reliability. (2011).  Klaus Krippendorff. 2011. Computing Krippendorff's Alpha-Reliability. (2011)."},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1952.10483441"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-45442-5_26"},{"key":"e_1_3_2_2_29_1","unstructured":"Eddy Maddalena Davide Ceolin and Stefano Mizzaro. 2018. Multidimensional News Quality: A Comparison of Crowdsourcing and Nichesourcing See citeNDBLP:conf\/cikm\/2018w.  Eddy Maddalena Davide Ceolin and Stefano Mizzaro. 2018. Multidimensional News Quality: A Comparison of Crowdsourcing and Nichesourcing See citeNDBLP:conf\/cikm\/2018w."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3002172"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3121050.3121060"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1306"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S19-2149"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-98932-7_32"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025727"},{"key":"e_1_3_2_2_36_1","volume-title":"Overclaiming, Familiarity, and Analytic Thinking. Journal of Personality","author":"Pennycook Gordon","year":"2018","unstructured":"Gordon Pennycook and David G Rand . 2018. Who Falls for Fake News? The Roles of Bullshit Receptivity , Overclaiming, Familiarity, and Analytic Thinking. Journal of Personality ( 2018 ), 63. Gordon Pennycook and David G Rand. 2018. Who Falls for Fake News? The Roles of Bullshit Receptivity, Overclaiming, Familiarity, and Analytic Thinking. Journal of Personality (2018), 63."},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1806781116"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cognition.2018.06.011"},{"key":"e_1_3_2_2_40_1","unstructured":"Kevin Roitero Gianluca Demartini Stefano Mizzaro and Damiano Spina. 2018a. How Many Truth Levels? Six? One Hundred? Even More? Validating Truthfulness of Statements via Crowdsourcing See citeNDBLP:conf\/cikm\/2018w.  Kevin Roitero Gianluca Demartini Stefano Mizzaro and Damiano Spina. 2018a. How Many Truth Levels? Six? One Hundred? Even More? Validating Truthfulness of Statements via Crowdsourcing See citeNDBLP:conf\/cikm\/2018w."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210052"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132877"},{"key":"e_1_3_2_2_43_1","volume-title":"Time for Some A Priori Thinking About Post Hoc Testing. Behavioral ecology","author":"Ruxton Graeme D","year":"2008","unstructured":"Graeme D Ruxton and Guy Beauchamp . 2008. Time for Some A Priori Thinking About Post Hoc Testing. Behavioral ecology , Vol. 19 , 3 ( 2008 ), 690--693. Graeme D Ruxton and Guy Beauchamp. 2008. Time for Some A Priori Thinking About Post Hoc Testing. Behavioral ecology, Vol. 19, 3 (2008), 690--693."},{"key":"e_1_3_2_2_44_1","volume-title":"An Approximate Distribution of Estimates of Variance Components. Biometrics bulletin","author":"Satterthwaite Franklin E","year":"1946","unstructured":"Franklin E Satterthwaite . 1946. An Approximate Distribution of Estimates of Variance Components. Biometrics bulletin , Vol. 2 , 6 ( 1946 ), 110--114. Franklin E Satterthwaite. 1946. An Approximate Distribution of Estimates of Variance Components. Biometrics bulletin, Vol. 2, 6 (1946), 110--114."},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.2307\/2413207"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1972.10481232"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-70096-0_59"},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-30796-7_20"},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.26615\/978-954-452-056-4_141"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-2508"},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210037"},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331248"},{"key":"e_1_3_2_2_53_1","volume-title":"Science","volume":"359","author":"Vosoughi Soroush","year":"2018","unstructured":"Soroush Vosoughi , Deb Roy , and Sinan Aral . 2018 . The Spread of True and False News Online . Science , Vol. 359 , 6380 (2018), 1146--1151. Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The Spread of True and False News Online. Science, Vol. 359, 6380 (2018), 1146--1151."},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-2067"},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219903"},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3358006"},{"key":"e_1_3_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13278-014-0163-y"}],"event":{"name":"SIGIR '20: The 43rd International ACM SIGIR conference on research and development in Information Retrieval","location":"Virtual Event China","acronym":"SIGIR '20","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397271.3401112","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3397271.3401112","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:31:38Z","timestamp":1750195898000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3397271.3401112"}},"subtitle":["The Effects of Judgment Scale and Assessor's Background"],"short-title":[],"issued":{"date-parts":[[2020,7,25]]},"references-count":56,"alternative-id":["10.1145\/3397271.3401112","10.1145\/3397271"],"URL":"https:\/\/doi.org\/10.1145\/3397271.3401112","relation":{},"subject":[],"published":{"date-parts":[[2020,7,25]]},"assertion":[{"value":"2020-07-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}