{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T15:05:27Z","timestamp":1776092727493,"version":"3.50.1"},"reference-count":71,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,9,16]],"date-time":"2021-09-16T00:00:00Z","timestamp":1631750400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,9,16]],"date-time":"2021-09-16T00:00:00Z","timestamp":1631750400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"crossref","award":["DP190102141"],"award-info":[{"award-number":["DP190102141"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"crossref","award":["DE200100064"],"award-info":[{"award-number":["DE200100064"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100005801","name":"Facebook","doi-asserted-by":"publisher","award":["Facebook Research Award"],"award-info":[{"award-number":["Facebook Research Award"]}],"id":[{"id":"10.13039\/100005801","id-type":"DOI","asserted-by":"publisher"}]},{"name":"MIT International Science and Technology Initiatives","award":["Seed Fund (MIT-FVG Seed Fund Project)"],"award-info":[{"award-number":["Seed Fund (MIT-FVG Seed Fund Project)"]}]},{"DOI":"10.13039\/501100009874","name":"Regione Autonoma Friuli Venezia Giulia","doi-asserted-by":"publisher","award":["HEaD \u2013 Higher Education and Development - 1619942002 \/ 1420AFPLO1"],"award-info":[{"award-number":["HEaD \u2013 Higher Education and Development - 1619942002 \/ 1420AFPLO1"]}],"id":[{"id":"10.13039\/501100009874","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100008252","name":"Universit\u00e0 degli Studi di Udine","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100008252","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Pers Ubiquit Comput"],"published-print":{"date-parts":[[2023,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Recently, the misinformation problem has been addressed with a crowdsourcing-based approach: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of non-expert is exploited. We study whether crowdsourcing is an effective and reliable method to assess truthfulness during a pandemic, targeting statements related to COVID-19, thus addressing (mis)information that is both related to a sensitive and personal issue and very recent as compared to when the judgment is done. In our experiments, crowd workers are asked to assess the truthfulness of statements, and to provide evidence for the assessments. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we report results on workers\u2019 behavior, agreement among workers, effect of aggregation functions, of scales transformations, and of workers background and bias. We perform a longitudinal study by re-launching the task multiple times with both novice and experienced workers, deriving important insights on how the behavior and quality change over time. Our results show that workers are able to detect and objectively categorize online (mis)information related to COVID-19; both crowdsourced and expert judgments can be transformed and aggregated to improve quality; worker background and other signals (e.g., source of information, behavior) impact the quality of the data. The longitudinal study demonstrates that the time-span has a major effect on the quality of the judgments, for both novice and experienced workers. Finally, we provide an extensive failure analysis of the statements misjudged by the crowd-workers.<\/jats:p>","DOI":"10.1007\/s00779-021-01604-6","type":"journal-article","created":{"date-parts":[[2021,9,16]],"date-time":"2021-09-16T12:02:48Z","timestamp":1631793768000},"page":"59-89","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Can the crowd judge truthfulness? A longitudinal study on recent misinformation about COVID-19"],"prefix":"10.1007","volume":"27","author":[{"given":"Kevin","family":"Roitero","sequence":"first","affiliation":[]},{"given":"Michael","family":"Soprano","sequence":"additional","affiliation":[]},{"given":"Beatrice","family":"Portelli","sequence":"additional","affiliation":[]},{"given":"Massimiliano","family":"De Luise","sequence":"additional","affiliation":[]},{"given":"Damiano","family":"Spina","sequence":"additional","affiliation":[]},{"given":"Vincenzo Della","family":"Mea","sequence":"additional","affiliation":[]},{"given":"Giuseppe","family":"Serra","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2852-168X","authenticated-orcid":false,"given":"Stefano","family":"Mizzaro","sequence":"additional","affiliation":[]},{"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,9,16]]},"reference":[{"key":"1604_CR1","doi-asserted-by":"publisher","unstructured":"Amig\u00f3 E, Gonzalo J, Mizzaro S, Carrillo-de Albornoz J (2020) An effectiveness metric for ordinal classification: formal properties and experimental results. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, ACL \u201920, pp 3938\u20133949. https:\/\/doi.org\/10.18653\/v1\/2020.acl-main.363","DOI":"10.18653\/v1\/2020.acl-main.363"},{"key":"1604_CR2","doi-asserted-by":"publisher","unstructured":"Atanasova P, Nakov P, M\u00e0rquez L, Barr\u00f3n-Cede\u00f1o A, Karadzhov G, Mihaylova T, Mohtarami M, Glass J (2019) Automatic fact-checking using context and discourse information. Journal of Data and Information Quality 11(3). https:\/\/doi.org\/10.1145\/3297722","DOI":"10.1145\/3297722"},{"issue":"3","key":"1604_CR3","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1609\/aimag.v38i3.2704","volume":"38","author":"K Atkinson","year":"2017","unstructured":"Atkinson K, Baroni P, Giacomin M, Hunter A, Prakken H, Reed C, Simari G, Thimm M, Villata S (2017) Towards artificial argumentation. AI Mag 38(3):25\u201336. https:\/\/doi.org\/10.1609\/aimag.v38i3.2704","journal-title":"AI Mag"},{"key":"1604_CR4","doi-asserted-by":"crossref","unstructured":"Barr\u00f3n-Cedeno A, Elsayed T, Nakov P, Da San Martino G, Hasanain M, Suwaileh R, Haouari F, Babulkov N, Hamdan B, Nikolov A et al (2020) Overview of checkthat! 2020: automatic identification and verification of claims in social media. In: International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, CLEF \u201920, pp 215\u2013236","DOI":"10.1007\/978-3-030-58219-7_17"},{"key":"1604_CR5","doi-asserted-by":"crossref","unstructured":"Bullock J, Luccioni A, Pham KH, Lam CSN, Luengo-Oroz MA (2020) Mapping the landscape of artificial intelligence applications against COVID-19. 2003.11336","DOI":"10.1613\/jair.1.12162"},{"key":"1604_CR6","doi-asserted-by":"publisher","unstructured":"Cerone A, Naghizade E, Scholer F, Mallal D, Skelton R, Spina D (2020) Watch \u2018n\u2019 Check: towards a social media monitoring tool to assist fact-checking experts. In: 2020 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pp 607\u2013613. https:\/\/doi.org\/10.1109\/DSAA49011.2020.00085","DOI":"10.1109\/DSAA49011.2020.00085"},{"key":"1604_CR7","doi-asserted-by":"crossref","unstructured":"Checco A, Roitero K, Maddalena E, Mizzaro S, Demartini G (2017) Let\u2019s agree to disagree: fixing agreement measures for Crowdsourcing. In: Dow S, Kalai A T (eds) Proceedings of the Fifth AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2017, 23-26 October 2017. AAAI Press, Qu\u00e9bec City, Qu\u00e9bec, Canada, pp 11\u201320","DOI":"10.1609\/hcomp.v5i1.13306"},{"issue":"5","key":"1604_CR8","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1016\/j.acalib.2015.07.003","volume":"41","author":"X Chen","year":"2015","unstructured":"Chen X, Sin SCJ, Theng YL, Lee CS (2015) Why students share misinformation on social media: motivation, gender, and study-level differences. The Journal of Academic Librarianship 41(5):583\u2013592. https:\/\/doi.org\/10.1016\/j.acalib.2015.07.003","journal-title":"The Journal of Academic Librarianship"},{"key":"1604_CR9","doi-asserted-by":"crossref","unstructured":"Cinelli M, Quattrociocchi W, Galeazzi A, Valensise CM, Brugnoli E, Schmidt AL, Zola P, Zollo F, Scala A (2020) The COVID-19 Social Media Infodemic. 2003.05004","DOI":"10.1038\/s41598-020-73510-5"},{"issue":"3","key":"1604_CR10","first-page":"65","volume":"43","author":"G Demartini","year":"2020","unstructured":"Demartini G, Mizzaro S, Spina D (2020) Human-in-the-loop artificial intelligence for fighting online misinformation: challenges and opportunities. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 43(3):65\u201374. http:\/\/sites.computer.org\/debull\/A20sept\/p65.pdf","journal-title":"Bulletin of the IEEE Computer Society Technical Committee on Data Engineering"},{"issue":"5","key":"1604_CR11","doi-asserted-by":"publisher","first-page":"473","DOI":"10.1038\/s43018-020-0065-z","volume":"1","author":"A Desai","year":"2020","unstructured":"Desai A, Warner J, Kuderer N, Thompson M, Painter C, Lyman G, Lopes G (2020) Crowdsourcing a crisis response for COVID-19 in oncology. Nature Cancer 1(5):473\u2013476. https:\/\/doi.org\/10.1038\/s43018-020-0065-z","journal-title":"Nature Cancer"},{"key":"1604_CR12","doi-asserted-by":"crossref","unstructured":"Elsayed T, Nakov P, Barr\u00f3n-Cede\u00f1o A, Hasanain M, Suwaileh R, Da San Martino G, Atanasova P (2019) Overview of the CLEF-2019 CheckThat! Lab: automatic identification and verification of claims. In: International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, CLEF \u201919, pp 301\u2013321","DOI":"10.1007\/978-3-030-28577-7_25"},{"issue":"3","key":"1604_CR13","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1177\/014662169602000302","volume":"20","author":"SE Embretson","year":"1996","unstructured":"Embretson SE (1996) Item response theory models and spurious interaction effects in factorial ANOVA designs. Appl Psychol Meas 20(3):201\u2013212. https:\/\/doi.org\/10.1177\/014662169602000302","journal-title":"Appl Psychol Meas"},{"key":"1604_CR14","doi-asserted-by":"publisher","unstructured":"Ferro N, Silvello G (2016) A general linear mixed models approach to study system component effects. In: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, Association for Computing Machinery, New York, NY, USA, SIGIR \u201916, pp 25\u201334. https:\/\/doi.org\/10.1145\/2911451.2911530","DOI":"10.1145\/2911451.2911530"},{"issue":"2","key":"1604_CR15","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1002\/asi.23910","volume":"69","author":"N Ferro","year":"2018","unstructured":"Ferro N, Silvello G (2018) Toward an anatomy of IR system component performances. Journal of the Association for Information Science and Technology 69(2):187\u2013200. https:\/\/doi.org\/10.1002\/asi.23910","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"1604_CR16","doi-asserted-by":"publisher","unstructured":"Ferro N, Kim Y, Sanderson M (2019) Using collection shards to study retrieval performance effect sizes. ACM Transactions on Information Systems 37(3). https:\/\/doi.org\/10.1145\/3310364","DOI":"10.1145\/3310364"},{"issue":"S1","key":"1604_CR17","doi-asserted-by":"publisher","first-page":"298","DOI":"10.1093\/poq\/nfw006","volume":"80","author":"S Flaxman","year":"2016","unstructured":"Flaxman S, Goel S, Rao JM (2016) Filter bubbles, echo chambers, and online news consumption. Public Opinion Quarterly 80 (S1):298\u2013320. https:\/\/doi.org\/10.1093\/poq\/nfw006, https:\/\/academic.oup.com\/poq\/article-pdf\/80\/S1\/298\/17120810\/nfw006.pdf","journal-title":"Public Opinion Quarterly"},{"issue":"4","key":"1604_CR18","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1257\/089533005775196732","volume":"19","author":"S Frederick","year":"2005","unstructured":"Frederick S (2005) Cognitive reflection and decision making. J Econ Perspect 19(4):25\u201342. https:\/\/doi.org\/10.1257\/089533005775196732","journal-title":"J Econ Perspect"},{"key":"1604_CR19","doi-asserted-by":"crossref","unstructured":"Gallotti R, Valle F, Castaldo N, Sacco P, Domenico MD (2020) Assessing the risks of \u201cinfodemics\u201d in response to covid-19 epidemics. 2004.03997","DOI":"10.1101\/2020.04.08.20057968"},{"issue":"2","key":"1604_CR20","doi-asserted-by":"publisher","first-page":"184","DOI":"10.1093\/bioinformatics\/btl572","volume":"23","author":"JR de Haan","year":"2006","unstructured":"de Haan JR, Wehrens R, Bauerschmidt S, Piek E, Schaik RCv, Buydens LMC (2006) Interpretation of ANOVA models for microarray data using PCA. Bioinformatics 23(2):184\u2013190. https:\/\/doi.org\/10.1093\/bioinformatics\/btl572","journal-title":"Bioinformatics"},{"key":"1604_CR21","doi-asserted-by":"publisher","unstructured":"Han L, Roitero K, Gadiraju U, Sarasua C, Checco A, Maddalena E, Demartini G (2019) All those wasted hours: on task abandonment in crowdsourcing. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, Association for Computing Machinery, New York, NY, USA, WSDM \u201919, pp 321\u2013329. https:\/\/doi.org\/10.1145\/3289600.3291035","DOI":"10.1145\/3289600.3291035"},{"issue":"01","key":"1604_CR22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TKDE.2019.2948168","volume":"1","author":"L Han","year":"2019","unstructured":"Han L, Roitero K, Gadiraju U, Sarasua C, Checco A, Maddalena E, Demartini G (2019) The impact of task abandonment in crowdsourcing. IEEE Trans Knowl Data Eng 1(01):1\u20131. https:\/\/doi.org\/10.1109\/TKDE.2019.2948168","journal-title":"IEEE Trans Knowl Data Eng"},{"key":"1604_CR23","doi-asserted-by":"publisher","unstructured":"Han L, Roitero K, Maddalena E, Mizzaro S, Demartini G (2019) On transforming relevance scales. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Association for Computing Machinery, New York, NY, USA, CIKM \u201919, pp 39\u201348. https:\/\/doi.org\/10.1145\/3357384.3357988","DOI":"10.1145\/3357384.3357988"},{"key":"1604_CR24","doi-asserted-by":"crossref","unstructured":"Horne B, Adali S (2017) This just in: fake news packs a lot in title, uses simpler, repetitive content in text body, more similar to satire than real news. Proceedings of the International AAAI Conference on Web and Social Media 11(1). https:\/\/ojs.aaai.org\/index.php\/ICWSM\/article\/view\/14976","DOI":"10.1609\/icwsm.v11i1.14976"},{"key":"1604_CR25","doi-asserted-by":"publisher","unstructured":"Kim J, Kim D, Oh A (2019) Homogeneity-based transmissive process to model true and false news in social networks. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, Association for Computing Machinery, New York, NY, USA, WSDM \u201919, pp 348\u2013356. https:\/\/doi.org\/10.1145\/3289600.3291009","DOI":"10.1145\/3289600.3291009"},{"key":"1604_CR26","unstructured":"Krippendorff K (2011) Computing Krippendorff\u2019s alpha- reliability. http:\/\/repository.upenn.edu\/asc_papers\/43"},{"key":"1604_CR27","doi-asserted-by":"crossref","unstructured":"La Barbera D, Roitero K, Demartini G, Mizzaro S, Spina D (2020) Crowdsourcing truthfulness: the impact of judgment scale and assessor bias. In: Jose JM, Yilmaz E, Magalh\u00e3es J, Castells P, Ferro N, Silva MJ, Martins F (eds) Advances in information retrieval: proceedings of the European Conference on Information Retrieval, Springer International Publishing, Cham, ECIR \u201920, pp 207\u2013214","DOI":"10.1007\/978-3-030-45442-5_26"},{"issue":"4","key":"1604_CR28","doi-asserted-by":"publisher","first-page":"765","DOI":"10.1162\/coli_a_00364","volume":"45","author":"J Lawrence","year":"2020","unstructured":"Lawrence J, Reed C (2020) Argument mining: a survey. Computational Linguistics 45 (4):765\u2013818. https:\/\/doi.org\/10.1162\/coli_a_00364, https:\/\/direct.mit.edu\/coli\/article-pdf\/45\/4\/765\/1847520\/coli_a_00364.pdf","journal-title":"Computational Linguistics"},{"issue":"3","key":"1604_CR29","doi-asserted-by":"publisher","first-page":"205316801878684","DOI":"10.1177\/2053168018786848","volume":"5","author":"C Lim","year":"2018","unstructured":"Lim C (2018) Checking how fact-checkers check. Research & Politics 5(3):2053168018786848. https:\/\/doi.org\/10.1177\/2053168018786848","journal-title":"Research & Politics"},{"key":"1604_CR30","doi-asserted-by":"publisher","unstructured":"Maddalena E, Mizzaro S, Scholer F, Turpin A (2017) On crowdsourcing relevance magnitudes for information retrieval evaluation. ACM Trans Inf Syst 35(3). https:\/\/doi.org\/10.1145\/3002172","DOI":"10.1145\/3002172"},{"key":"1604_CR31","doi-asserted-by":"publisher","unstructured":"Maddalena E, Roitero K, Demartini G, Mizzaro S (2017) Considering assessor agreement in IR evaluation. In: Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval, Association for Computing Machinery, New York, NY, USA, ICTIR \u201917, pp 75\u201382. https:\/\/doi.org\/10.1145\/3121050.3121060","DOI":"10.1145\/3121050.3121060"},{"key":"1604_CR32","doi-asserted-by":"crossref","unstructured":"McCullagh P (2018) Generalized Linear Models. Routledge","DOI":"10.1201\/9780203753736"},{"key":"1604_CR33","unstructured":"Mejova Y, Kalimeri K (2020) Advertisers Jump on Coronavirus Bandwagon: Politics, News, and Business. 2003.00923"},{"key":"1604_CR34","doi-asserted-by":"publisher","unstructured":"Mihaylova T, Karadzhov G, Atanasova P, Baly R, Mohtarami M, Nakov P (2019) SemEval-2019 Task 8: fact checking in community question answering forums. In: Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Minneapolis, Minnesota, USA, SemEval-2019, pp 860\u2013869. https:\/\/doi.org\/10.18653\/v1\/S19-2149","DOI":"10.18653\/v1\/S19-2149"},{"key":"1604_CR35","doi-asserted-by":"crossref","unstructured":"Morrison DF (2005) Multivariate analysis of variance. Encyclopedia of Biostatistics 5","DOI":"10.1002\/0470011815.b2a13045"},{"issue":"3","key":"1604_CR36","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1111\/j.1745-4557.1977.tb00942.x","volume":"1","author":"H Moskowitz","year":"1977","unstructured":"Moskowitz H (1977) Magnitude estimation: notes on what, how, when, and why to use it. J Food Qual 1(3):195\u2013227","journal-title":"J Food Qual"},{"key":"1604_CR37","doi-asserted-by":"crossref","unstructured":"Nakov P, Barr\u00f3n-Cede\u00f1o A, Elsayed T, Suwaileh R, M\u00e0rquez L, Zaghouani W, Atanasova P, Kyuchukov S, Da San Martino G (2018) Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims. In: Bellot P, Trabelsi C, Mothe J, Murtagh F, Nie JY, Soulier L, SanJuan E, Cappellato L, Ferro N (eds) Experimental IR Meets Multilinguality, Multimodality, and Interaction. Springer International Publishing, Cham, pp 372\u2013387","DOI":"10.1007\/978-3-319-98932-7_32"},{"issue":"2","key":"1604_CR38","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1017\/epi.2018.32","volume":"17","author":"CT Nguyen","year":"2020","unstructured":"Nguyen CT (2020) Echo chambers and epistemic bubbles. Episteme 17 (2):141\u2013161. https:\/\/doi.org\/10.1017\/epi.2018.32","journal-title":"Episteme"},{"key":"1604_CR39","doi-asserted-by":"publisher","unstructured":"Nguyen TT, Weidlich M, Yin H, Zheng B, Nguyen QH, Nguyen QVH (2020) FactCatch: incremental pay-as-you-go fact checking with minimal user effort. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Association for Computing Machinery, New York, NY, USA, SIGIR \u201920, pp 2165\u20132168. https:\/\/doi.org\/10.1145\/3397271.3401408","DOI":"10.1145\/3397271.3401408"},{"issue":"4","key":"1604_CR40","doi-asserted-by":"publisher","first-page":"434","DOI":"10.1037\/1082-989X.8.4.434","volume":"8","author":"S Olejnik","year":"2003","unstructured":"Olejnik S, Algina J (2003) Generalized eta and omega squared statistics: measures of effect size for some common research designs. Psychological Methods 8(4):434","journal-title":"Psychological Methods"},{"key":"1604_CR41","unstructured":"Politifact (2021) Politifact IFCN principles. https:\/\/ifcncodeofprinciples.poynter.org\/profile\/politifact, (accessed: 20.04.2021)"},{"key":"1604_CR42","unstructured":"Politifact (2021) The principles of the Truth-O-Meter: PolitiFact\u2019s methodology for independent fact-checking. https:\/\/www.politifact.com\/article\/2018\/feb\/12\/principles-truth-o-meter-politifacts-methodology-i\/#Truth-O-Meter%20ratings, (accessed: 20.04.2021)"},{"key":"1604_CR43","doi-asserted-by":"publisher","unstructured":"Popat KK (2019) Credibility analysis of textual claims with explainable evidence. PhD thesis, Saarland University. https:\/\/doi.org\/10.22028\/D291-30005","DOI":"10.22028\/D291-30005"},{"issue":"1","key":"1604_CR44","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1609\/hcomp.v7i1.5264","volume":"7","author":"R Qarout","year":"2019","unstructured":"Qarout R, Checco A, Demartini G, Bontcheva K (2019) Platform-related factors in repeatability and reproducibility of crowdsourcing tasks. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 7(1):135\u2013143. https:\/\/ojs.aaai.org\/index.php\/HCOMP\/article\/view\/5264","journal-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing"},{"key":"1604_CR45","doi-asserted-by":"publisher","unstructured":"Rashkin H, Choi E, Jang JY, Volkova S, Choi Y (2017) Truth of varying shades: analyzing language in fake news and political fact-checking. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics. Denmark, Copenhagen, pp 2931\u20132937, DOI https:\/\/doi.org\/10.18653\/v1\/D17-1317, (to appear in print)","DOI":"10.18653\/v1\/D17-1317"},{"key":"1604_CR46","doi-asserted-by":"publisher","unstructured":"Roberts K, Alam T, Bedrick S, Demner-Fushman D, Lo K, Soboroff I, Voorhees E, Wang LL, Hersh WR (2020) TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19. Journal of the American Medical Informatics Association https:\/\/doi.org\/10.1093\/jamia\/ocaa091","DOI":"10.1093\/jamia\/ocaa091"},{"key":"1604_CR47","unstructured":"Roitero K, Demartini G, Mizzaro S, Spina D (2018) How many truth levels? six? one hundred? even more? validating truthfulness of statements via Crowdsourcing. In: Proceedings of the CIKM Workshop on Rumours and Deception in Social Media (RDSM\u201918)"},{"key":"1604_CR48","doi-asserted-by":"publisher","unstructured":"Roitero K, Maddalena E, Demartini G, Mizzaro S (2018) On fine-grained relevance scales. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Association for Computing Machinery, New York, NY, USA, SIGIR \u201918, pp 675\u2013684. https:\/\/doi.org\/10.1145\/3209978.3210052","DOI":"10.1145\/3209978.3210052"},{"key":"1604_CR49","doi-asserted-by":"publisher","unstructured":"Roitero K, Maddalena E, Demartini G, Mizzaro S (2018) On fine-grained relevance scales. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Association for Computing Machinery, New York, NY, USA, SIGIR \u201918, pp 675\u2013684. https:\/\/doi.org\/10.1145\/3209978.3210052","DOI":"10.1145\/3209978.3210052"},{"key":"1604_CR50","doi-asserted-by":"publisher","unstructured":"Roitero K, Carterrete B, Mehrotra R, Lalmas M (2020) Leveraging behavioral heterogeneity across markets for cross-market training of recommender systems. In: Companion Proceedings of the Web Conference 2020, Association for Computing Machinery, New York, NY, USA, WWW \u201920, pp 694\u2013702. https:\/\/doi.org\/10.1145\/3366424.3384362","DOI":"10.1145\/3366424.3384362"},{"key":"1604_CR51","doi-asserted-by":"publisher","unstructured":"Roitero K, Soprano M, Fan S, Spina D, Mizzaro S, Demartini G (2020) Can the crowd identify misinformation objectively? the effects of judgment scale and assessor\u2019s background. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Association for Computing Machinery, New York, NY, USA, SIGIR \u201920, pp 439\u2013448. https:\/\/doi.org\/10.1145\/3397271.3401112","DOI":"10.1145\/3397271.3401112"},{"key":"1604_CR52","doi-asserted-by":"publisher","unstructured":"Roitero K, Soprano M, Portelli B, Spina D, Della Mea V, Serra G, Mizzaro S, Demartini G (2020) The COVID-19 infodemic: can the crowd judge recent misinformation objectively? In: Proceedings of the 29th ACM International Conference on Information and Knowledge Management, Association for Computing Machinery, New York, NY, USA, CIKM \u201920, pp 1305\u20131314. https:\/\/doi.org\/10.1145\/3340531.3412048","DOI":"10.1145\/3340531.3412048"},{"key":"1604_CR53","doi-asserted-by":"publisher","unstructured":"Sethi R, Rangaraju R (2018) Extinguishing the backfire effect: using emotions in online social collaborative argumentation for fact checking. In: 2018 IEEE International conference on web services (ICWS), vol 1, pp 363\u2013366. https:\/\/doi.org\/10.1109\/ICWS.2018.00062","DOI":"10.1109\/ICWS.2018.00062"},{"key":"1604_CR54","doi-asserted-by":"publisher","unstructured":"Sethi RJ (2017) Spotting fake news: a social argumentation framework for scrutinizing alternative facts. In: 2017 IEEE International Conference on Web Services (ICWS), pp 866\u2013869. https:\/\/doi.org\/10.1109\/ICWS.2017.108","DOI":"10.1109\/ICWS.2017.108"},{"key":"1604_CR55","doi-asserted-by":"crossref","unstructured":"Sethi RJ, Rangaraju R, Shurts B (2019) Fact checking misinformation using recommendations from emotional pedagogical agents. In: International Conference on Intelligent Tutoring Systems, Springer, pp 99\u2013104","DOI":"10.1007\/978-3-030-22244-4_13"},{"key":"1604_CR56","doi-asserted-by":"publisher","unstructured":"Shrestha A, Spezzano F (2021) Textual characteristics of news title and body to detect fake news: a reproducibility study. In: Hiemstra D, Moens M, Mothe J, Perego R, Potthast M, Sebastiani F (eds) Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April 1, 2021, Proceedings, Part II, Springer, Lecture Notes in Computer Science, vol 12657, pp 120\u2013133. https:\/\/doi.org\/10.1007\/978-3-030-72240-1_9","DOI":"10.1007\/978-3-030-72240-1_9"},{"key":"1604_CR57","unstructured":"Snaith M, Lawrence J, Pease A, Reed C (2020) A modular platform for argument and dialogue. In: 8Th international conference on computational models of argument, COMMA 2020, pp 473\u2013474"},{"issue":"6","key":"1604_CR58","doi-asserted-by":"publisher","first-page":"e19659","DOI":"10.2196\/19659","volume":"22","author":"V Tangcharoensathien","year":"2020","unstructured":"Tangcharoensathien V, Calleja N, Nguyen T, Purnat T, D\u2019Agostino M, Garcia-Saiso S, Landry M, Rashidian A, Hamilton C, AbdAllah A, Ghiga I, Hill A, Hougendobler D, van Andel J, Nunn M, Brooks I, Sacco PL, De Domenico M, Mai P, Gruzd A, Alaphilippe A, Briand S (2020) Framework for managing the covid-19 infodemic: Methods and results of an online, crowdsourced who technical consultation. J Med Internet Res 22(6):e19659. https:\/\/doi.org\/10.2196\/19659","journal-title":"J Med Internet Res"},{"key":"1604_CR59","doi-asserted-by":"crossref","unstructured":"Tchechmedjiev A, Fafalios P, Boland K, Gasquet M, Zloch M, Zapilko B, Dietze S, Todorov K (2019) ClaimsKG: a knowledge graph of fact-checked claims. In: Ghidini C, Hartig O, Maleshkova M, Sv\u00e1tek V, Cruz I, Hogan A, Song J, Lefran\u0107ois M, Gandon F (eds) The semantic web \u2013 ISWC, vol 2019. Springer International Publishing, Cham, pp 309\u2013324","DOI":"10.1007\/978-3-030-30796-7_20"},{"key":"1604_CR60","unstructured":"Times TNY (2021) The real story about fake news is partisanship. https:\/\/www.nytimes.com\/2017\/01\/11\/upshot\/the-real-story-about-fake-news-is-partisanship.html, (accessed: 20.04.2021)"},{"key":"1604_CR61","volume-title":"The uses of argument","author":"SE Toulmin","year":"1958","unstructured":"Toulmin SE (1958) The uses of argument. Cambridge University Press, Cambridge"},{"issue":"9","key":"1604_CR62","doi-asserted-by":"publisher","first-page":"e0203958","DOI":"10.1371\/journal.pone.0203958","volume":"13","author":"P T\u00f6rnberg","year":"2018","unstructured":"T\u00f6rnberg P (2018) Echo chambers and viral misinformation: Modeling fake news as complex contagion. PLoS One 13(9):e0203958","journal-title":"PLoS One"},{"issue":"9","key":"1604_CR63","doi-asserted-by":"publisher","first-page":"e22767","DOI":"10.2196\/22767","volume":"22","author":"JY Tsai","year":"2020","unstructured":"Tsai JY, Phua J, Pan S, Cc Yang (2020) Intergroup contact, covid-19 news consumption, and the moderating role of digital media trust on prejudice toward asians in the united states: Cross-sectional study. J Med Internet Res 22(9):e22767. https:\/\/doi.org\/10.2196\/22767","journal-title":"J Med Internet Res"},{"issue":"11","key":"1604_CR64","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1145\/3397189","volume":"63","author":"J Visser","year":"2020","unstructured":"Visser J, Lawrence J, Reed C (2020) Reason-checking fake news. Commun ACM 63 (11):38\u201340. https:\/\/doi.org\/10.1145\/3397189","journal-title":"Commun ACM"},{"key":"1604_CR65","doi-asserted-by":"publisher","unstructured":"Wang WY (2017) \u201cLiar, Liar Pants on Fire\u201d: a new benchmark dataset for fake news detection. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Vancouver, Canada, ACL \u201917, pp 422\u2013426. https:\/\/doi.org\/10.18653\/v1\/P17-2067","DOI":"10.18653\/v1\/P17-2067"},{"key":"1604_CR66","doi-asserted-by":"crossref","unstructured":"Wani A, Joshi I, Khandve S, Wagh V, Joshi R (2021) Evaluating deep learning approaches for covid19 fake news detection. In: Chakraborty T, Shu K, Bernard HR, Liu H, Akhtar MS (eds) Combating online hostile posts in regional languages during emergency situation. Springer International Publishing, Cham, pp 153\u2013163","DOI":"10.1007\/978-3-030-73696-5_15"},{"key":"1604_CR67","doi-asserted-by":"publisher","unstructured":"Webber W, Moffat A, Zobel J (2010) A similarity measure for indefinite rankings. ACM Trans Inf Syst 28(4). https:\/\/doi.org\/10.1145\/1852102.1852106","DOI":"10.1145\/1852102.1852106"},{"issue":"3","key":"1604_CR68","doi-asserted-by":"publisher","first-page":"399","DOI":"10.2307\/2183556","volume":"69","author":"FL Will","year":"1960","unstructured":"Will FL (1960) The uses of argument. The Philosophical Review 69(3):399\u2013403. http:\/\/www.jstor.org\/stable\/2183556","journal-title":"The Philosophical Review"},{"key":"1604_CR69","unstructured":"Yang KC, Torres-Lugo C, Menczer F (2020) Prevalence of Low-Credibility Information on Twitter During the COVID-19 Outbreak. 2004.14484"},{"key":"1604_CR70","doi-asserted-by":"publisher","unstructured":"Zampieri F, Roitero K, Culpepper JS, Kurland O, Mizzaro S (2019) On topic difficulty in ir evaluation: the effect of systems, corpora, and system components. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, Association for Computing Machinery, New York, NY, USA, SIGIR\u201919, pp 909\u2013912. https:\/\/doi.org\/10.1145\/3331184.3331279","DOI":"10.1145\/3331184.3331279"},{"issue":"1","key":"1604_CR71","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s13278-014-0163-y","volume":"4","author":"A Zubiaga","year":"2014","unstructured":"Zubiaga A, Ji H (2014) Tweet, but verify: epistemic study of information verification on twitter. Social Network Analysis and Mining 4(1):1\u201312","journal-title":"Social Network Analysis and Mining"}],"container-title":["Personal and Ubiquitous Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00779-021-01604-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00779-021-01604-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00779-021-01604-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,5]],"date-time":"2023-01-05T09:09:23Z","timestamp":1672909763000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00779-021-01604-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,16]]},"references-count":71,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,2]]}},"alternative-id":["1604"],"URL":"https:\/\/doi.org\/10.1007\/s00779-021-01604-6","relation":{},"ISSN":["1617-4909","1617-4917"],"issn-type":[{"value":"1617-4909","type":"print"},{"value":"1617-4917","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,16]]},"assertion":[{"value":"1 February 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 July 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 September 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}