{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:09:13Z","timestamp":1755839353645,"version":"3.41.0"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T00:00:00Z","timestamp":1672185600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Facebook Research award, the Australian Research Council","award":["DP190102141, and DE200100064"],"award-info":[{"award-number":["DP190102141, and DE200100064"]}]},{"name":"ARC Training Centre for Information Resilience","award":["IC200100022"],"award-info":[{"award-number":["IC200100022"]}]},{"name":"ARC Centre of Excellence for Automated Decision-Making and Society","award":["CE200100005"],"award-info":[{"award-number":["CE200100005"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Data and Information Quality"],"published-print":{"date-parts":[[2023,3,31]]},"abstract":"<jats:p>Automatically detecting online misinformation at scale is a challenging and interdisciplinary problem. Deciding what is to be considered truthful information is sometimes controversial and also difficult for educated experts. As the scale of the problem increases, human-in-the-loop approaches to truthfulness that combine both the scalability of machine learning (ML) and the accuracy of human contributions have been considered.<\/jats:p>\n          <jats:p>In this work, we look at the potential to automatically combine machine-based systems with human-based systems. The former exploit superviseds ML approaches; the latter involve either crowd workers (i.e., human non-experts) or human experts. Since both ML and crowdsourcing approaches can produce a score indicating the level of confidence on their truthfulness judgments (either algorithmic or self-reported, respectively), we address the question of whether it is feasible to make use of such confidence scores to effectively and efficiently combine three approaches: (i) machine-based methods, (ii) crowd workers, and (iii) human experts. The three approaches differ significantly, as they range from available, cheap, fast, scalable, but less accurate to scarce, expensive, slow, not scalable, but highly accurate.<\/jats:p>","DOI":"10.1145\/3546916","type":"journal-article","created":{"date-parts":[[2022,7,11]],"date-time":"2022-07-11T11:22:49Z","timestamp":1657538569000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Combining Human and Machine Confidence in Truthfulness Assessment"],"prefix":"10.1145","volume":"15","author":[{"given":"Yunke","family":"Qu","sequence":"first","affiliation":[{"name":"The University of Queensland, Brisbane, Australia"}]},{"given":"Kevin","family":"Roitero","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, Italy"}]},{"given":"David La","family":"Barbera","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, Italy"}]},{"given":"Damiano","family":"Spina","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}]},{"given":"Stefano","family":"Mizzaro","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, Italy"}]},{"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[{"name":"The University of Queensland, Brisbane, Australia"}]}],"member":"320","published-online":{"date-parts":[[2022,12,28]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.abf4393"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i4.2513"},{"key":"e_1_3_2_4_2","unstructured":"Lora Aroyo and Chris Welty. 2013. Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. In The 5th International ACM Conference on Web Science in 2013 (WebSci\u201913) . ACM Paris France."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/2700832"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v5i1.13306"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1561\/1800000025"},{"issue":"3","key":"e_1_3_2_9_2","first-page":"65","article-title":"Human-in-the-loop artificial intelligence for fighting online misinformation: Challenges and opportunities","volume":"43","author":"Demartini Gianluca","year":"2020","unstructured":"Gianluca Demartini, Stefano Mizzaro, and Damiano Spina. 2020. Human-in-the-loop artificial intelligence for fighting online misinformation: Challenges and opportunities. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 43, 3 (Sep2020), 65\u201374. http:\/\/sites.computer.org\/debull\/A20sept\/p65.pdf","journal-title":"Bulletin of the IEEE Computer Society Technical Committee on Data Engineering"},{"key":"e_1_3_2_10_2","volume-title":"Proceedings of the Biennial Conference on Innovative Data Systems Research (CIDR\u201913)","author":"Demartini Gianluca","year":"2013","unstructured":"Gianluca Demartini, Beth Trushkowsky, Tim Kraska, Michael J. Franklin, and UC Berkeley. 2013. CrowdQ: Crowdsourced query understanding. In Proceedings of the Biennial Conference on Innovative Data Systems Research (CIDR\u201913). www.cidrdb.org, Asilomar, CA, USA. http:\/\/cidrdb.org\/cidr2013\/Papers\/CIDR13_Paper137.pdf"},{"key":"e_1_3_2_11_2","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume":"1810","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR abs\/1810.04805 (2018). arXiv:1810.04805http:\/\/arxiv.org\/abs\/1810.04805","journal-title":"CoRR"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1423"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883030"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989331"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/mis.2015.66"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3119930"},{"key":"e_1_3_2_17_2","series-title":"Proceedings of the 34th International Conference on Machine Learning","first-page":"1321","volume":"70","author":"Guo Chuan","year":"2017","unstructured":"Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 1321\u20131330. https:\/\/proceedings.mlr.press\/v70\/guo17a.html"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1221403"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/HotWeb.2015.9"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2007.289"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487595"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-45442-5_26"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3055601.3055607"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3121050.3121060"},{"key":"e_1_3_2_25_2","unstructured":"Amit Mandelbaum and Daphna Weinshall. 2017. Distance-based confidence score for neural network classifiers. (2017). https:\/\/arxiv.org\/abs\/1709.09844"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2021\/619"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/1743384.1743478"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.559"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401112"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00779-021-01604-6"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412048"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0255702"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35176-1_33"},{"key":"e_1_3_2_34_2","volume-title":"ICLR 2022","author":"Sellam Thibault","year":"2022","unstructured":"Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander Nicholas D\u2019Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Turc, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick. 2022. The MultiBERTs: BERT reproductions for robustness analysis. In ICLR 2022. https:\/\/arxiv.org\/abs\/2106.16163"},{"key":"e_1_3_2_35_2","volume-title":"Active Learning Literature Survey","author":"Settles Burr","year":"2009","unstructured":"Burr Settles. 2009. Active Learning Literature Survey. Technical Report. University of Wisconsin-Madison Department of Computer Sciences. http:\/\/digital.library.wisc.edu\/1793\/60660"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2018.07.010"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2021.102710"},{"key":"e_1_3_2_38_2","first-page":"5998","volume-title":"Proceedings of NeurIPS","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS. Curran Associates, Inc., Long Beach, CA, 5998\u20136008. https:\/\/proceedings.neurips.cc\/paper\/2017\/file\/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3316809"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.196"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-2067"}],"container-title":["Journal of Data and Information Quality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3546916","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3546916","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:41Z","timestamp":1750186841000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3546916"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,28]]},"references-count":40,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,3,31]]}},"alternative-id":["10.1145\/3546916"],"URL":"https:\/\/doi.org\/10.1145\/3546916","relation":{},"ISSN":["1936-1955","1936-1963"],"issn-type":[{"type":"print","value":"1936-1955"},{"type":"electronic","value":"1936-1963"}],"subject":[],"published":{"date-parts":[[2022,12,28]]},"assertion":[{"value":"2021-11-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-05-25","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}