{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T21:34:31Z","timestamp":1790285671102,"version":"4.1.0"},"reference-count":45,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2021,4,15]],"date-time":"2021-04-15T00:00:00Z","timestamp":1618444800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Open Society Foundation","award":["OR2017-35395"],"award-info":[{"award-number":["OR2017-35395"]}]},{"name":"Omidyar Network"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Digital Threats"],"published-print":{"date-parts":[[2021,6,30]]},"abstract":"<jats:p>In an effort to assist factcheckers in the process of factchecking, we tackle the claim detection task, one of the necessary stages prior to determining the veracity of a claim. It consists of identifying the set of sentences, out of a long text, deemed capable of being factchecked. This article is a collaborative work between Full Fact, an independent factchecking charity, and academic partners. Leveraging the expertise of professional factcheckers, we develop an annotation schema and a benchmark for automated claim detection that is more consistent across time, topics, and annotators than are previous approaches. Our annotation schema has been used to crowdsource the annotation of a dataset with sentences from UK political TV shows. We introduce an approach based on universal sentence representations to perform the classification, achieving an F1 score of 0.83, with over 5% relative improvement over the state-of-the-art methods ClaimBuster and ClaimRank. The system was deployed in production and received positive user feedback.<\/jats:p>","DOI":"10.1145\/3412869","type":"journal-article","created":{"date-parts":[[2021,4,15]],"date-time":"2021-04-15T16:06:21Z","timestamp":1618502781000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":71,"title":["Toward Automated Factchecking"],"prefix":"10.1145","volume":"2","author":[{"given":"Lev","family":"Konstantinovskiy","sequence":"first","affiliation":[{"name":"Full Fact, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oliver","family":"Price","sequence":"additional","affiliation":[{"name":"University of Warwick, Coventry, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mevan","family":"Babakar","sequence":"additional","affiliation":[{"name":"Full Fact, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arkaitz","family":"Zubiaga","sequence":"additional","affiliation":[{"name":"Queen Mary University of London, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,4,15]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"ACM. 2018. ACM Code of Ethics. Retrieved from https:\/\/www.acm.org\/code-of-ethics.  ACM. 2018. ACM Code of Ethics. Retrieved from https:\/\/www.acm.org\/code-of-ethics."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2019.02.016"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201918)","author":"Barr\u00f3n-Cedeno Alberto","year":"2018","unstructured":"Alberto Barr\u00f3n-Cedeno , Tamer Elsayed , Reem Suwaileh , Llu\u00eds M\u00e0rquez , Pepa Atanasova , Wajdi Zaghouani , Spas Kyuchukov , Giovanni Da San Martino , and Preslav Nakov . 2018 . Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims. Task 2: Factuality . In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201918) . Alberto Barr\u00f3n-Cedeno, Tamer Elsayed, Reem Suwaileh, Llu\u00eds M\u00e0rquez, Pepa Atanasova, Wajdi Zaghouani, Spas Kyuchukov, Giovanni Da San Martino, and Preslav Nakov. 2018. Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims. Task 2: Factuality. In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201918)."},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the World Wide Web Conference. International World Wide Web Conferences Steering Committee, 1795--1804","author":"Cao Yixuan","year":"2018","unstructured":"Yixuan Cao , Hongwei Li , Ping Luo , and Jiaquan Yao . 2018 . Toward automatic numerical cross-checking: Extracting formulas from text . In Proceedings of the World Wide Web Conference. International World Wide Web Conferences Steering Committee, 1795--1804 . Yixuan Cao, Hongwei Li, Ping Luo, and Jiaquan Yao. 2018. Toward automatic numerical cross-checking: Extracting formulas from text. In Proceedings of the World Wide Web Conference. International World Wide Web Conferences Steering Committee, 1795--1804."},{"key":"e_1_2_1_6_1","volume-title":"Positive Learning in the Age of Information","author":"Ciampaglia Giovanni Luca","unstructured":"Giovanni Luca Ciampaglia . 2018. The digital misinformation pipeline . In Positive Learning in the Age of Information . Springer , 413--421. Giovanni Luca Ciampaglia. 2018. The digital misinformation pipeline. In Positive Learning in the Age of Information. Springer, 413--421."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0128193"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1070"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1111\/pops.12394"},{"key":"e_1_2_1_10_1","unstructured":"Diane Francis and Mevan Babakar. 2016. HeroX Factchecking challenge. Retrieved from https:\/\/www.herox.com\/factcheck\/5-practise-claims.  Diane Francis and Mevan Babakar. 2016. HeroX Factchecking challenge. Retrieved from https:\/\/www.herox.com\/factcheck\/5-practise-claims."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1080\/10584609.2014.914613"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the International Conference on Recent Advances in Natural Language Processing. INCOMA Ltd., 267--276","author":"Gencheva Pepa","year":"2017","unstructured":"Pepa Gencheva , Preslav Nakov , Llu\u00eds M\u00e0rquez , Alberto Barr\u00f3n-Cede\u00f1o , and Ivan Koychev . 2017 . A context-aware approach for detecting worth-checking claims in political debates . In Proceedings of the International Conference on Recent Advances in Natural Language Processing. INCOMA Ltd., 267--276 . Pepa Gencheva, Preslav Nakov, Llu\u00eds M\u00e0rquez, Alberto Barr\u00f3n-Cede\u00f1o, and Ivan Koychev. 2017. A context-aware approach for detecting worth-checking claims in political debates. In Proceedings of the International Conference on Recent Advances in Natural Language Processing. INCOMA Ltd., 267--276."},{"key":"e_1_2_1_13_1","volume-title":"FACTSHEET: Understanding the Promise and Limits of Automated Fact-Checking. Technical Report","author":"Graves Lucas","year":"2018","unstructured":"Lucas Graves . 2018 . FACTSHEET: Understanding the Promise and Limits of Automated Fact-Checking. Technical Report . Reuters Institute for the Study of Journalism, University of Oxford, Oxford, UK. Lucas Graves. 2018. FACTSHEET: Understanding the Promise and Limits of Automated Fact-Checking. Technical Report. Reuters Institute for the Study of Journalism, University of Oxford, Oxford, UK."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1111\/jcom.12198"},{"key":"e_1_2_1_15_1","unstructured":"Vigdis Hanto and Mats Tostrup. 2018. Toward Automated Fake News Classification\u2014On Building Collections for Claim Analysis Research. Master\u2019s thesis. NTNU.  Vigdis Hanto and Mats Tostrup. 2018. Toward Automated Fake News Classification\u2014On Building Collections for Claim Analysis Research. Master\u2019s thesis. NTNU."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098131"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806416.2806652"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137765.3137815"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_2_1_20_1","volume-title":"Companion of the Web Conference 2018. International World Wide Web Conferences Steering Committee, 1595--1598","author":"Huynh Viet-Phi","year":"2018","unstructured":"Viet-Phi Huynh and Paolo Papotti . 2018 . Toward a benchmark for fact checking with knowledge bases . In Companion of the Web Conference 2018. International World Wide Web Conferences Steering Committee, 1595--1598 . Viet-Phi Huynh and Paolo Papotti. 2018. Toward a benchmark for fact checking with knowledge bases. In Companion of the Web Conference 2018. International World Wide Web Conferences Steering Committee, 1595--1598."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.","author":"Jaradat Israa","year":"2018","unstructured":"Israa Jaradat , Pepa Gencheva , Alberto Barr\u00f3n-Cede\u00f1o , Llu\u00eds M\u00e0rquez , and Preslav Nakov . 2018 . ClaimRank: Detecting check-worthy claims in Arabic and English . In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Israa Jaradat, Pepa Gencheva, Alberto Barr\u00f3n-Cede\u00f1o, Llu\u00eds M\u00e0rquez, and Preslav Nakov. 2018. ClaimRank: Detecting check-worthy claims in Arabic and English. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies."},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence","author":"Kohavi Ron","year":"1995","unstructured":"Ron Kohavi . 1995 . A study of cross-validation and bootstrap for accuracy estimation and model selection . In Proceedings of the International Joint Conference on Artificial Intelligence . Montreal, Canada, 1137--1145. Ron Kohavi. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the International Joint Conference on Artificial Intelligence. Montreal, Canada, 1137--1145."},{"key":"e_1_2_1_23_1","unstructured":"Klaus Krippendorff. 1980. Reliability. Wiley Online Library.  Klaus Krippendorff. 1980. Reliability. Wiley Online Library."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0168344"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the 25th International Conference on Computational Linguistics: Technical Papers (COLING\u201914)","author":"Levy Ran","year":"2014","unstructured":"Ran Levy , Yonatan Bilu , Daniel Hershcovich , Ehud Aharoni , and Noam Slonim . 2014 . Context dependent claim detection . In Proceedings of the 25th International Conference on Computational Linguistics: Technical Papers (COLING\u201914) . 1489--1500. Ran Levy, Yonatan Bilu, Daniel Hershcovich, Ehud Aharoni, and Noam Slonim. 2014. Context dependent claim detection. In Proceedings of the 25th International Conference on Computational Linguistics: Technical Papers (COLING\u201914). 1489--1500."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 8th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)","volume":"2","author":"Long Yunfei","year":"2017","unstructured":"Yunfei Long , Qin Lu , Rong Xiang , Minglei Li , and Chu-Ren Huang . 2017 . Fake news detection through multi-perspective speaker profiles . In Proceedings of the 8th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , Vol. 2 . 252--256. Yunfei Long, Qin Lu, Rong Xiang, Minglei Li, and Chu-Ren Huang. 2017. Fake news detection through multi-perspective speaker profiles. In Proceedings of the 8th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Vol. 2. 252--256."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806416.2806607"},{"key":"e_1_2_1_28_1","unstructured":"Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. Retrieved from https:\/\/arXiv:1301.3781.  Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. Retrieved from https:\/\/arXiv:1301.3781."},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201918)","author":"Nakov Preslav","year":"2018","unstructured":"Preslav Nakov , Alberto Barr\u00f3n-Cede\u00f1o , Tamer Elsayed , Reem Suwaileh , Llu\u00eds M\u00e0rquez , Wajdi Zaghouani , Pepa Gencheva , Spas Kyuchukov , and Giovanni Da San Martino . 2018 . CLEF-2018 lab on automatic identification and verification of claims in political debates . In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201918) . Preslav Nakov, Alberto Barr\u00f3n-Cede\u00f1o, Tamer Elsayed, Reem Suwaileh, Llu\u00eds M\u00e0rquez, Wajdi Zaghouani, Pepa Gencheva, Spas Kyuchukov, and Giovanni Da San Martino. 2018. CLEF-2018 lab on automatic identification and verification of claims in political debates. In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201918)."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 1st Workshop on Argumentation Mining. 29--38","author":"Park Joonsuk","year":"2014","unstructured":"Joonsuk Park and Claire Cardie . 2014 . Identifying appropriate support for propositions in online user comments . In Proceedings of the 1st Workshop on Argumentation Mining. 29--38 . Joonsuk Park and Claire Cardie. 2014. Identifying appropriate support for propositions in online user comments. In Proceedings of the 1st Workshop on Argumentation Mining. 29--38."},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the ACM Conference on Information and Knowledge Management. ACM, 2259--2262","author":"Patwari Ayush","year":"2017","unstructured":"Ayush Patwari , Dan Goldwasser , and Saurabh Bagchi . 2017 . TATHYA: A multi-classifier system for detecting check-worthy statements in political debates . In Proceedings of the ACM Conference on Information and Knowledge Management. ACM, 2259--2262 . Ayush Patwari, Dan Goldwasser, and Saurabh Bagchi. 2017. TATHYA: A multi-classifier system for detecting check-worthy statements in political debates. In Proceedings of the ACM Conference on Information and Knowledge Management. ACM, 2259--2262."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the LREC Workshop on New Challenges for NLP Frameworks. ELRA, Valletta, Malta, 45--50","author":"\u0158eh\u016f\u0159ek Radim","year":"2010","unstructured":"Radim \u0158eh\u016f\u0159ek and Petr Sojka . 2010 . Software framework for topic modelling with large corpora . In Proceedings of the LREC Workshop on New Challenges for NLP Frameworks. ELRA, Valletta, Malta, 45--50 . Retrieved from http:\/\/is.muni.cz\/publication\/884893\/en. Radim \u0158eh\u016f\u0159ek and Petr Sojka. 2010. Software framework for topic modelling with large corpora. In Proceedings of the LREC Workshop on New Challenges for NLP Frameworks. ELRA, Valletta, Malta, 45--50. Retrieved from http:\/\/is.muni.cz\/publication\/884893\/en."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2016.04.015"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3137597.3137600"},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics. 3346--3359","author":"Thorne James","year":"2018","unstructured":"James Thorne and Andreas Vlachos . 2018 . Automated fact checking: Task formulations, methods and future directions . In Proceedings of the 27th International Conference on Computational Linguistics. 3346--3359 . James Thorne and Andreas Vlachos. 2018. Automated fact checking: Task formulations, methods and future directions. In Proceedings of the 27th International Conference on Computational Linguistics. 3346--3359."},{"key":"e_1_2_1_39_1","volume-title":"FEVER: A large-scale dataset for fact extraction and verification.","author":"Thorne James","year":"2018","unstructured":"James Thorne , Andreas Vlachos , Christos Christodoulopoulos , and Arpit Mittal . 2018 . FEVER: A large-scale dataset for fact extraction and verification. Retrieved from https:\/\/arXiv:1803.05355. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A large-scale dataset for fact extraction and verification. Retrieved from https:\/\/arXiv:1803.05355."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-2508"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","volume":"2","author":"Wang William Yang","year":"2017","unstructured":"William Yang Wang . 2017 . \u201c Liar, liar pants on fire\u201d: A new benchmark dataset for fake news detection . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics , Vol. 2 . 422--426. William Yang Wang. 2017. \u201cLiar, liar pants on fire\u201d: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vol. 2. 422--426."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732286.2732295"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2350190.2350203"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2019.03.004"},{"key":"e_1_2_1_45_1","unstructured":"Arkaitz Zubiaga. 2018. Learning class-specific word representations for early detection of hoaxes in social media. Retrieved from https:\/\/arXiv:1801.07311.  Arkaitz Zubiaga. 2018. Learning class-specific word representations for early detection of hoaxes in social media. Retrieved from https:\/\/arXiv:1801.07311."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3161603"},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201918)","volume":"8995","author":"Zuo Chaoyuan","year":"2018","unstructured":"Chaoyuan Zuo , Ayla Ida Karakas , and Ritwik Banerjee . 2018 . A hybrid recognition system for check-worthy clams using heuristics and supervised learning . In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201918) , Vol. 8995 . CEUR-WS.org, 171--175. Chaoyuan Zuo, Ayla Ida Karakas, and Ritwik Banerjee. 2018. A hybrid recognition system for check-worthy clams using heuristics and supervised learning. In Proceedings of the Conference and Labs of the Evaluation Forum (CLEF\u201918), Vol. 8995. CEUR-WS.org, 171--175."}],"container-title":["Digital Threats: Research and Practice"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3412869","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3412869","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:02Z","timestamp":1750193222000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3412869"}},"subtitle":["Developing an Annotation Schema and Benchmark for Consistent Automated Claim Detection"],"short-title":[],"issued":{"date-parts":[[2021,4,15]]},"references-count":45,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,6,30]]}},"alternative-id":["10.1145\/3412869"],"URL":"https:\/\/doi.org\/10.1145\/3412869","relation":{},"ISSN":["2692-1626","2576-5337"],"issn-type":[{"value":"2692-1626","type":"print"},{"value":"2576-5337","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,15]]},"assertion":[{"value":"2019-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-04-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}