{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T00:31:50Z","timestamp":1760661110396,"version":"build-2065373602"},"reference-count":96,"publisher":"Association for Computing Machinery (ACM)","issue":"7","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,10,18]]},"abstract":"<jats:p>\n            Given the volume of potentially false claims online, claim prioritization is essential in allocating limited human resources available for fact-checking. In this study, we perceive claim prioritization as an information retrieval (IR) task: just as multidimensional IR relevance, with many factors influencing which search results a user deems relevant, checkworthiness is also multi-faceted, subjective, and even personal, with many factors influencing how fact-checkers triage and select which claims to check. Our study investigates both the multidimensional nature of checkworthiness and effective tool support to assist fact-checkers in claim prioritization. Methodologically, we pursue\n            <jats:italic toggle=\"yes\">Research through Design<\/jats:italic>\n            combined with mixed-method evaluation.\n          <\/jats:p>\n          <jats:p>Specifically, we develop an AI-assisted claim prioritization prototype as a probe to explore how fact-checkers use multidimensional checkworthy factors to prioritize claims, simultaneously probing fact-checker needs and exploring the design space to meet those needs. With 16 professional fact-checkers participating in our study, we uncover a hierarchical prioritization strategy fact-checkers implicitly use, revealing an underexplored aspect of their workflow, with actionable design recommendations for improving claim triage across multidimensional checkworthiness and tailoring this process with LLM integration.<\/jats:p>","DOI":"10.1145\/3757473","type":"journal-article","created":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:32:00Z","timestamp":1760635920000},"page":"1-49","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Exploring Multidimensional Checkworthiness: Designing AI-assisted Claim Prioritization for Human Fact-checkers"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0983-6202","authenticated-orcid":false,"given":"Houjiang","family":"Liu","sequence":"first","affiliation":[{"name":"School of Information, University of Texas at Austin, Austin, TX, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2273-3996","authenticated-orcid":false,"given":"Jacek","family":"Gwizdka","sequence":"additional","affiliation":[{"name":"School of Information, University of Texas at Austin, Austin, TX, USA and Institute of Applied Computer Science, \u0141\u00f3d\u017a University of Technology, \u0141\u00f3d\u017a, Poland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0056-2834","authenticated-orcid":false,"given":"Matthew","family":"Lease","sequence":"additional","affiliation":[{"name":"School of Information, University of Texas at Austin, Austin, TX, USA"}]}],"member":"320","published-online":{"date-parts":[[2025,10,16]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"2024. Generative AI search trends & transformations. https:\/\/www.thinkwithgoogle.com\/marketing-strategies\/search\/ generative-ai-search-trends-and-transformations\/. Accessed: 2024-7-1."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.56"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-61841-4_1"},{"key":"e_1_2_2_4_1","unstructured":"Mike Ananny. 2018. The partnership press: Lessons for platform-publisher collaborations as Facebook and news outlets team to fight misinformation. Technical Report. https:\/\/doi.org\/10.7916\/D85B1JG9"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3272973.3272974"},{"key":"e_1_2_2_6_1","unstructured":"Phoebe Arnold. 2020. The challenges of online fact checking. Technical Report. Full Fact."},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSS.2021.3096038"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-56069-9_62"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-42448-9_20"},{"key":"e_1_2_2_10_1","volume-title":"A Arif, and K Starbird.","author":"Beers A","year":"2020","unstructured":"A Beers, M M C Haughey, A Arif, and K Starbird. 2020. Examining the digital toolsets of journalists reporting on disinformation. cj2020.northeastern.edu."},{"key":"e_1_2_2_11_1","article-title":"Knowledge Work in Platform Fact-Checking Partnerships","volume":"17","author":"B\u00e9lair-Gagnon Val\u00e9rie","year":"2023","unstructured":"Val\u00e9rie B\u00e9lair-Gagnon, Rebekah Larsen, Lucas Graves, and Oscar Westlund. 2023. Knowledge Work in Platform Fact-Checking Partnerships. International Journal of Communication Systems 17, 0 (29 Jan. 2023), 21.","journal-title":"International Journal of Communication Systems"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2317956.2317968"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300234"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1255175.1255267"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.229"},{"key":"e_1_2_2_17_1","article-title":"Why Do Fact-Checking Organizations Go Beyond Fact-Checking? A Leap Toward Media and Information Literacy Education","volume":"16","author":"\u00c7\u00f6mlek\u00e7i Mehmet Fatih","year":"2022","unstructured":"Mehmet Fatih \u00c7\u00f6mlek\u00e7i. 2022. Why Do Fact-Checking Organizations Go Beyond Fact-Checking? A Leap Toward Media and Information Literacy Education. International Journal of Communication Systems 16, 0 (25 Sept. 2022), 21.","journal-title":"International Journal of Communication Systems"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm"},{"key":"e_1_2_2_19_1","unstructured":"Design Council. 2015. Framework for Innovation: Design Council's evolved Double Diamond."},{"key":"e_1_2_2_20_1","article-title":"Automated fact-checking to support professional practices: systematic literature review and meta-analysis","volume":"17","author":"Dierickx Laurence","year":"2023","unstructured":"Laurence Dierickx, Carl-Gustav Lind\u00e9n, and Andreas Lothe Opdahl. 2023. Automated fact-checking to support professional practices: systematic literature review and meta-analysis. International Journal of Communication 17, 0 (15 Aug. 2023), 21-21.","journal-title":"International Journal of Communication"},{"key":"e_1_2_2_21_1","volume-title":"Unveiling and Manipulating Prompt Influence in Large Language Models. In The Twelfth International Conference on Learning Representations.","author":"Feng Zijian","year":"2023","unstructured":"Zijian Feng, Hanzhang Zhou, Zixiao Zhu, Junlang Qian, and Kezhi Mao. 2023. Unveiling and Manipulating Prompt Influence in Large Language Models. In The Twelfth International Conference on Learning Representations."},{"key":"e_1_2_2_22_1","volume-title":"DRS Biennial Conference Series.","author":"Frankel Lois","year":"2010","unstructured":"Lois Frankel and Martin Racine. 2010. The Complex Field of Research: for Design, through Design, and about Design. In DRS Biennial Conference Series."},{"key":"e_1_2_2_23_1","volume-title":"Research in art and design","author":"Frayling Christopher","year":"1993","unstructured":"Christopher Frayling. 1993. Research in art and design. Royal College of Art research papers 1 (1993), 1-5."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208538"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10439-023-03272-4"},{"key":"e_1_2_2_26_1","volume-title":"Deciding What's True: The Rise of Political Fact-Checking in American Journalism","author":"Graves Lucas","unstructured":"Lucas Graves. 2016. Deciding What's True: The Rise of Political Fact-Checking in American Journalism. Columbia University Press."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1111\/cccr.12163"},{"key":"e_1_2_2_28_1","unstructured":"Lucas Graves. 2018. Understanding the Promise and Limits of Automated Fact-Checking. Technical Report. Reuters Institute. 1-7 pages."},{"key":"e_1_2_2_29_1","volume-title":"Digital News Project Report","author":"Graves L","unstructured":"L Graves and F Cherubini. 2016. The Rise of Fact-Checking Sites in Europe. In Digital News Project Report. Reuters Institute for the Study of Journalism."},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-923x.12896"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3588319"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00454"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767877"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1177\/0093650218819671"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098131"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/302979.303030"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.intcom.2006.10.001"},{"key":"e_1_2_2_38_1","volume-title":"Lexicon of lies: terms for problematic information","author":"Jack C","year":"2017","unstructured":"C Jack. 2017. Lexicon of lies: terms for problematic information. Data & Society 3, 22 (9 Aug. 2017), 1094-1096."},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517649"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609633"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080840"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3555143"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240167.3240170"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3412869"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-09316-6_10"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3686962"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1080\/21670811.2023.2193344"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/p19-3026"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1121949.1121979"},{"key":"e_1_2_2_50_1","volume-title":"Maslow's hierarchy of needs. Simply psychology 1, 1-18","author":"McLeod Saul","year":"2007","unstructured":"Saul McLeod. 2007. Maslow's hierarchy of needs. Simply psychology 1, 1-18 (2007)."},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1080\/17512786.2018.1547655"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3512974"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3605943"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2021\/"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533205"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1006\/cogp.2000.0746"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000025"},{"key":"e_1_2_2_58_1","volume-title":"UNDR: User-Needs-Driven Ranking of Products in E-Commerce. (13","author":"Papenmeier Andrea","year":"2023","unstructured":"Andrea Papenmeier, Daniel Hienert, Firas Sabbah, Norbert Fuhr, and Dagmar Kern. 2023. UNDR: User-Needs-Driven Ranking of Products in E-Commerce. (13 Feb. 2023). arXiv:2302.06398 [cs.IR]"},{"key":"e_1_2_2_59_1","first-page":"1","article-title":"Information inflation: Can the legal system adapt? Annual survey","volume":"13","author":"Paul George L","year":"2006","unstructured":"George L Paul and Jason R Baron. 2006. Information inflation: Can the legal system adapt? Annual survey. Richmond Journal of Law and Technology 13, 3 (2006), 1-42.","journal-title":"Richmond Journal of Law and Technology"},{"key":"e_1_2_2_60_1","volume-title":"Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12","author":"Pedregosa Fabian","year":"2011","unstructured":"Fabian Pedregosa, Ga\u00ebl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, and Others. 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825-2830."},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2598510.2598525"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.destud.2021.101061"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.36190\/2023.28"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/d19-1410"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1504\/ijssci.2008.017590"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.37016\/mr-2020-81"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630260604"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20682"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.5555\/1315930.1315946"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463121"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(90)90050-C"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3641010"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.332"},{"key":"e_1_2_2_74_1","volume-title":"Junyi Jessy Li, and Matthew Lease","author":"Singh Prakhar","year":"2021","unstructured":"Prakhar Singh, Anubrata Das, Junyi Jessy Li, and Matthew Lease. 2021. The Case for Claim Difficulty Assessment in Automatic Fact Checking. (20 Sept. 2021). arXiv:2109.09689 [cs.CL]"},{"key":"e_1_2_2_75_1","volume-title":"The fact checker's bible","author":"Smith Sarah Harrison","unstructured":"Sarah Harrison Smith. 2003. The fact checker's bible. Anchor Books."},{"key":"e_1_2_2_76_1","first-page":"3","volume-title":"Proceedings of the ASIST Annual Meeting","volume":"35","author":"Spink Amanda","year":"1998","unstructured":"Amanda Spink, Howard Greisdorf, and Judy Bateman. 1998. Examining different regions of relevance: From highly relevant to not relevant. In Proceedings of the ASIST Annual Meeting, Vol. 35. 3-12."},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-publhealth-040119-094127"},{"key":"e_1_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2016.11.004"},{"key":"e_1_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(97)00081-2"},{"key":"e_1_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/3563657.3595997"},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1177\/14614448231213942"},{"key":"e_1_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.2217\/cer.13.17"},{"key":"e_1_2_2_83_1","volume-title":"Information disorder: Toward an interdisciplinary framework for research and policymaking","author":"Wardle Claire","unstructured":"Claire Wardle and Hossein Derakhshan. 2017. Information disorder: Toward an interdisciplinary framework for research and policymaking. Vol. 27. Council of Europe."},{"key":"e_1_2_2_84_1","volume-title":"Technologies and fact-checking. A sociotechnical mapping. Disinformations Studies: Perspectives from An Emerging Field","author":"Westlund Oscar","year":"2022","unstructured":"Oscar Westlund, Rebekah Larsen, Lucas Graves, Lasha Kavtaradze, and Steen Steensen. 2022. Technologies and fact-checking. A sociotechnical mapping. Disinformations Studies: Perspectives from An Emerging Field (2022)."},{"key":"e_1_2_2_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581068"},{"key":"e_1_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026702"},{"key":"e_1_2_2_87_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20361"},{"key":"e_1_2_2_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/642611.642681"},{"key":"e_1_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/3579536"},{"key":"e_1_2_2_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609577"},{"key":"e_1_2_2_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/3639372"},{"key":"e_1_2_2_92_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395046"},{"key":"e_1_2_2_93_1","doi-asserted-by":"publisher","DOI":"10.1115\/1.4067527"},{"key":"e_1_2_2_94_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4939-0378-8_8"},{"key":"e_1_2_2_95_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240624.1240704"},{"key":"e_1_2_2_96_1","doi-asserted-by":"publisher","DOI":"10.1145\/1858171.1858228"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3757473","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:43:21Z","timestamp":1760636601000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3757473"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,16]]},"references-count":96,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,10,18]]}},"alternative-id":["10.1145\/3757473"],"URL":"https:\/\/doi.org\/10.1145\/3757473","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,16]]},"assertion":[{"value":"2025-10-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}