{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,19]],"date-time":"2026-09-19T06:19:28Z","timestamp":1789798768784,"version":"4.0.1"},"reference-count":54,"publisher":"Emerald","issue":"7","license":[{"start":{"date-parts":[[2024,9,26]],"date-time":"2024-09-26T00:00:00Z","timestamp":1727308800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["OIR"],"published-print":{"date-parts":[[2024,11,26]]},"abstract":"<jats:sec>\n                    <jats:title content-type=\"abstract-subheading\">Purpose<\/jats:title>\n                    <jats:p>This study aims to evaluate the performance of LLMs with various prompt engineering strategies in the context of health fact-checking.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title content-type=\"abstract-subheading\">Design\/methodology\/approach<\/jats:title>\n                    <jats:p>Inspired by Dual Process Theory, we introduce two kinds of prompts: Conclusion-first (System 1) and Explanation-first (System 2), and their respective retrieval-augmented variations. We evaluate the performance of these prompts across accuracy, argument elements, common errors and cost-effectiveness. Our study, conducted on two public health fact-checking datasets, categorized 10,212 claims as knowledge, anecdotes and news. To further analyze the reasoning process of LLM, we delve into the argument elements of health fact-checking generated by different prompts, revealing their tendencies in using evidence and contextual qualifiers. We conducted content analysis to identify and compare the common errors across various prompts.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title content-type=\"abstract-subheading\">Findings<\/jats:title>\n                    <jats:p>Results indicate that the Conclusion-first prompt performs well in knowledge (89.70%,66.09%), anecdote (79.49%,79.99%) and news (85.61%,85.95%) claims even without retrieval augmentation, proving to be cost-effective. In contrast, the Explanation-first prompt often classifies claims as unknown. However, it significantly boosts accuracy for news claims (87.53%,88.60%) and anecdote claims (87.28%,90.62%) with retrieval augmentation. The Explanation-first prompt is more focused on context specificity and user intent understanding during health fact-checking, showing high potential with retrieval augmentation. Additionally, retrieval-augmented LLMs concentrate more on evidence and context, highlighting the importance of the relevance and safety of retrieved content.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title content-type=\"abstract-subheading\">Originality\/value<\/jats:title>\n                    <jats:p>This study offers insights into how a balanced integration could enhance the overall performance of LLMs in critical applications, paving the way for future research on optimizing LLMs for complex cognitive tasks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title content-type=\"abstract-subheading\">Peer review<\/jats:title>\n                    <jats:p>\n                      The peer review history for this article is available at:\n                      <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/publons.com\/publon\/10.1108\/OIR-02-2024-0111\">https:\/\/publons.com\/publon\/10.1108\/OIR-02-2024-0111<\/jats:ext-link>\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.1108\/oir-02-2024-0111","type":"journal-article","created":{"date-parts":[[2024,9,25]],"date-time":"2024-09-25T23:34:24Z","timestamp":1727307264000},"page":"1488-1514","source":"Crossref","is-referenced-by-count":11,"title":["Scientific evidence and specific context: leveraging large language models for health fact-checking"],"prefix":"10.1108","volume":"48","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1422-2168","authenticated-orcid":false,"given":"Zhenni","family":"Ni","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3803-2505","authenticated-orcid":false,"given":"Yuxing","family":"Qian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-7597-1017","authenticated-orcid":false,"given":"Shuaipu","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4445-7494","authenticated-orcid":false,"given":"Marie-Christine","family":"Jaulent","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cedric","family":"Bousquet","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"140","published-online":{"date-parts":[[2024,9,26]]},"reference":[{"key":"key2024121608580898300_ref001","article-title":"Palm 2 technical report","year":"2023"},{"issue":"8","key":"key2024121608580898300_ref002","doi-asserted-by":"publisher","first-page":"1608","DOI":"10.1037\/xge0000729","article-title":"Fake news, fast and slow: deliberation reduces belief in false (but not true) news headlines","volume":"149","year":"2020","journal-title":"Journal of Experimental Psychology: General"},{"issue":"4","key":"key2024121608580898300_ref003","doi-asserted-by":"publisher","DOI":"10.1177\/10597123231206604","article-title":"Dual process theory for large Language Models: an overview of using psychology to address hallucination and reliability issues","volume":"32","year":"2023","journal-title":"Adaptive Behavior"},{"key":"key2024121608580898300_ref004","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.04812","article-title":"Influence of external information on large language models mirrors social cognitive patterns","year":"2023","journal-title":"arXiv"},{"issue":"5","key":"key2024121608580898300_ref005","doi-asserted-by":"publisher","first-page":"868","DOI":"10.1007\/s10439-023-03172-7","article-title":"Role of chat GPT in public health","volume":"51","year":"2023","journal-title":"Annals of Biomedical Engineering"},{"key":"key2024121608580898300_ref006","doi-asserted-by":"publisher","first-page":"15042","DOI":"10.1609\/aaai.v35i17.17765","article-title":"Thinking fast and slow in AI","year":"2021"},{"key":"key2024121608580898300_ref008","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.01431","article-title":"Benchmarking large language models in retrieval-augmented generation","year":"2023","journal-title":"arXiv"},{"key":"key2024121608580898300_ref009","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.00240","article-title":"Factllama: optimizing instruction-following language models with external knowledge for automated fact-checking","year":"2023","journal-title":"arXiv"},{"key":"key2024121608580898300_ref010","doi-asserted-by":"publisher","first-page":"492","DOI":"10.1145\/3394486.3403092","article-title":"DETERRENT: knowledge guided graph attention network for detecting healthcare misinformation","year":"2020"},{"key":"key2024121608580898300_ref011","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2403.03627","article-title":"Multimodal large language models to support real-world fact-checking","year":"2024","journal-title":"arXiv"},{"key":"key2024121608580898300_ref012","volume-title":"Custom Search JSON API: Introduction | Programmable Search Engine","author":"Google","year":"2023"},{"key":"key2024121608580898300_ref013","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2203.05794","article-title":"BERTopic: neural topic modeling with a class-based TF-IDF procedure","year":"2022","journal-title":"arXiv"},{"key":"key2024121608580898300_ref014","article-title":"Using ChatGPT to fight misinformation: ChatGPT nails 72% of 12,000 verified claims","year":"2023","journal-title":"PsyArXiv"},{"key":"key2024121608580898300_ref015","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.12247","article-title":"Bad actor, good advisor: exploring the role of large language models in fake news detection","year":"2023","journal-title":"arXiv"},{"key":"key2024121608580898300_ref016","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.05177","article-title":"Do large language models know about facts?","year":"2023","journal-title":"arXiv"},{"key":"key2024121608580898300_ref017","article-title":"Towards reasoning in large language models: a survey","year":"2023","journal-title":"arXiv"},{"issue":"12","key":"key2024121608580898300_ref018","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3571730","article-title":"Survey of hallucination in natural language generation","volume":"55","year":"2023","journal-title":"ACM Computing Surveys"},{"key":"key2024121608580898300_ref019","unstructured":"Jurafsky, D. and Martin, J.H. (2021), \u201cN-Gram Language Models\u201d, in Speech and Language Processing, 3rd ed., available at: https:\/\/web.stanford.edu\/\u223cjurafsky\/slp3\/"},{"key":"key2024121608580898300_ref020","volume-title":"Thinking, Fast and Slow","year":"2011"},{"key":"key2024121608580898300_ref021","doi-asserted-by":"publisher","volume-title":"Reason and Inquiry: The Erotetic Theory","year":"2022","DOI":"10.1093\/oso\/9780198823766.001.0001"},{"key":"key2024121608580898300_ref022","article-title":"Humans in humans out: on GPT converging toward common sense in both success and failure","year":"2023","journal-title":"arXiv"},{"key":"key2024121608580898300_ref023","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2011.03870","article-title":"Explainable automated fact-checking: a survey","year":"2020","journal-title":"arXiv"},{"key":"key2024121608580898300_ref024","doi-asserted-by":"publisher","first-page":"7740","DOI":"10.18653\/v1\/2020.emnlp-main.623","article-title":"Explainable automated fact-checking for public health claims","year":"2020"},{"key":"key2024121608580898300_ref025","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3492855","article-title":"HC-COVID: a hierarchical crowdsource knowledge graph approach to explainable COVID-19 misinformation detection","volume":"6","year":"2022","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"key":"key2024121608580898300_ref026","unstructured":"LangChain (n.d.), available at: https:\/\/www.langchain.com\/ (accessed 4 September 2024)."},{"key":"key2024121608580898300_ref027","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.14623","article-title":"Self-checker: plug-and-play modules for fact-checking with large language models","year":"2023","journal-title":"arXiv"},{"key":"key2024121608580898300_ref028","article-title":"When not to trust language models: investigating effectiveness of parametric and non-parametric memories","year":"2023","journal-title":"arXiv"},{"key":"key2024121608580898300_ref029","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.08896","article-title":"Selfcheckgpt: zero-resource black-box hallucination detection for generative large language models","year":"2023","journal-title":"arXiv"},{"key":"key2024121608580898300_ref007","article-title":"Language models are few-shot learners","year":"2020","journal-title":"arXiv preprint arXiv:2005.14165"},{"issue":"1","key":"key2024121608580898300_ref030","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1186\/s41235-020-00252-3","article-title":"Reliance on emotion promotes belief in fake news","volume":"5","year":"2020","journal-title":"Cognitive Research: Principles and Implications"},{"issue":"3","key":"key2024121608580898300_ref031","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/bm.2012.031","article-title":"Interrater reliability: the kappa statistic","volume":"22","year":"2012","journal-title":"Biochemia Medica"},{"key":"key2024121608580898300_ref032","article-title":"Neemakot\/health-fact-checking: dataset and code for \u201cexplainable automated fact-checking for public health claims\u201d from EMNLP 2020","author":"Neemakot","year":"2020","journal-title":"GitHub"},{"key":"key2024121608580898300_ref033","article-title":"Capabilities of gpt-4 on medical challenge problems","year":"2023","journal-title":"arXiv Preprint"},{"key":"key2024121608580898300_ref034","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2303.08774","article-title":"GPT-4 technical report","author":"OpenAI","year":"2023","journal-title":"arXiv"},{"key":"key2024121608580898300_ref035","unstructured":"OpenAI Pricing (n.d.), available at: https:\/\/openai.com\/pricing (accessed 5 December 2023)."},{"key":"key2024121608580898300_ref036","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.12744","article-title":"Fact-checking complex claims with program-guided reasoning","year":"2023","journal-title":"arXiv"},{"issue":"6","key":"key2024121608580898300_ref037","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2023.103510","article-title":"What is the limitation of multimodal LLMs? A deeper look into multimodal LLMs through prompt probing","volume":"60","year":"2023","journal-title":"Information Processing and Management"},{"key":"key2024121608580898300_ref038","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.13549","article-title":"The perils and promises of fact-checking with large language models","year":"2023","journal-title":"arXiv"},{"issue":"2","key":"key2024121608580898300_ref039","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1017\/S1930297500008640","article-title":"Beyond \u2018fake news\u2019: analytic thinking and the detection of false and hyperpartisan news headlines","volume":"16","year":"2021","journal-title":"Judgment and Decision Making"},{"key":"key2024121608580898300_ref040","first-page":"31210","article-title":"Large Language models can Be easily distracted by irrelevant context","year":"2023"},{"key":"key2024121608580898300_ref041","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.12558","article-title":"Large language models help humans verify truthfulness\u2014except when they are convincingly wrong","year":"2023","journal-title":"arXiv"},{"issue":"3","key":"key2024121608580898300_ref042","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1080\/00461520.2020.1730181","article-title":"Evaluating sources of scientific evidence and claims in the post-truth era may require reappraising plausibility judgments","volume":"55","year":"2020","journal-title":"Educational Psychologist"},{"key":"key2024121608580898300_ref043","doi-asserted-by":"publisher","first-page":"2949","DOI":"10.1145\/3477495.3531726","article-title":"Monant medical misinformation dataset: mapping articles to fact-checked claims","year":"2022"},{"issue":"8","key":"key2024121608580898300_ref044","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","year":"2023","journal-title":"Nature Medicine"},{"key":"key2024121608580898300_ref045","volume-title":"The Uses of Argument","year":"2003"},{"issue":"12","key":"key2024121608580898300_ref046","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41591-023-02594-z","article-title":"Large language models should be used as scientific reasoning engines, not knowledge databases","volume":"29","year":"2023","journal-title":"Nature Medicine"},{"issue":"7947","key":"key2024121608580898300_ref047","doi-asserted-by":"publisher","first-page":"224","DOI":"10.1038\/d41586-023-00288-7","article-title":"ChatGPT: five priorities for research","volume":"614","year":"2023","journal-title":"Nature"},{"key":"key2024121608580898300_ref048","article-title":"Explainable claim verification via knowledge-grounded reasoning with large language models","year":"2023","journal-title":"arXiv"},{"key":"key2024121608580898300_ref049","article-title":"Self-Consistency Improves Chain of Thought Reasoning in Language Models","year":"2023","journal-title":"arXiv"},{"key":"key2024121608580898300_ref050","first-page":"24824","article-title":"Chain-of-Thought prompting elicits reasoning in large Language Models","volume":"35","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"key":"key2024121608580898300_ref051","volume-title":"A Guide to Good Reasoning: Cultivating Intellectual Virtues","year":"1999","edition":"2nd ed."},{"key":"key2024121608580898300_ref052","doi-asserted-by":"publisher","DOI":"10.1101\/2023.04.18.23288752","article-title":"Faithful AI in medicine: a systematic review with large Language Models and beyond","year":"2023","journal-title":"medRxiv"},{"key":"key2024121608580898300_ref053","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.10601","article-title":"Tree of thoughts: deliberate problem solving with large language models","year":"2023","journal-title":"arXiv"},{"key":"key2024121608580898300_ref054","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2304.03728","article-title":"Interpretable unified language checking","year":"2023","journal-title":"arXiv"}],"container-title":["Online Information Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/OIR-02-2024-0111\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/OIR-02-2024-0111\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T22:42:21Z","timestamp":1753396941000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/oir\/article\/48\/7\/1488-1514\/1222112"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,26]]},"references-count":54,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2024,9,26]]},"published-print":{"date-parts":[[2024,11,26]]}},"alternative-id":["10.1108\/OIR-02-2024-0111"],"URL":"https:\/\/doi.org\/10.1108\/oir-02-2024-0111","relation":{"has-review":[{"id-type":"doi","id":"10.1108\/OIR-02-2024-0111\/v3\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-02-2024-0111\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-02-2024-0111\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-02-2024-0111\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-02-2024-0111\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-02-2024-0111\/v2\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-02-2024-0111\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-02-2024-0111\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-02-2024-0111\/v3\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-02-2024-0111\/v3\/decision1","asserted-by":"object"}]},"ISSN":["1468-4527"],"issn-type":[{"value":"1468-4527","type":"print"}],"subject":[],"published":{"date-parts":[[2024,9,26]]}}}