{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T22:11:07Z","timestamp":1759961467245,"version":"3.40.3"},"publisher-location":"Cham","reference-count":47,"publisher":"Springer Nature Switzerland","isbn-type":[{"type":"print","value":"9783031478956"},{"type":"electronic","value":"9783031478963"}],"license":[{"start":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T00:00:00Z","timestamp":1672531200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T00:00:00Z","timestamp":1699920000000},"content-version":"vor","delay-in-days":317,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Large language models have enabled the rapid production of misleading or fake narratives, presenting a challenge for direct detection methods. Considering that generative artificial intelligence tools are likely to be used either to inform or to disinform, evaluating the (non)human nature of machine-generated content is questioned, especially regarding the \u2018hallucination\u2019 phenomenon, which relates to generated content that does not correspond to real-world input. In this study, we argue that assessing machine-generated content is most reliable when done by humans because doing so involves critical consideration of the meaning of the information and its informative, misinformative or disinformative value, which is related to the accuracy and reliability of the news. To explore human-based judgement methods, we developed the Information Disorder Level (IDL) index, a language-independent metric to evaluate the factuality of machine-generated content. It has been tested on a corpus of forty made-up and actual news stories generated with ChatGPT. For newsrooms using generative AI, results suggest that every piece of machine-generated content should be vetted and post-edited by humans before being published. From a digital media literacy perspective, the IDL index is a valuable tool to understand the limits of generative AI and trigger a reflection on what constitutes the factuality of a reported event.<\/jats:p>","DOI":"10.1007\/978-3-031-47896-3_5","type":"book-chapter","created":{"date-parts":[[2023,11,13]],"date-time":"2023-11-13T20:01:33Z","timestamp":1699905693000},"page":"60-71","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["The Information Disorder Level (IDL) Index: A Human-Based Metric to\u00a0Assess the\u00a0Factuality of\u00a0Machine-Generated Content"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5926-8720","authenticated-orcid":false,"given":"Laurence","family":"Dierickx","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7301-4401","authenticated-orcid":false,"given":"Carl-Gustav","family":"Lind\u00e9n","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3141-1385","authenticated-orcid":false,"given":"Andreas L.","family":"Opdahl","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,11,14]]},"reference":[{"key":"5_CR1","doi-asserted-by":"crossref","unstructured":"Giansiracusa, N.: How algorithms create and prevent fake news: Exploring the impacts of social media, deepfakes, GPT-3, and more. APress (2021)","DOI":"10.1007\/978-1-4842-7155-1"},{"key":"5_CR2","doi-asserted-by":"crossref","unstructured":"Ferrara, E.: Social bot detection in the age of ChatGPT: Challenges and opportunities. First Monday (2023)","DOI":"10.5210\/fm.v28i6.13185"},{"key":"5_CR3","unstructured":"Wardle, C., Derakhshan, H.: Information disorder: toward an interdisciplinary framework for research and policymaking. Council of Europe Strasbourg (2017)"},{"key":"5_CR4","doi-asserted-by":"publisher","first-page":"1166120","DOI":"10.3389\/fpubh.2023.1166120","volume":"11","author":"L De Angelis","year":"2023","unstructured":"De Angelis, L., et al.: ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front. Public Health 11, 1166120 (2023)","journal-title":"Front. Public Health"},{"key":"5_CR5","doi-asserted-by":"crossref","unstructured":"Ray, P.: ChatGPT: a comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems (2023)","DOI":"10.1016\/j.iotcps.2023.04.003"},{"key":"5_CR6","unstructured":"Weber-Wulff, D., et al.: Testing of detection tools for AI-generated text. ArXiv [cs.CL]. (2023). http:\/\/arxiv.org\/abs\/2306.15666"},{"key":"5_CR7","unstructured":"Crothers, E., Japkowicz, N., Viktor, H.: Machine generated text: A comprehensive survey of threat models and detection methods. ArXiv [cs.CL] (2022). http:\/\/arxiv.org\/abs\/2210.07321"},{"key":"5_CR8","unstructured":"Kirchner, J., Ahmad, L., Aaronson, S., Leike, J.: New AI classifier for indicating AI-written text. OpenAI (2023)"},{"key":"5_CR9","doi-asserted-by":"crossref","unstructured":"Gehrmann, S., Strobelt, H., Rush, A.: GLTR: Statistical detection and visualization of generated text. ArXiv [cs.CL]. (2019). http:\/\/arxiv.org\/abs\/1906.04043","DOI":"10.18653\/v1\/P19-3019"},{"key":"5_CR10","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1038\/s41746-023-00819-6","volume":"6","author":"C Gao","year":"2023","unstructured":"Gao, C., et al.: Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. NPJ Digital Med. 6, 75 (2023)","journal-title":"NPJ Digital Med."},{"key":"5_CR11","unstructured":"Henriksson, T.: New survey finds half of newsrooms use Generative AI tools; only 20% have guidelines in place - WAN-IFRA. World Association Of News Publishers (2023). https:\/\/wan-ifra.org\/2023\/05\/new-genai-survey\/"},{"key":"5_CR12","doi-asserted-by":"crossref","unstructured":"Dwivedi, Y., Kshetri, N., Hughes, L., Slade, E., Jeyaraj, A., Kar, A.: So what if ChatGPT wrote it?\u201d Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. Int. J. Inf. Manage. 71, 102642 (2023)","DOI":"10.1016\/j.ijinfomgt.2023.102642"},{"key":"5_CR13","unstructured":"Hanley, H., Durumeric, Z.: Machine-made media: monitoring the mobilization of machine-generated articles on misinformation and mainstream news websites. ArXiv [cs.CY] (2023). http:\/\/arxiv.org\/abs\/2305.09820"},{"key":"5_CR14","unstructured":"Li, Z.: The dark side of ChatGPT: legal and ethical challenges from stochastic parrots and hallucination. ArXiv [cs.CY] (2023). http:\/\/arxiv.org\/abs\/2304.1434"},{"key":"5_CR15","doi-asserted-by":"crossref","unstructured":"Ferrara, E. Should ChatGPT be biased? Challenges and risks of bias in large language models. ArXiv [cs.CY] (2023). http:\/\/arxiv.org\/abs\/2304.03738","DOI":"10.5210\/fm.v28i11.13346"},{"key":"5_CR16","doi-asserted-by":"publisher","first-page":"148","DOI":"10.3390\/socsci12030148","volume":"12","author":"D Rozado","year":"2023","unstructured":"Rozado, D.: The political biases of ChatGPT. Soc. Sci. 12, 148 (2023)","journal-title":"Soc. Sci."},{"key":"5_CR17","unstructured":"Tang, R., Chuang, Y., Hu, X.: The science of detecting LLM-generated texts. ArXiv [cs.CL] (2023). http:\/\/arxiv.org\/abs\/2303.07205"},{"key":"5_CR18","unstructured":"Zellers, R., et al.: Defending a. ArXiv [cs.CL] (2019). http:\/\/arxiv.org\/abs\/1905.12616"},{"key":"5_CR19","doi-asserted-by":"publisher","first-page":"499","DOI":"10.1162\/coli_a_00380","volume":"46","author":"T Schuster","year":"2020","unstructured":"Schuster, T., Schuster, R., Shah, D., Barzilay, R.: The limitations of stylometry for detecting machine-generated fake news. Comput. Linguist. 46, 499\u2013510 (2020). https:\/\/doi.org\/10.1162\/coli_a_00380","journal-title":"Comput. Linguist."},{"key":"5_CR20","unstructured":"Kumarage, T., et al.: phJ-Guard: Journalism Guided Adversarially Robust Detection of AI-generated News. arXiv preprint arXiv:2309.03164 (2023)"},{"key":"5_CR21","unstructured":"Pu, J., et al.: Deepfake text detection: Limitations and opportunities. ArXiv [cs.CR] (2022). http:\/\/arxiv.org\/abs\/2210.09421"},{"key":"5_CR22","unstructured":"Guo, B., et al.: How close is ChatGPT to human experts? Comparison corpus, evaluation, and detection. ArXiv [cs.CL] (2023). http:\/\/arxiv.org\/abs\/2301.07597"},{"key":"5_CR23","doi-asserted-by":"publisher","first-page":"42","DOI":"10.3390\/designs5030042","volume":"5","author":"E Lazarski","year":"2021","unstructured":"Lazarski, E., Al-Khassaweneh, M., Howard, C.: Using NLP for fact checking: a survey. Designs 5, 42 (2021). https:\/\/doi.org\/10.3390\/designs5030042","journal-title":"Designs"},{"key":"5_CR24","first-page":"21","volume":"17","author":"L Dierickx","year":"2023","unstructured":"Dierickx, L., Lind\u00e9n, C., Opdahl, A.L.: Automated fact-checking to support professional practices: systematic literature review and meta-analysis. Int. J. Commun. 17, 21 (2023)","journal-title":"Int. J. Commun."},{"key":"5_CR25","unstructured":"Graves, D.: Understanding the promise and limits of automated fact-checking. Reuters Institute for the Study of Journalism (2018)"},{"key":"5_CR26","unstructured":"Schlichtkrull, M., Ousidhoum, N., Vlachos, A.: The intended uses of automated fact-checking artefacts: Why, how and who. ArXiv [cs.CL] (2023). http:\/\/arxiv.org\/abs\/2304.14238"},{"key":"5_CR27","first-page":"1","volume":"15","author":"H Alkaissi","year":"2023","unstructured":"Alkaissi, H., McFarlane, S.: Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus. 15, 1\u20135 (2023)","journal-title":"Cureus."},{"key":"5_CR28","doi-asserted-by":"crossref","unstructured":"Buholayka, M., Zouabi, R., Tadinada, A.: Is ChatGPT ready to write scientific case reports independently? A comparative evaluation between human and artificial intelligence. Cureus. 15, 1\u20136 (2023). https:\/\/doi.org\/10.7759252Fcureus.39386","DOI":"10.7759\/cureus.39386"},{"key":"5_CR29","unstructured":"Thomson, C., Reiter, E.: A gold standard methodology for evaluating accuracy in data-to-text systems. ArXiv [cs.CL] (2020). http:\/\/arxiv.org\/abs\/2011.03992"},{"key":"5_CR30","doi-asserted-by":"publisher","DOI":"10.1016\/j.csl.2020.101151","volume":"67","author":"C van der Lee","year":"2021","unstructured":"van der Lee, C., Gatt, A., Miltenburg, E., Krahmer, E.: Human evaluation of automatically generated text: current trends and best practice guidelines. Comput. Speech Lang. 67, 101151 (2021)","journal-title":"Comput. Speech Lang."},{"key":"5_CR31","first-page":"65","volume":"23","author":"B Lester","year":"1997","unstructured":"Lester, B.: Developing and empirically evaluating robust explanation generators: The KNIGHT experiments. Comput. Linguist. 23, 65\u2013101 (1997)","journal-title":"Comput. Linguist."},{"key":"5_CR32","unstructured":"Belz, A., Reiter, E.: Comparing automatic and human evaluation of NLG systems. In: 11th Conference of the European Chapter of the Association For Computational Linguistics, pp. 313\u2013320 (2006)"},{"key":"5_CR33","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1162\/coli.2009.35.4.35405","volume":"35","author":"A Belz","year":"2009","unstructured":"Belz, A., Reiter, E.: An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Comput. Linguist. 35, 529\u2013558 (2009)","journal-title":"Comput. Linguist."},{"key":"5_CR34","unstructured":"Dale, R., White, M.: Shared tasks and comparative evaluation in natural language generation. In: Proceedings of the Workshop on Shared Tasks and Comparative Evaluation in Natural Language Generation, pp. 1\u20136 (2007)"},{"key":"5_CR35","unstructured":"Graefe, A., Haim, M., Haarmann, B., Brosius, H.: Perception of automated computer-generated news: credibility, expertise, and readability. 11th Dubrovnik Media Days, Dubrovnik (2015)"},{"key":"5_CR36","first-page":"1044","volume":"5","author":"M Haim","year":"2017","unstructured":"Haim, M., Graefe, A.: Automated news: better than expected? Digit. J. 5, 1044\u20131059 (2017)","journal-title":"Digit. J."},{"key":"5_CR37","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1177\/1464884918757072","volume":"22","author":"A W\u00f6lker","year":"2021","unstructured":"W\u00f6lker, A., Powell, T.: Algorithms in the newsroom? News readers\u2019 perceived credibility and selection of automated journalism. Journalism (London, England). 22, 86\u2013103 (2021). https:\/\/doi.org\/10.1177\/1464884918757072","journal-title":"Journalism (London, England)."},{"key":"5_CR38","doi-asserted-by":"publisher","first-page":"43356","DOI":"10.1109\/access.2018.2861987","volume":"6","author":"M Melin","year":"2018","unstructured":"Melin, M., Back, A., Sodergard, C., Munezero, M., Leppanen, L., Toivonen, H.: No landslide for the human journalist - an empirical study of computer-generated election news in Finland. IEEE Access Pract. Innov. Open Solut. 6, 43356\u201343367 (2018). https:\/\/doi.org\/10.1109\/access.2018.2861987","journal-title":"IEEE Access Pract. Innov. Open Solut."},{"key":"5_CR39","doi-asserted-by":"publisher","first-page":"519","DOI":"10.1080\/17512786.2014.883116","volume":"8","author":"C Clerwall","year":"2014","unstructured":"Clerwall, C.: Enter the robot journalist: users\u2019 perceptions of automated content. J. Pract. 8, 519\u2013531 (2014). https:\/\/doi.org\/10.1080\/17512786.2014.883116","journal-title":"J. Pract."},{"key":"5_CR40","unstructured":"Van Der Kaa, H., Krahmer, E.: Journalist versus news consumer: the perceived credibility of machine-written news. In: Proceedings of the Computation+Journalism Conference. (2014)"},{"key":"5_CR41","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1080\/1461670x.2011.629803","volume":"13","author":"J Johnston","year":"2012","unstructured":"Johnston, J., Graham, C.: The new, old journalism: narrative writing in contemporary newspapers. J. Stud. 13, 517\u2013533 (2012). https:\/\/doi.org\/10.1080\/1461670x.2011.629803","journal-title":"J. Stud."},{"key":"5_CR42","doi-asserted-by":"crossref","unstructured":"Tandoc Jr, E., Thomas, R., Bishop, L.: What is (fake) news? Analyzing news values (and more) in fake stories. Med. Commun. 9, 110\u2013119 (2021). https:\/\/doi.org\/10.17645252Fmac.v9i1.3331","DOI":"10.17645\/mac.v9i1.3331"},{"key":"5_CR43","doi-asserted-by":"crossref","unstructured":"Jurish, B., W\u00fcrzner, K.: Word and sentence tokenization with hidden Markov models. J. Lang. Technol. Comput. Linguist. 28, 61\u201383 (2013). https:\/\/doi.org\/10.21248252Fjlcl.28.2013.176","DOI":"10.21248\/jlcl.28.2013.176"},{"key":"5_CR44","unstructured":"Matusov, E., Leusch, G., Bender, O., Ney, H.: Evaluating machine translation output with automatic sentence segmentation. In: Proceedings of the Second International Workshop on Spoken Language Translation (2005)"},{"key":"5_CR45","doi-asserted-by":"crossref","unstructured":"Hanitzsch, T. Deconstructing journalism culture: toward a universal theory. Communication Theory. 17, 367\u2013385 (2007). https:\/\/doi.org\/10.1111252Fj.1468-2885.2007.00303.x","DOI":"10.1111\/j.1468-2885.2007.00303.x"},{"key":"5_CR46","doi-asserted-by":"crossref","unstructured":"Ward, S.: Truth and Objectivity. The Routledge Handbook of Mass Media Ethics, pp. 101\u2013114 (2020). https:\/\/doi.org\/10.4324252F9781315545929-8","DOI":"10.4324\/9781315545929-8"},{"key":"5_CR47","doi-asserted-by":"crossref","unstructured":"Walter, N., Salovich, N.: Unchecked vs. uncheckable: how opinion-based claims can impede corrections of misinformation. Mass Commun. Soc. 24, 500\u2013526 (2021). https:\/\/doi.org\/10.1080252F15205436.2020.1864406","DOI":"10.1080\/15205436.2020.1864406"}],"container-title":["Lecture Notes in Computer Science","Disinformation in Open Online Media"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-47896-3_5","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,13]],"date-time":"2023-11-13T20:03:17Z","timestamp":1699905797000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-47896-3_5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"ISBN":["9783031478956","9783031478963"],"references-count":47,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-47896-3_5","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"type":"print","value":"0302-9743"},{"type":"electronic","value":"1611-3349"}],"subject":[],"published":{"date-parts":[[2023]]},"assertion":[{"value":"14 November 2023","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"MISDOOM","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Multidisciplinary International Symposium on Disinformation in Open Online Media","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Amsterdam","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"The Netherlands","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2023","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"21 November 2023","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"22 November 2023","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"5","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"misdoom2023","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/event.cwi.nl\/misdoom-2023\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Single-blind","order":1,"name":"type","label":"Type","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"openReview.net","order":2,"name":"conference_management_system","label":"Conference Management System","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"19","order":3,"name":"number_of_submissions_sent_for_review","label":"Number of Submissions Sent for Review","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"13","order":4,"name":"number_of_full_papers_accepted","label":"Number of Full Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"0","order":5,"name":"number_of_short_papers_accepted","label":"Number of Short Papers Accepted","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"68% - The value is computed by the equation \"Number of Full Papers Accepted \/ Number of Submissions Sent for Review * 100\" and then rounded to a whole number.","order":6,"name":"acceptance_rate_of_full_papers","label":"Acceptance Rate of Full Papers","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"2.375","order":7,"name":"average_number_of_reviews_per_paper","label":"Average Number of Reviews per Paper","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"2.71","order":8,"name":"average_number_of_papers_per_reviewer","label":"Average Number of Papers per Reviewer","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}},{"value":"No","order":9,"name":"external_reviewers_involved","label":"External Reviewers Involved","group":{"name":"ConfEventPeerReviewInformation","label":"Peer Review Information (provided by the conference organizers)"}}]}}