{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T17:38:49Z","timestamp":1792172329564,"version":"4.4.0"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2023,8,6]],"date-time":"2023-08-06T00:00:00Z","timestamp":1691280000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,8,6]],"date-time":"2023-08-06T00:00:00Z","timestamp":1691280000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100016386","name":"Conselleria de Innovaci\u00f3n, Universidades, Ciencia y Sociedad Digital, Generalitat Valenciana","doi-asserted-by":"publisher","award":["ACIF\/2020\/177"],"award-info":[{"award-number":["ACIF\/2020\/177"]}],"id":[{"id":"10.13039\/501100016386","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100014440","name":"Ministerio de Ciencia, Innovaci\u00f3n y Universidades","doi-asserted-by":"publisher","award":["PID2021-122263OBC22"],"award-info":[{"award-number":["PID2021-122263OBC22"]}],"id":[{"id":"10.13039\/100014440","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004837","name":"Ministerio de Ciencia e Innovaci\u00f3n","doi-asserted-by":"publisher","award":["PDC2022-133146-C22"],"award-info":[{"award-number":["PDC2022-133146-C22"]}],"id":[{"id":"10.13039\/501100004837","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100014849","name":"Conselleria de Cultura, Educaci\u00f3n y Ciencia, Generalitat Valenciana","doi-asserted-by":"publisher","award":["CIPROM\/ 2021\/21"],"award-info":[{"award-number":["CIPROM\/ 2021\/21"]}],"id":[{"id":"10.13039\/501100014849","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100009092","name":"Universidad de Alicante","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100009092","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Lang Resources &amp; Evaluation"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The development of the internet and digital technologies has inadvertently facilitated the huge disinformation problem that faces society nowadays. This phenomenon impacts ideologies, politics and public health. The 2016 US presidential elections, the Brexit referendum, the COVID-19 pandemic and the Russia-Ukraine war have been ideal scenarios for the spreading of fake news and hoaxes, due to the massive dissemination of information. Assuming that fake news mixes reliable and unreliable information, we propose RUN-AS (Reliable and Unreliable Annotation Scheme), a fine-grained annotation scheme that enables the labelling of the structural parts and essential content elements of a news item and their classification into Reliable and Unreliable. This annotation proposal aims to detect disinformation patterns in text and to classify the global reliability of news. To this end, a dataset in Spanish was built and manually annotated with RUN-AS and several experiments using this dataset were conducted to validate the annotation scheme by using Machine Learning (ML) and Deep Learning (DL) algorithms. The experiments evidence the validity of the annotation scheme proposed, obtaining the best <jats:inline-formula><jats:alternatives><jats:tex-math>$$\\textbf F_\\textbf 1\\textbf m$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:msub>\n                      <mml:mi>F<\/mml:mi>\n                      <mml:mn>1<\/mml:mn>\n                    <\/mml:msub>\n                    <mml:mi>m<\/mml:mi>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>, 0.948, with the Decision Tree algorithm.<\/jats:p>","DOI":"10.1007\/s10579-023-09678-9","type":"journal-article","created":{"date-parts":[[2023,8,6]],"date-time":"2023-08-06T14:01:21Z","timestamp":1691330481000},"page":"609-639","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["RUN-AS: a novel approach to annotate news reliability for disinformation detection"],"prefix":"10.1007","volume":"58","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7172-0094","authenticated-orcid":false,"given":"Alba","family":"Bonet-Jover","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Robiert","family":"Sep\u00falveda-Torres","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Estela","family":"Saquete","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patricio","family":"Mart\u00ednez-Barco","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mario","family":"Nieto-P\u00e9rez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,8,6]]},"reference":[{"key":"9678_CR1","doi-asserted-by":"crossref","unstructured":"Assaf, R., & Saheb, M. (2021). Dataset for arabic fake news. In 2021 IEEE 15th International Conference on Application of Information and Communication Technologies (AICT), IEEE (pp. 1\u20134).","DOI":"10.1109\/AICT52784.2021.9620228"},{"key":"9678_CR2","doi-asserted-by":"publisher","first-page":"192","DOI":"10.1016\/j.ins.2011.12.028","volume":"191","author":"C Bergmeir","year":"2012","unstructured":"Bergmeir, C., & Ben\u00edtez, J. M. (2012). On the use of cross-validation for time series predictor evaluation. Information Sciences, 191, 192\u2013213.","journal-title":"Information Sciences"},{"key":"9678_CR3","first-page":"2020","volume":"2020","author":"J Canete","year":"2020","unstructured":"Ca\u00f1ete, J., Chaperon, G., Fuentes, R., Ho, J. H., Kang, H., & P\u00e9rez, J. (2020). Spanish pre-trained bert model and evaluation data. Pml4dc at iclr, 20202020, 1-10","journal-title":"Pml4dc at iclr"},{"issue":"3","key":"9678_CR4","doi-asserted-by":"publisher","first-page":"747","DOI":"10.13053\/cys-22-3-3016","volume":"22","author":"K Chakma","year":"2018","unstructured":"Chakma, K., & Das, A. (2018). A 5w1h based annotation scheme for semantic role labeling of english tweets. Computaci\u00f3n y Sistemas, 22(3), 747\u2013755.","journal-title":"Computaci\u00f3n y Sistemas"},{"key":"9678_CR5","doi-asserted-by":"crossref","unstructured":"Chakma, K., Swamy, S. D., Das, A., & Debbarma, S. (2020). 5w1h-based semantic segmentation of tweets for event detection using bert. In International Conference on Machine Learning, Image Processing, Network Security and Data Sciences (pp 57\u201372). Springer","DOI":"10.1007\/978-981-15-6315-7_5"},{"issue":"2","key":"9678_CR6","doi-asserted-by":"publisher","first-page":"385","DOI":"10.1177\/1077699018792272","volume":"96","author":"TI DeAngelo","year":"2019","unstructured":"DeAngelo, T. I., & Yegiyan, N. S. (2019). Looking for efficiency: How online news structure and emotional tone influence processing time and memory. Journalism & Mass Communication Quarterly, 96(2), 385\u2013405.","journal-title":"Journalism & Mass Communication Quarterly"},{"key":"9678_CR7","doi-asserted-by":"publisher","unstructured":"Ferreira, W., & Vlachos, A. (2016). Emergent: a novel data-set for stance classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pp 1163\u20131168. https:\/\/doi.org\/10.18653\/v1\/N16-1138, https:\/\/aclanthology.org\/N16-1138.","DOI":"10.18653\/v1\/N16-1138"},{"key":"9678_CR8","doi-asserted-by":"publisher","first-page":"817","DOI":"10.1016\/j.procs.2017.11.106","volume":"121","author":"\u00c1 Figueira","year":"2017","unstructured":"Figueira, \u00c1., & Oliveira, L. (2017). The current state of fake news: Challenges and opportunities. Procedia Computer Science, 121, 817\u2013825.","journal-title":"Procedia Computer Science"},{"key":"9678_CR9","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4842-7155-1","volume-title":"How Algorithms Create and Prevent Fake News","author":"N Giansiracusa","year":"2021","unstructured":"Giansiracusa, N. (2021). How Algorithms Create and Prevent Fake News. Springer."},{"key":"9678_CR10","unstructured":"Gruppi, M., Horne, B. D., & Adali, S. (2018). An exploration of unreliable news classification in brazil and the us. arXiv preprint arXiv:1806.02875."},{"key":"9678_CR11","doi-asserted-by":"crossref","unstructured":"Hamborg, F., Breitinger, C., Schubotz, M., Lachnit, S., & Gipp, B. (2018). Extraction of main event descriptors from news articles by answering the journalistic five w and one h questions. InProceedings of the 18th ACM\/IEEE on Joint Conference on Digital Libraries, (pp. 339\u2013340).","DOI":"10.1145\/3197026.3203899"},{"key":"9678_CR12","unstructured":"Hordofa, B. A. (2020). Event extraction and representation model from news articles. 16, 1\u20138."},{"key":"9678_CR13","doi-asserted-by":"crossref","unstructured":"Horne, B., & Adali, S. (2017). This just in: Fake news packs a lot in title, uses simpler, repetitive content in text body, more similar to satire than real news. In Proceedings of the international AAAI conference on web and social media, (pp. 759\u2013766).","DOI":"10.1609\/icwsm.v11i1.14976"},{"key":"9678_CR14","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1016\/j.knosys.2014.11.017","volume":"76","author":"L Hou","year":"2015","unstructured":"Hou, L., Li, J., Wang, Z., Tang, J., Zhang, P., Yang, R., & Zheng, Q. (2015). Newsminer: Multifaceted news analysis for event search. Knowledge-Based Systems, 76, 17\u201329.","journal-title":"Knowledge-Based Systems"},{"key":"9678_CR15","doi-asserted-by":"crossref","unstructured":"Khodra, M. L. (2015). Event extraction on indonesian news article using multiclass categorization. In 2015 2nd International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA), IEEE, pp 1\u20135.","DOI":"10.1109\/ICAICTA.2015.7335365"},{"key":"9678_CR16","first-page":"683","volume":"3","author":"S Mottola","year":"2020","unstructured":"Mottola, S. (2020). Las fake news como fen\u00f3meno social. an\u00e1lisis ling\u00fc\u00edstico y poder persuasivo de bulos en italiano y espa\u00f1ol. Discurso & Sociedad, 3, 683\u2013706.","journal-title":"Discurso & Sociedad"},{"issue":"2","key":"9678_CR17","doi-asserted-by":"publisher","first-page":"310","DOI":"10.1016\/j.jbi.2010.11.001","volume":"44","author":"A N\u00e9v\u00e9ol","year":"2011","unstructured":"N\u00e9v\u00e9ol, A., Do\u011fan, R. I., & Lu, Z. (2011). Semi-automatic semantic annotation of pubmed queries: a study on quality, efficiency, satisfaction. Journal of biomedical informatics, 44(2), 310\u2013318.","journal-title":"Journal of biomedical informatics"},{"key":"9678_CR18","unstructured":"Norambuena, B., Horning, M., & Mitra, T. (2020). Evaluating the inverted pyramid structure through automatic 5w1h extraction and summarization. In Computational Journalism Symposium."},{"issue":"107","key":"9678_CR19","first-page":"393","volume":"107","author":"WS Paka","year":"2021","unstructured":"Paka, W. S., Bansal, R., Kaushik, A., et al. (2021). Cross-sean: A cross-stitch semi-supervised neural attention model for covid-19 fake news detection. Applied Soft Computing, 107(107), 393.","journal-title":"Applied Soft Computing"},{"issue":"10","key":"9678_CR20","doi-asserted-by":"publisher","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","volume":"22","author":"SJ Pan","year":"2010","unstructured":"Pan, S. J., & Yang, Q. (2010). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 1345\u20131359.","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"9678_CR21","doi-asserted-by":"crossref","unstructured":"Patwa, P., Sharma, S., Pykl, S., Guptha, V., Kumari, G., Akhtar, M. S., ... & Chakraborty, T. (2021). Fighting an infodemic: Covid-19 fake news dataset. In International Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situation (pp. 21\u201329). Springer.","DOI":"10.1007\/978-3-030-73696-5_3"},{"key":"9678_CR22","unstructured":"Pennebaker, J. W., Boyd, R. L., Jordan, K., & Blackburn, K. (2015).The development and psychometric properties of liwc2015. Tech. rep."},{"key":"9678_CR23","unstructured":"P\u00e9rez-Rosas, V., Kleinberg, B., Lefevre, A., & Mihalcea, R. (2017). Automatic detection of fake news. arXiv preprint arXiv:1708.07104."},{"issue":"5","key":"9678_CR24","doi-asserted-by":"publisher","first-page":"4869","DOI":"10.3233\/JIFS-179034","volume":"36","author":"JP Posadas-Dur\u00e1n","year":"2019","unstructured":"Posadas-Dur\u00e1n, J. P., G\u00f3mez-Adorno, H., Sidorov, G., & Escobar, J. J. M. (2019). Detection of fake news in a new corpus for the spanish language. Journal of Intelligent & Fuzzy Systems, 36(5), 4869\u20134876.","journal-title":"Journal of Intelligent & Fuzzy Systems"},{"key":"9678_CR25","doi-asserted-by":"crossref","unstructured":"Rashkin, H., Choi, E., Jang, J. Y., Volkova, S., & Choi, Y. (2017). Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 conference on empirical methods in natural language processing, (pp. 2931\u20132937).","DOI":"10.18653\/v1\/D17-1317"},{"issue":"112","key":"9678_CR26","first-page":"943","volume":"141","author":"E Saquete","year":"2020","unstructured":"Saquete, E., Tom\u00e1s, D., Moreda, P., Mart\u00ednez-Barco, P., & Palomar, M. (2020). Fighting post-truth using natural language processing: A review and open challenges. Expert systems with applications, 141(112), 943.","journal-title":"Expert systems with applications"},{"key":"9678_CR27","unstructured":"Sep\u00falveda-Torres, R., & Saquete Bor\u00f3, E. (2021). Gplsi team at checkthat! 2021: Fine-tuning beto and roberta. CEUR."},{"key":"9678_CR28","unstructured":"Shahi, G. K., Stru\u00df, J. M., & Mandl, T. (2021). Overview of the clef-2021 checkthat! lab task 3 on fake news detection. Working Notes of CLEF."},{"key":"9678_CR29","unstructured":"Shao, C., Ciampaglia, G. L., Varol, O., Flammini, A., & Menczer, F. (2017). The spread of fake news by social bots. arXiv preprint arXiv:1707.0759296, 104."},{"key":"9678_CR30","doi-asserted-by":"crossref","unstructured":"Shu, K., Wang, S., Lee, D., & Liu, H. (2020). Mining disinformation and fake news: Concepts, methods, and recent advancements. In Disinformation, Misinformation, and Fake News in Social Media, (pp. 1\u201319).","DOI":"10.1007\/978-3-030-42699-6_1"},{"issue":"113","key":"9678_CR31","first-page":"199","volume":"146","author":"RM Silva","year":"2020","unstructured":"Silva, R. M., Santos, R. L., Almeida, T. A., & Pardo, T. A. (2020). Towards automatically filtering fake news in portuguese. Expert Systems with Applications, 146(113), 199.","journal-title":"Expert Systems with Applications"},{"key":"9678_CR32","unstructured":"Stenetorp, P., Pyysalo, S., Topi\u0107, G., Ohta, T., Ananiadou, S., & Tsujii, J. I. (2012). Brat: a web-based tool for nlp-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics (pp. 102\u2013107)."},{"issue":"5","key":"9678_CR33","doi-asserted-by":"publisher","first-page":"1299","DOI":"10.1109\/TMI.2016.2535302","volume":"35","author":"N Tajbakhsh","year":"2016","unstructured":"Tajbakhsh, N., Shin, J. Y., Gurudu, S. R., Hurst, R. T., Kendall, C. B., Gotway, M. B., & Liang, J. (2016). Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE transactions on medical imaging, 35(5), 1299\u20131312.","journal-title":"IEEE transactions on medical imaging"},{"issue":"2","key":"9678_CR34","doi-asserted-by":"publisher","first-page":"212","DOI":"10.1080\/14616700701848261","volume":"9","author":"EA Thomson","year":"2008","unstructured":"Thomson, E. A., White, P. R., & Kitley, P. (2008). objectivity and hard news reporting across cultures: Comparing the news report in english, french, japanese and indonesian journalism. Journalism studies, 9(2), 212\u2013228.","journal-title":"Journalism studies"},{"key":"9678_CR35","doi-asserted-by":"crossref","unstructured":"Vieira, S. M., Kaymak, U., & Sousa, J. M. (2010). Cohen\u2019s kappa coefficient as a performance measure for feature selection. In International Conference on Fuzzy Systems , IEEE (pp. 1\u20138).","DOI":"10.1109\/FUZZY.2010.5584447"},{"key":"9678_CR36","doi-asserted-by":"crossref","unstructured":"Vlachos, A., & Riedel, S. (2014). Fact checking: Task definition and dataset construction. InProceedings of the ACL 2014 workshop on language technologies and computational social science (pp. 18\u201322).","DOI":"10.3115\/v1\/W14-2508"},{"key":"9678_CR37","doi-asserted-by":"crossref","unstructured":"Wang, W. Y. (2017). \u201cliar, liar pants on fire\u201d: A new benchmark dataset for fake news detection. arXiv preprint arXiv:1705.00648.","DOI":"10.18653\/v1\/P17-2067"},{"key":"9678_CR38","doi-asserted-by":"crossref","unstructured":"Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., ... & Rush, A. M. (2019). Huggingface\u2019s transformers: State-of-the-art natural language processing. arXiv:1910.03771.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"9678_CR39","doi-asserted-by":"crossref","unstructured":"Zhang, A. X., Ranganathan, A., Metz, S. E., Appling, S., Sehat, C. M., Gilmore, N., ... & Mina, A. X. (2018). A structured response to misinformation: Defining and annotating credibility indicators in news articles. In Companion Proceedings of the The Web Conference (pp 603\u2013612).","DOI":"10.1145\/3184558.3188731"}],"container-title":["Language Resources and Evaluation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-023-09678-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10579-023-09678-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-023-09678-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,28]],"date-time":"2024-05-28T18:30:52Z","timestamp":1716921052000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10579-023-09678-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,6]]},"references-count":39,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["9678"],"URL":"https:\/\/doi.org\/10.1007\/s10579-023-09678-9","relation":{},"ISSN":["1574-020X","1574-0218"],"issn-type":[{"value":"1574-020X","type":"print"},{"value":"1574-0218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,6]]},"assertion":[{"value":"13 June 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 August 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}