{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,25]],"date-time":"2025-10-25T12:43:50Z","timestamp":1761396230418,"version":"build-2065373602"},"reference-count":57,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,2,28]],"date-time":"2022-02-28T00:00:00Z","timestamp":1646006400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Ministero dell'Universit\u00e0 e della Ricerca - Fondo Integrativo Speciale per la Ricerca","award":["COKE_FISR2020IP_00621"],"award-info":[{"award-number":["COKE_FISR2020IP_00621"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>The COVID-19 pandemic highlighted the importance of validated and updated scientific information to help policy makers, healthcare professionals, and the public. The speed in disseminating reliable information and the subsequent guidelines and policy implementation are also essential to save as many lives as possible. Trustworthy guidelines should be based on a systematic evidence review which uses reproducible analytical methods to collect secondary data and analyse them. However, the guidelines\u2019 drafting process is time consuming and requires a great deal of resources. This paper aims to highlight the importance of accelerating and streamlining the extraction and synthesis of scientific evidence, specifically within the systematic review process. To do so, this paper describes the COKE (COVID-19 Knowledge Extraction framework for next generation discovery science) Project, which involves the use of machine reading and deep learning to design and implement a semi-automated system that supports and enhances the systematic literature review and guideline drafting processes. Specifically, we propose a framework for aiding in the literature selection and navigation process that employs natural language processing and clustering techniques for selecting and organizing the literature for human consultation, according to PICO (Population\/Problem, Intervention, Comparison, and Outcome) elements. We show some preliminary results of the automatic classification of sentences on a dataset of abstracts related to COVID-19.<\/jats:p>","DOI":"10.3390\/info13030117","type":"journal-article","created":{"date-parts":[[2022,2,28]],"date-time":"2022-02-28T20:11:14Z","timestamp":1646079074000},"page":"117","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Semi-Automatic Systematic Literature Reviews and Information Extraction of COVID-19 Scientific Evidence: Description and Preliminary Results of the COKE Project"],"prefix":"10.3390","volume":"13","author":[{"given":"Davide","family":"Golinelli","sequence":"first","affiliation":[{"name":"Department of Biomedical and Neuromotor Sciences (DIBINEM), Alma Mater Studiorum\u2014University of Bologna, 40126 Bologna, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2928-9496","authenticated-orcid":false,"given":"Andrea Giovanni","family":"Nuzzolese","sequence":"additional","affiliation":[{"name":"STLab, Institute for Cognitive Sciences and Technologies (ISTC)-CNR, 00185 Roma, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Francesco","family":"Sanmarchi","sequence":"additional","affiliation":[{"name":"Department of Biomedical and Neuromotor Sciences (DIBINEM), Alma Mater Studiorum\u2014University of Bologna, 40126 Bologna, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1165-853X","authenticated-orcid":false,"given":"Luana","family":"Bulla","sequence":"additional","affiliation":[{"name":"STLab, Institute for Cognitive Sciences and Technologies (ISTC)-CNR, 00185 Roma, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0528-5490","authenticated-orcid":false,"given":"Misael","family":"Mongiov\u00ec","sequence":"additional","affiliation":[{"name":"STLab, Institute for Cognitive Sciences and Technologies (ISTC)-CNR, via Gaifami 18, 95126 Catania, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aldo","family":"Gangemi","sequence":"additional","affiliation":[{"name":"STLab, Institute for Cognitive Sciences and Technologies (ISTC)-CNR, 00185 Roma, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paola","family":"Rucci","sequence":"additional","affiliation":[{"name":"Department of Biomedical and Neuromotor Sciences (DIBINEM), Alma Mater Studiorum\u2014University of Bologna, 40126 Bologna, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,2,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1177\/0141076820930665","article-title":"Uncertainty and COVID-19: How are we to respond?","volume":"113","author":"Koffman","year":"2020","journal-title":"J. R. Soc. Med."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e2117359","DOI":"10.1001\/jamanetworkopen.2021.17359","article-title":"Exploring the gap between excess mortality and COVID-19 deaths in 67 countries","volume":"4","author":"Sanmarchi","year":"2021","journal-title":"JAMA Netw. Open"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Hua, J., and Shaw, R. (2020). Corona virus (COVID-19) \u201cinfodemic\u201d and emerging issues through a data lens: The case of China. Int. J. Environ. Res. Public Health, 17.","DOI":"10.3390\/ijerph17072309"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1111\/hir.12311","article-title":"The Covid-19 \u2018infodemic\u2019: A new front for information professionals","volume":"37","author":"Bhatti","year":"2020","journal-title":"Health Inf. Libr. J."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"The Lancet Infectious Diseases (2020). The COVID-19 infodemic. Lancet Infect. Dis., 20, 875.","DOI":"10.1016\/S1473-3099(20)30565-X"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"148","DOI":"10.15344\/2456-8007\/2020\/148","article-title":"COVID-19 guidelines changing faster than the virus: Implications of a clinical decision support app","volume":"5","author":"Saperstein","year":"2020","journal-title":"Int. J. Clin. Res. Trials"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"570","DOI":"10.1089\/bio.2020.0134","article-title":"Management and data sharing of COVID-19 pandemic information","volume":"18","author":"Gao","year":"2020","journal-title":"Biopreserv. Biobank."},{"key":"ref_8","unstructured":"WHO (2021, March 31). Therapeutics and COVID-19: Living Guideline. Available online: https:\/\/www.who.int\/publications\/i\/item\/WHO-2019-nCoV-therapeutics-2021.1."},{"key":"ref_9","unstructured":"Centers for Disease Control and Prevention (2021, July 06). Interim Public Health Recommendations for Fully Vaccinated People, Available online: https:\/\/stacks.cdc.gov\/view\/cdc\/105629."},{"key":"ref_10","unstructured":"European Centre for Disease Prevention and Control, and World Health Organization Regional Office for Europe (2021). Methods for the Detection and Characterisation of SARS-CoV-2 Variants, WHO Regional Office for Europe."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"E123","DOI":"10.1503\/cmaj.131237","article-title":"Guidelines 2.0: Systematic development of a comprehensive checklist for a successful guideline enterprise","volume":"186","author":"Wiercioch","year":"2014","journal-title":"Can. Med. Assoc. J."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"e19","DOI":"10.1136\/medethics-2020-106639","article-title":"Frauds in scientific research and how to possibly overcome them","volume":"47","author":"Boetto","year":"2021","journal-title":"J. Med. Ethics"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"6761","DOI":"10.1007\/s11192-021-04059-x","article-title":"Publication patterns\u2019 changes due to the COVID-19 pandemic: A longitudinal and short-term scientometric analysis","volume":"126","author":"Rosenfeld","year":"2021","journal-title":"Scientometrics"},{"key":"ref_14","first-page":"1243","article-title":"Extracting healthcare quality information from unstructured data","volume":"2017","author":"Malmasi","year":"2018","journal-title":"AMIA Annu. Symp. Proc."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1055\/s-0040-1702001","article-title":"Medical information extraction in the age of deep learning","volume":"29","author":"Hahn","year":"2020","journal-title":"Yearb. Med. Inform."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"e74","DOI":"10.2196\/jmir.1943","article-title":"Using internet search engines to obtain medical information: A comparative study","volume":"14","author":"Wang","year":"2012","journal-title":"J. Med. Internet Res."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1186\/s40537-019-0217-0","article-title":"Big data in healthcare: Management, analysis and future prospects","volume":"6","author":"Dash","year":"2019","journal-title":"J. Big Data"},{"key":"ref_18","unstructured":"World Health Organization (2014). WHO Handbook for Guideline Development, World Health Organization. [2nd ed.]."},{"key":"ref_19","first-page":"39","article-title":"Artificial intelligence to win the Nobel prize and beyond: Creating the engine for scientific discovery","volume":"37","author":"Kitano","year":"2016","journal-title":"AI Mag."},{"key":"ref_20","unstructured":"(2020, June 23). Nature Index. Available online: https:\/\/www.natureindex.com\/news-blog\/the-top-coronavirus-research-articles-by-metrics."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1177\/014107680309600304","article-title":"Five steps to conducting a systematic review","volume":"96","author":"Khan","year":"2003","journal-title":"J. R. Soc. Med."},{"key":"ref_22","unstructured":"Cooper, H. (2015). Research Synthesis and Meta-Analysis: A Step-by-Step Approach, SAGE Publications."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"e1","DOI":"10.1016\/j.jclinepi.2009.06.006","article-title":"The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration","volume":"62","author":"Liberati","year":"2009","journal-title":"J. Clin. Epidemiol."},{"key":"ref_24","unstructured":"Boaz, A., Ashby, D., and Young, K. (2002). Systematic Reviews: What Have They Got to Offer Evidence Based Policy and Practice?, ESRC UK Centre for Evidence Based Policy and Practice."},{"key":"ref_25","unstructured":"Oliver, S., Dickson, K., and Bangpan, M. (2015). Systematic Reviews: Making Them Policy Relevant. A Briefing for Policy Makers and Systematic Reviewers, UCL Institute of Education."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1186\/s13643-016-0249-x","article-title":"Searching for qualitative research for inclusion in systematic reviews: A structured methodological review","volume":"5","author":"Booth","year":"2016","journal-title":"Syst. Rev."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1016\/j.jclinepi.2010.04.026","article-title":"GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables","volume":"64","author":"Guyatt","year":"2011","journal-title":"J. Clin. Epidemiol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1186\/s13643-019-1074-9","article-title":"Toward systematic review automation: A practical guide to using machine learning tools in research synthesis","volume":"8","author":"Marshall","year":"2019","journal-title":"Syst. Rev."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"401","DOI":"10.12688\/f1000research.51117.1","article-title":"Data extraction methods for systematic review (semi)automation: A living systematic review","volume":"10","author":"Schmidt","year":"2021","journal-title":"F1000Research"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1016\/j.jclinepi.2021.12.005","article-title":"Tools to support the automation of systematic reviews: A scoping review","volume":"144","author":"Khalil","year":"2021","journal-title":"J. Clin. Epidemiol."},{"key":"ref_31","unstructured":"Etzioni, O., Banko, M., and Cafarella, M.J. (2006). Machine Reading, Association for the Advancement of Artificial Intelligence."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"e012545","DOI":"10.1136\/bmjopen-2016-012545","article-title":"Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry","volume":"7","author":"Borah","year":"2017","journal-title":"BMJ Open"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"873","DOI":"10.3233\/SW-160240","article-title":"Semantic web machine reading with FRED","volume":"8","author":"Gangemi","year":"2017","journal-title":"Semant. Web"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1093\/jamia\/ocv044","article-title":"RobotReviewer: Evaluation of a system for automatically assessing bias in clinical trials","volume":"23","author":"Marshall","year":"2015","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Hogan, A., Blomqvist, E., Cochez, M., d\u2019Amato, C., de Melo, G., Gutierrez, C., Kirrane, S., Labra Gayo, J.E., Navigli, R., and Neumaier, R. (2021). Knowledge Graphs. Synthesis Lectures on Data, Semantics, and Knowledge, Morgan and Claypool Publishers.","DOI":"10.1007\/978-3-031-01918-0"},{"key":"ref_37","first-page":"1457","article-title":"Peirce\u2019s Notion of Interpretant","volume":"91","author":"Eco","year":"1976","journal-title":"Comp. Lit."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1016\/j.knosys.2016.05.023","article-title":"Identifying motifs for evaluating open knowledge extraction on the web","volume":"108","author":"Gangemi","year":"2016","journal-title":"Knowl. Based Syst."},{"key":"ref_39","unstructured":"Schuler, K.K. (2005). VerbNet: A Broad-Coverage, Comprehensive Verb Lexicon, University of Pennsylvania."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Auer, S., Bizer, C., Kobilarov, G., Lehmann, G., Lehmann, J., Cyganiak, R., and Ives, Z. (2007). Dbpedia: A Nucleus for a Web of Open Data, Springer.","DOI":"10.1007\/978-3-540-76298-0_52"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Jin, D., and Szolovits, P. (2018, January 19). Pico element detection in medical text via long short-term memory neural networks. Proceedings of the BioNLP Workshop, Melbourne, Australia.","DOI":"10.18653\/v1\/W18-2308"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10994-016-5589-1","article-title":"Special issue on discovery science","volume":"105","author":"Kocev","year":"2016","journal-title":"Mach. Learn."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Priem, J., Groth, P., and Taraborelli, D. (2012). The altmetrics collection. PLoS ONE, 7.","DOI":"10.1371\/journal.pone.0048753"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"539","DOI":"10.1007\/s11192-018-2988-z","article-title":"Do altmetrics work for assessing research quality?","volume":"118","author":"Nuzzolese","year":"2019","journal-title":"Scientometrics"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1189","DOI":"10.1007\/s11192-020-03809-7","article-title":"Using altmetrics for detecting impactful research in quasi-zero-day time-windows: The case of COVID-19","volume":"126","author":"Boetto","year":"2021","journal-title":"Scientometrics"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1038\/s42256-020-00287-7","article-title":"An open source machine learning framework for efficient and transparent systematic reviews","volume":"3","author":"Schram","year":"2021","journal-title":"Nat. Mach. Intell."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"D267","DOI":"10.1093\/nar\/gkh061","article-title":"The unified medical language system (UMLS): Integrating biomedical terminology","volume":"32","author":"Bodenreider","year":"2004","journal-title":"Nucleic Acids Res."},{"key":"ref_48","first-page":"279","article-title":"SNOMED-CT: The advanced terminology and coding system for eHealth","volume":"121","author":"Donnelly","year":"2006","journal-title":"Stud. Health Technol. Inform."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1145\/2629489","article-title":"Wikidata: A free collaborative knowledgebase","volume":"57","year":"2014","journal-title":"Commun. ACM"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"3856","DOI":"10.1093\/bioinformatics\/btaa256","article-title":"Advancing PICO element detection in biomedical text via deep neural networks","volume":"36","author":"Jin","year":"2020","journal-title":"Bioinformatics"},{"key":"ref_51","unstructured":"Devlin, J., Chang, M.-W., Lee, K.-t., and Toutanova, K. (2019, January 2\u20137). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: A pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2019","journal-title":"Bioinformatics"},{"key":"ref_53","unstructured":"Fraser, K.C., Nejadgholi, I., De Bruijn, B., Li, M., LaPlante, A., and El Abidine, K.Z. (2019). Extracting UMLS concepts from medical text using general and domain-specific deep learning models. arXiv."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Raiman, J., and Raiman, O. (2018, January 2\u20137). Deeptype: Multilingual entity linking by neural type system evolution. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.12008"},{"key":"ref_55","unstructured":"Mohan, S., and Li, D. (2019, January 20\u201321). Medmentions: A large biomedical corpus annotated with UMLS concepts. Proceedings of the Automated Knowledge Base Construction (AKBC), Amherst, MA, USA."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1002\/widm.53","article-title":"Algorithms for hierarchical clustering: An overview","volume":"2","author":"Murtagh","year":"2012","journal-title":"Wiley Interdiscip. Rev. Data Min. Knowl. Discov."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1145\/1113343.1113356","article-title":"Exploratory search interfaces: Categorization, clustering and beyond: Report on the xsi 2005 workshop at the human-computer interaction laboratory, University of Maryland","volume":"39","author":"White","year":"2005","journal-title":"ACM SIGIR Forum."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/3\/117\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:29:02Z","timestamp":1760135342000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/3\/117"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,28]]},"references-count":57,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["info13030117"],"URL":"https:\/\/doi.org\/10.3390\/info13030117","relation":{},"ISSN":["2078-2489"],"issn-type":[{"type":"electronic","value":"2078-2489"}],"subject":[],"published":{"date-parts":[[2022,2,28]]}}}