{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T04:00:52Z","timestamp":1772164852548,"version":"3.50.1"},"reference-count":43,"publisher":"MIT Press","issue":"3","license":[{"start":{"date-parts":[[2022,8,25]],"date-time":"2022-08-25T00:00:00Z","timestamp":1661385600000},"content-version":"vor","delay-in-days":236,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000780","name":"European Commission","doi-asserted-by":"publisher","award":["101017452"],"award-info":[{"award-number":["101017452"]}],"id":[{"id":"10.13039\/501100000780","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100009104","name":"Universit\u00e0 Degli Studi di Modena e Reggio Emilia","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100009104","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The importance of open bibliographic repositories is widely accepted by the scientific community. For evaluation processes, however, there is still some skepticism: Even if large repositories of open access articles and free publication indexes exist and are continuously growing, assessment procedures still rely on proprietary databases, mainly due to the richness of the data available in these proprietary databases and the services provided by the companies they are offered by. This paper investigates the status of open bibliographic data of three of the most used open resources, namely Microsoft Academic Graph, Crossref, and OpenAIRE, evaluating their potentialities as substitutes of proprietary databases for academic evaluation processes. We focused on the Italian National Scientific Qualification (NSQ), the Italian process for university professor qualification, which uses data from commercial indexes, and investigated similarities and differences between research areas, disciplines, and application roles. The main conclusion is that open data sets are ready to be used for some disciplines, among them mathematics, natural sciences, economics, and statistics, even if there is still room for improvement; but there is still a large gap to fill in others\u2014such as history, philosophy, pedagogy, and psychology\u2014and greater effort is required from researchers and institutions.<\/jats:p>","DOI":"10.1162\/qss_a_00203","type":"journal-article","created":{"date-parts":[[2022,8,25]],"date-time":"2022-08-25T14:35:17Z","timestamp":1661438117000},"page":"512-528","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":5,"title":["Open bibliographic data and the Italian National Scientific Qualification: Measuring coverage of academic fields"],"prefix":"10.1162","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3845-8266","authenticated-orcid":true,"given":"Federica","family":"Bologna","sequence":"first","affiliation":[{"name":"Bowers College of Computing and Information Science, Cornell University, Ithaca, NY"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6893-7452","authenticated-orcid":true,"given":"Angelo","family":"Di Iorio","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, University of Bologna, Bologna, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0530-4305","authenticated-orcid":true,"given":"Silvio","family":"Peroni","sequence":"additional","affiliation":[{"name":"Research Centre for Open Scholarly Metadata, Department of Classical Philology and Italian Studies, University of Bologna, Bologna, Italy"},{"name":"Digital Humanities Advanced Research Centre (\/DH.arc), Department of Classical Philology and Italian Studies, University of Bologna, Bologna, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6577-5606","authenticated-orcid":true,"given":"Francesco","family":"Poggi","sequence":"additional","affiliation":[{"name":"Department of Communication and Economics, University of Modena and Reggio Emilia, Reggio Emilia, Italy"},{"name":"Institute of Cognitive Sciences and Technologies, Italian National Research Council (CNR), Rome, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2022,11,1]]},"reference":[{"issue":"1","key":"2022111422093436300_bib1","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1162\/qss_a_00019","article-title":"Scopus as a curated, high-quality bibliometric data source for academic research in quantitative science studies","volume":"1","author":"Baas","year":"2020","journal-title":"Quantitative Science Studies"},{"key":"2022111422093436300_bib2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2105.14838","article-title":"Does the venue of scientific conferences leverage their impact? A large scale study on computer science conferences","author":"Bedogni","year":"2021","journal-title":"arXiv:2105.14838"},{"issue":"1","key":"2022111422093436300_bib3","doi-asserted-by":"publisher","first-page":"363","DOI":"10.1162\/qss_a_00018","article-title":"Web of Science as a data source for research on scientific and scholarly activity","volume":"1","author":"Birkle","year":"2020","journal-title":"Quantitative Science Studies"},{"key":"2022111422093436300_bib4","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2105.08599","article-title":"Can we assess research using open scientific knowledge graphs? A case study within the Italian National Scientific Qualification","author":"Bologna","year":"2021","journal-title":"arXiv:2105.08599"},{"key":"2022111422093436300_bib5","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.5025114","article-title":"Data and code for \u201cOpen bibliographic data and the Italian National Scientific Qualification: Measuring coverage of academic fields.\u201d","author":"Bologna","year":"2021","journal-title":"Zenodo"},{"key":"2022111422093436300_bib6","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2103.07942","article-title":"Do open citations inform the qualitative peer-review evaluation in research assessments? An analysis of the Italian National Scientific Qualification","author":"Bologna","year":"2021","journal-title":"arXiv:2103.07942"},{"key":"2022111422093436300_bib7","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1109\/COINFO.2009.66","article-title":"DataCite\u2014A global registration agency for research data","volume-title":"Fourth International Conference on Cooperation and Promotion of Information Resources in Science and Technology","author":"Brase","year":"2009"},{"issue":"4","key":"2022111422093436300_bib8","doi-asserted-by":"publisher","first-page":"35","DOI":"10.2478\/jdis-2020-0037","article-title":"Can Crossref citations replace Web of Science for research evaluation? The share of open citations","volume":"5","author":"Chudlarsk\u00fd","year":"2020","journal-title":"Journal of Data and Information Science"},{"key":"2022111422093436300_bib9","article-title":"January 2021 Public Data File from Crossref","author":"Crossref","year":"2021","journal-title":"Academic Torrents"},{"key":"2022111422093436300_bib10","article-title":"Redefinition of scientific disciplines (Rideterminazione dei settori concorsuali) (Prot. N. 159). Gazzetta Ufficiale Serie Generale n.137 del 14\/06\/2012\u2014Suppl. Ordinario n.119","author":"D. L. 2012","year":"2012"},{"key":"2022111422093436300_bib11","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1007\/978-3-030-02511-3_4","article-title":"Google Scholar as a data source for research assessment","volume-title":"Springer handbook of science and technology indicators","author":"Delgado L\u00f3pez-C\u00f3zar","year":"2019"},{"key":"2022111422093436300_bib12","doi-asserted-by":"publisher","DOI":"10.2777\/707440","volume-title":"Towards a reform of the research assessment system: Scoping report (KI-09-21-484-EN-N)","author":"Directorate-General for Research and Innovation, European Commission","year":"2021"},{"key":"2022111422093436300_bib13","first-page":"2133","article-title":"Open data to evaluate academic researchers: An experiment with the Italian Scientific Habilitation","volume-title":"Proceedings of the 17th International Conference on Scientometrics and Informetrics (ISSI 2019)","author":"Di Iorio","year":"2019"},{"issue":"3","key":"2022111422093436300_bib14","doi-asserted-by":"publisher","first-page":"1637","DOI":"10.1007\/s11192-016-2026-y","article-title":"Microsoft Academic (Search): A Phoenix arisen from the ashes?","volume":"108","author":"Harzing","year":"2016","journal-title":"Scientometrics"},{"issue":"1","key":"2022111422093436300_bib15","doi-asserted-by":"publisher","first-page":"341","DOI":"10.1007\/s11192-019-03114-y","article-title":"Two new kids on the block: How do Crossref and Dimensions compare with Google Scholar, Microsoft Academic, Scopus and the Web of Science?","volume":"120","author":"Harzing","year":"2019","journal-title":"Scientometrics"},{"issue":"1","key":"2022111422093436300_bib16","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1007\/s11192-016-2185-x","article-title":"Microsoft Academic: Is the phoenix getting wings?","volume":"110","author":"Harzing","year":"2017","journal-title":"Scientometrics"},{"issue":"3","key":"2022111422093436300_bib17","doi-asserted-by":"publisher","first-page":"1887","DOI":"10.1007\/s11192-017-2454-3","article-title":"Microsoft Academic is one year old: The phoenix is ready to leave the nest","volume":"112","author":"Harzing","year":"2017","journal-title":"Scientometrics"},{"issue":"2","key":"2022111422093436300_bib18","doi-asserted-by":"publisher","first-page":"1213","DOI":"10.1007\/s11192-019-03217-6","article-title":"Software review: COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations","volume":"121","author":"Heibi","year":"2019","journal-title":"Scientometrics"},{"issue":"1","key":"2022111422093436300_bib50","doi-asserted-by":"publisher","first-page":"414","DOI":"10.1162\/qss_a_00022","article-title":"Crossref: The sustainable source of community-owned scholarly metadata","volume":"1","author":"Hendricks","year":"2020","journal-title":"Quantitative Science Studies"},{"issue":"1","key":"2022111422093436300_bib19","doi-asserted-by":"publisher","first-page":"387","DOI":"10.1162\/qss_a_00020","article-title":"Dimensions: Bringing down barriers between scientometricians and data","volume":"1","author":"Herzog","year":"2020","journal-title":"Quantitative Science Studies"},{"issue":"2","key":"2022111422093436300_bib20","doi-asserted-by":"publisher","first-page":"445","DOI":"10.1162\/qss_a_00031","article-title":"Comparison of bibliographic data sources: Implications for the robustness of university rankings","volume":"1","author":"Huang","year":"2020","journal-title":"Quantitative Science Studies"},{"issue":"3","key":"2022111422093436300_bib21","doi-asserted-by":"publisher","first-page":"1551","DOI":"10.1007\/s11192-017-2535-3","article-title":"The coverage of Microsoft Academic: Analyzing the publication output of a university","volume":"113","author":"Hug","year":"2017","journal-title":"Scientometrics"},{"issue":"2","key":"2022111422093436300_bib22","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1162\/qss_c_00138","article-title":"A tipping point for open citation data","volume":"2","author":"Hutchins","year":"2021","journal-title":"Quantitative Science Studies"},{"key":"2022111422093436300_bib23","article-title":"Rules concerning the organization of the universities, academic employees and recruitment procedures, empowering the government to foster the quality and efficiency of the university system (Norme in materia di organizzazione delle universit\u00e0, di personale accademico e reclutamento, nonch\u00e9 delega al Governo per incentivare la qualit\u00e0 e l\u2019efficienza del sistema universitario). Gazzetta Ufficiale Serie Generale n.10 del 14\/01\/2011\u2014Suppl. Ordinario n.11","author":"L. 240\/2010","year":"2011"},{"issue":"9\/10","key":"2022111422093436300_bib24","doi-asserted-by":"publisher","DOI":"10.1045\/september2012-manghi","article-title":"OpenAIREplus: The European scholarly communication data infrastructure","volume":"18","author":"Manghi","year":"2012","journal-title":"D-Lib Magazine"},{"key":"2022111422093436300_bib25","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.4707307","article-title":"OpenAIRE Research Graph Dump (3.0)","author":"Manghi","year":"2021","journal-title":"Zenodo"},{"issue":"4","key":"2022111422093436300_bib26","doi-asserted-by":"publisher","first-page":"322","DOI":"10.1108\/PROG-08-2013-0045","article-title":"The D-NET software toolkit: A framework for the realization, maintenance, and operation of aggregative infrastructures","volume":"48","author":"Manghi","year":"2014","journal-title":"Program"},{"key":"2022111422093436300_bib27","article-title":"Coverage of open citation data approaches parity with Web of Science and Scopus [Blog]","author":"Mart\u00edn-Mart\u00edn","year":"2021","journal-title":"OpenCitations Blog"},{"issue":"4","key":"2022111422093436300_bib28","doi-asserted-by":"publisher","first-page":"1160","DOI":"10.1016\/j.joi.2018.09.002","article-title":"Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories","volume":"12","author":"Mart\u00edn-Mart\u00edn","year":"2018","journal-title":"Journal of Informetrics"},{"issue":"1","key":"2022111422093436300_bib29","doi-asserted-by":"publisher","first-page":"871","DOI":"10.1007\/s11192-020-03690-4","article-title":"Google Scholar, Microsoft Academic, Scopus, Dimensions, Web of Science, and OpenCitations\u2019 COCI: A multidisciplinary comparison of coverage via citations","volume":"126","author":"Mart\u00edn-Mart\u00edn","year":"2021","journal-title":"Scientometrics"},{"key":"2022111422093436300_bib30","article-title":"Microsoft Academic Graph (2020-01-23)","author":"Microsoft Academic","year":"2020","journal-title":"Internet Archive"},{"issue":"1","key":"2022111422093436300_bib31","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1007\/s11192-015-1765-5","article-title":"The journal coverage of Web of Science and Scopus: A comparative analysis","volume":"106","author":"Mongeon","year":"2016","journal-title":"Scientometrics"},{"key":"2022111422093436300_bib32","doi-asserted-by":"publisher","DOI":"10.6084\/m9.figshare.6741422","article-title":"COCI CSV dataset of all the citation data\u2014December 2020 dump","author":"OpenCitations","year":"2020","journal-title":"figshare"},{"key":"2022111422093436300_bib33","doi-asserted-by":"publisher","DOI":"10.6084\/m9.figshare.6741422.v14","article-title":"COCI CSV dataset of all the citation data\u2014March 2022 dump","author":"OpenCitations","year":"2022","journal-title":"figshare"},{"issue":"2","key":"2022111422093436300_bib34","doi-asserted-by":"publisher","first-page":"420","DOI":"10.3145\/epi.2018.mar.21","article-title":"Dimensions: Redescubriendo el ecosistema de la informaci\u00f3n cient\u00edfica","volume":"27","author":"Ordu\u00f1a-Malea","year":"2018","journal-title":"El Profesional de La Informaci\u00f3n"},{"issue":"1","key":"2022111422093436300_bib35","doi-asserted-by":"publisher","first-page":"253","DOI":"10.1007\/s11192-020-03397-6","article-title":"The practice of self-citations: A longitudinal study","volume":"123","author":"Peroni","year":"2020","journal-title":"Scientometrics"},{"issue":"6","key":"2022111422093436300_bib36","doi-asserted-by":"publisher","first-page":"5113","DOI":"10.1007\/s11192-021-03948-5","article-title":"The journal coverage of Web of Science, Scopus and Dimensions: A comparative analysis","volume":"126","author":"Singh","year":"2021","journal-title":"Scientometrics"},{"issue":"2","key":"2022111422093436300_bib37","doi-asserted-by":"publisher","first-page":"430","DOI":"10.1016\/j.joi.2018.03.006","article-title":"Dimensions: A competitor to Scopus and the Web of Science?","volume":"12","author":"Thelwall","year":"2018","journal-title":"Journal of Informetrics"},{"key":"2022111422093436300_bib38","doi-asserted-by":"publisher","DOI":"10.1038\/nature.2014.16269","article-title":"Google Scholar pioneer on search engine\u2019s future","author":"Van Noorden","year":"2014","journal-title":"Nature"},{"issue":"1","key":"2022111422093436300_bib39","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1162\/qss_a_00112","article-title":"Large-scale comparison of bibliographic data sources: Scopus, Web of Science, Dimensions, Crossref, and Microsoft Academic","volume":"2","author":"Visser","year":"2021","journal-title":"Quantitative Science Studies"},{"issue":"1","key":"2022111422093436300_bib40","doi-asserted-by":"publisher","first-page":"396","DOI":"10.1162\/qss_a_00021","article-title":"Microsoft Academic Graph: When experts are not enough","volume":"1","author":"Wang","year":"2020","journal-title":"Quantitative Science Studies"},{"key":"2022111422093436300_bib41","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3359115.3359119","article-title":"CiteSeerX: 20 years of service to scholarly big data","volume-title":"Proceedings of the Conference on Artificial Intelligence for Data Discovery and Reuse 2019","author":"Wu","year":"2019"},{"issue":"2","key":"2022111422093436300_bib42","doi-asserted-by":"publisher","first-page":"1097","DOI":"10.1007\/s11192-019-03311-9","article-title":"Nine million book items and eleven million citations: A study of book-based scholarly communication using OpenCitations","volume":"122","author":"Zhu","year":"2020","journal-title":"Scientometrics"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/3\/3\/512\/2057833\/qss_a_00203.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/3\/3\/512\/2057833\/qss_a_00203.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,14]],"date-time":"2022-11-14T17:10:50Z","timestamp":1668445850000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/3\/3\/512\/112761\/Open-bibliographic-data-and-the-Italian-National"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":43,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,11,1]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00203","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00203\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00203\/v2\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00203\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00203\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00203\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00203\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00203\/v2\/response1","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}