{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,20]],"date-time":"2026-01-20T03:50:30Z","timestamp":1768881030943,"version":"3.49.0"},"reference-count":72,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,11,21]],"date-time":"2022-11-21T00:00:00Z","timestamp":1668988800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,11,21]],"date-time":"2022-11-21T00:00:00Z","timestamp":1668988800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2023,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In the past, several works have investigated ways for combining quantitative and qualitative methods in research assessment exercises. Indeed, the Italian National Scientific Qualification (NSQ), i.e. the national assessment exercise which aims at deciding whether a scholar can apply to professorial academic positions as Associate Professor and Full Professor, adopts a quantitative and qualitative evaluation process: it makes use of bibliometrics followed by a peer-review process of candidates\u2019 CVs. The NSQ divides academic disciplines into two categories, i.e. <jats:italic>citation-based<\/jats:italic> disciplines (CDs) and <jats:italic>non-citation-based<\/jats:italic> disciplines (NDs), a division that affects the metrics used for assessing the candidates of that discipline in the first part of the process, which is based on bibliometrics. In this work, we aim at exploring whether citation-based metrics, calculated only considering open bibliographic and citation data, can support the human peer-review of NDs and yield insights on how it is conducted. To understand if and what citation-based (and, possibly, other) metrics provide relevant information, we created a series of machine learning models to replicate the decisions of the NSQ committees. As one of the main outcomes of our study, we noticed that the strength of the citational relationship between the candidate and the commission in charge of assessing his\/her CV seems to play a role in the peer-review phase of the NSQ of NDs.<\/jats:p>","DOI":"10.1007\/s11192-022-04581-6","type":"journal-article","created":{"date-parts":[[2022,11,21]],"date-time":"2022-11-21T05:03:07Z","timestamp":1669006987000},"page":"19-53","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Do open citations give insights on the qualitative peer-review evaluation in research assessments? An analysis of the Italian National Scientific Qualification"],"prefix":"10.1007","volume":"128","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3845-8266","authenticated-orcid":false,"given":"Federica","family":"Bologna","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6893-7452","authenticated-orcid":false,"given":"Angelo","family":"Di Iorio","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0530-4305","authenticated-orcid":false,"given":"Silvio","family":"Peroni","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6577-5606","authenticated-orcid":false,"given":"Francesco","family":"Poggi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,11,21]]},"reference":[{"key":"4581_CR1","doi-asserted-by":"publisher","unstructured":"Bologna, F., Di Iorio, A., Peroni, S., & Poggi, F. (2021). Data and code of the article \u201cDo open citations give insights the qualitative peer-review evaluation in research assessments? An analysis of the Italian National Scientific Qualification.\u201d https:\/\/doi.org\/10.5281\/ZENODO.4570782","DOI":"10.5281\/ZENODO.4570782"},{"key":"4581_CR2","doi-asserted-by":"publisher","DOI":"10.1093\/scipol\/scu086","author":"G Abramo","year":"2015","unstructured":"Abramo, G., D\u2019Angelo, C. A., & Rosati, F. (2015a). The determinants of academic career advancement: Evidence from Italy. Science and Public Policy. https:\/\/doi.org\/10.1093\/scipol\/scu086","journal-title":"Science and Public Policy"},{"issue":"4","key":"4581_CR3","doi-asserted-by":"publisher","first-page":"392","DOI":"10.1093\/reseval\/rvv019","volume":"24","author":"G Abramo","year":"2015","unstructured":"Abramo, G., D\u2019Angelo, C. A., & Rosati, F. (2015b). Selection committees for academic recruitment: Does gender matter? Research Evaluation, 24(4), 392\u2013404. https:\/\/doi.org\/10.1093\/reseval\/rvv019","journal-title":"Research Evaluation"},{"key":"4581_CR4","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1016\/j.labeco.2019.04.005","volume":"58","author":"M Bagues","year":"2019","unstructured":"Bagues, M., Sylos-Labini, M., & Zinovyeva, N. (2019). Connections in scientific committees and applicants\u2019 self-selection: Evidence from a natural randomized experiment. Labour Economics, 58, 81\u201397. https:\/\/doi.org\/10.1016\/j.labeco.2019.04.005","journal-title":"Labour Economics"},{"issue":"6","key":"4581_CR5","doi-asserted-by":"publisher","first-page":"829","DOI":"10.2307\/2657504","volume":"63","author":"S Baldi","year":"1998","unstructured":"Baldi, S. (1998). Normative versus social constructivist processes in the allocation of citations: A network-analytic model. American Sociological Review, 63(6), 829\u2013846. https:\/\/doi.org\/10.2307\/2657504","journal-title":"American Sociological Review"},{"key":"4581_CR6","doi-asserted-by":"publisher","DOI":"10.1108\/LHT-09-2021-0305","author":"L Bedogni","year":"2022","unstructured":"Bedogni, L., Cabri, G., Martoglia, R., & Poggi, F. (2022). Does the venue of scientific conferences leverage their impact? A large scale study on computer science conferences. Library Hi Tech. https:\/\/doi.org\/10.1108\/LHT-09-2021-0305","journal-title":"Library Hi Tech"},{"key":"4581_CR7","doi-asserted-by":"publisher","DOI":"10.1038\/srep10312","author":"ML Bermingham","year":"2015","unstructured":"Bermingham, M. L., Pong-Wong, R., Spiliopoulou, A., Hayward, C., Rudan, I., Campbell, H., Wright, A. F., Wilson, J. F., Agakov, F., Navarro, P., & Haley, C. S. (2015). Application of high-dimensional feature selection: Evaluation for genomic prediction in man. Scientific Reports. https:\/\/doi.org\/10.1038\/srep10312","journal-title":"Scientific Reports"},{"key":"4581_CR8","doi-asserted-by":"publisher","DOI":"10.1162\/qss_a_00203","volume-title":"Open bibliographic data and the Italian National Scientific Qualification: Measuring coverage of academic fields","author":"F Bologna","year":"2022","unstructured":"Bologna, F., Di Iorio, A., Peroni, S., & Poggi, F. (2022). Open bibliographic data and the Italian National Scientific Qualification: Measuring coverage of academic fields. In press. https:\/\/doi.org\/10.1162\/qss_a_00203"},{"key":"4581_CR9","doi-asserted-by":"publisher","DOI":"10.1515\/9783110308464-020","author":"L Bornmann","year":"2016","unstructured":"Bornmann, L. (2016). Scientific revolution in scientometrics: The broadening of impact from citation to societal. Theories of Informetrics and Scholarly Communication. https:\/\/doi.org\/10.1515\/9783110308464-020","journal-title":"Theories of Informetrics and Scholarly Communication"},{"issue":"12","key":"4581_CR10","doi-asserted-by":"publisher","first-page":"2389","DOI":"10.1002\/asi.21419","volume":"61","author":"KW Boyack","year":"2010","unstructured":"Boyack, K. W., & Klavans, R. (2010). Co-citation analysis, bibliographic coupling, and direct citation: Which citation approach represents the research front most accurately? Journal of the American Society for Information Science and Technology, 61(12), 2389\u20132404. https:\/\/doi.org\/10.1002\/asi.21419","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"1","key":"4581_CR11","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1016\/j.jfineco.2013.10.006","volume":"111","author":"J Brogaard","year":"2014","unstructured":"Brogaard, J., Engelberg, J., & Parsons, C. A. (2014). Networks and productivity: Causal evidence from editor rotations. Journal of Financial Economics, 111(1), 251\u2013270. https:\/\/doi.org\/10.1016\/j.jfineco.2013.10.006","journal-title":"Journal of Financial Economics"},{"issue":"2","key":"4581_CR12","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1007\/s11192-018-2973-6","volume":"118","author":"L Cai","year":"2019","unstructured":"Cai, L., Tian, J., Liu, J., Bai, X., Lee, I., Kong, X., & Xia, F. (2019). Scholarly impact assessment: A survey of citation weighting solutions. Scientometrics, 118(2), 453\u2013478. https:\/\/doi.org\/10.1007\/s11192-018-2973-6","journal-title":"Scientometrics"},{"issue":"3","key":"4581_CR13","doi-asserted-by":"publisher","first-page":"2071","DOI":"10.1007\/s11192-015-1762-8","volume":"105","author":"Y-W Chang","year":"2015","unstructured":"Chang, Y.-W., Huang, M.-H., & Lin, C.-W. (2015). Evolution of research subjects in library and information science based on keyword, bibliographical coupling, and co-citation analyses. Scientometrics, 105(3), 2071\u20132087. https:\/\/doi.org\/10.1007\/s11192-015-1762-8","journal-title":"Scientometrics"},{"issue":"7784","key":"4581_CR14","doi-asserted-by":"publisher","first-page":"596","DOI":"10.1038\/d41586-019-03662-6","volume":"575","author":"D Cheberkus","year":"2019","unstructured":"Cheberkus, D., & Nazarovets, S. (2019). Ukrainian open index maps local citations. Nature, 575(7784), 596\u2013596. https:\/\/doi.org\/10.1038\/d41586-019-03662-6","journal-title":"Nature"},{"key":"4581_CR15","doi-asserted-by":"publisher","DOI":"10.2478\/jdis-2020-0037","author":"T Chudlarsk\u00fd","year":"2020","unstructured":"Chudlarsk\u00fd, T., & Dvo\u0159\u00e1k, J. (2020). Can crossref citations replace web of science for research evaluation? The share of open citations. Journal of Data and Information Science. https:\/\/doi.org\/10.2478\/jdis-2020-0037","journal-title":"Journal of Data and Information Science"},{"issue":"1","key":"4581_CR16","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1162\/REST_a_00666","volume":"100","author":"T Colussi","year":"2018","unstructured":"Colussi, T. (2018). Social ties in academia: A friend is a treasure. The Review of Economics and Statistics, 100(1), 45\u201350. https:\/\/doi.org\/10.1162\/REST_a_00666","journal-title":"The Review of Economics and Statistics"},{"key":"4581_CR17","unstructured":"Ministerial Decree 159 (2012). Redefinition of scientific disciplines (Rideterminazione dei settori concorsuali)(Prot. n. 159), Gazzetta Ufficiale Serie Generale n.137 del 14\/06\/2012\u2014Suppl. Ordinario n.119. Retrived from https:\/\/www.gazzettaufficiale.it\/eli\/id\/2012\/06\/14\/12A06786\/sg"},{"key":"4581_CR18","unstructured":"Di Iorio, A., Poggi, F., & Peroni, S. (2019). Open data to evaluate academic researchers: An experiment with the Italian Scientific Habilitation. 17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings, 2, pp. 2133 - 21442. Scopus. Retrived from http:\/\/arxiv.org\/abs\/1902.03287"},{"issue":"2","key":"4581_CR19","doi-asserted-by":"publisher","first-page":"708","DOI":"10.1016\/j.joi.2019.03.018","volume":"13","author":"P Dondio","year":"2019","unstructured":"Dondio, P., Casnici, N., Grimaldo, F., Gilbert, N., & Squazzoni, F. (2019). The \u201cinvisible hand\u201d of peer review: The implications of author-referee networks on peer review in a scholarly journal. Journal of Informetrics, 13(2), 708\u2013716. https:\/\/doi.org\/10.1016\/j.joi.2019.03.018","journal-title":"Journal of Informetrics"},{"issue":"7","key":"4581_CR20","doi-asserted-by":"publisher","first-page":"e0133061","DOI":"10.1371\/journal.pone.0133061","volume":"10","author":"A Ebadi","year":"2015","unstructured":"Ebadi, A., & Schiffauerova, A. (2015). How to receive more funding for your research? Get connected to the right people! PLoS ONE, 10(7), e0133061. https:\/\/doi.org\/10.1371\/journal.pone.0133061","journal-title":"PLoS ONE"},{"issue":"7718","key":"4581_CR21","doi-asserted-by":"publisher","first-page":"290","DOI":"10.1038\/d41586-018-05968-3","volume":"560","author":"H Else","year":"2018","unstructured":"Else, H. (2018). How unpaywall is transforming open science. Nature, 560(7718), 290\u2013291. https:\/\/doi.org\/10.1038\/d41586-018-05968-3","journal-title":"Nature"},{"key":"4581_CR22","doi-asserted-by":"publisher","DOI":"10.1126\/science.aao0185","author":"S Fortunato","year":"2018","unstructured":"Fortunato, S., Bergstrom, C. T., B\u00f6rner, K., Evans, J. A., Helbing, D., Milojevi\u0107, S., Petersen, A. M., Radicchi, F., Sinatra, R., Uzzi, B., Vespignani, A., Waltman, L., Wang, D., & Barab\u00e1si, A.-L. (2018). Science of science. Science. https:\/\/doi.org\/10.1126\/science.aao0185","journal-title":"Science"},{"issue":"3","key":"4581_CR23","doi-asserted-by":"publisher","first-page":"1001","DOI":"10.1162\/qss_a_00057","volume":"1","author":"MF Fox","year":"2020","unstructured":"Fox, M. F. (2020). Gender, science, and academic rank: Key issues and approaches. Quantitative Science Studies, 1(3), 1001\u20131006. https:\/\/doi.org\/10.1162\/qss_a_00057","journal-title":"Quantitative Science Studies"},{"issue":"3","key":"4581_CR24","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.1162\/qss_a_00058","volume":"1","author":"K Frenken","year":"2020","unstructured":"Frenken, K. (2020). Geography of scientific knowledge: A proximity approach. Quantitative Science Studies, 1(3), 1007\u20131016. https:\/\/doi.org\/10.1162\/qss_a_00058","journal-title":"Quantitative Science Studies"},{"issue":"1","key":"4581_CR25","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1007\/s11192-010-0160-5","volume":"85","author":"LD Fu","year":"2010","unstructured":"Fu, L. D., & Aliferis, C. F. (2010). Using content-based and bibliometric features for machine learning models to predict citation counts in the biomedical literature. Scientometrics, 85(1), 257\u2013270. https:\/\/doi.org\/10.1007\/s11192-010-0160-5","journal-title":"Scientometrics"},{"issue":"3","key":"4581_CR26","doi-asserted-by":"publisher","first-page":"871","DOI":"10.1007\/s11192-013-0983-y","volume":"97","author":"LD Fu","year":"2013","unstructured":"Fu, L. D., Aphinyanaphongs, Y., & Aliferis, C. F. (2013). Computer models for identifying instrumental citations in the biomedical literature. Scientometrics, 97(3), 871\u2013882. https:\/\/doi.org\/10.1007\/s11192-013-0983-y","journal-title":"Scientometrics"},{"key":"4581_CR27","first-page":"84","volume":"1","author":"E Garfield","year":"1962","unstructured":"Garfield, E. (1962). Can citation indexing be automated? Essays of an Information Scientist, 1, 84\u201390.","journal-title":"Essays of an Information Scientist"},{"key":"4581_CR28","first-page":"1157","volume":"3","author":"I Guyon","year":"2003","unstructured":"Guyon, I., & Elisseeff, A. (2003). An introduction to variable and feature selection. Journal of Machine Learning Research, 3, 1157\u20131182.","journal-title":"Journal of Machine Learning Research"},{"issue":"2","key":"4581_CR29","doi-asserted-by":"publisher","first-page":"973","DOI":"10.1007\/s11192-018-2767-x","volume":"116","author":"S-U Hassan","year":"2018","unstructured":"Hassan, S.-U., Safder, I., Akram, A., & Kamiran, F. (2018). A novel machine-learning approach to measuring scientific knowledge flows using citation context analysis. Scientometrics, 116(2), 973\u2013996. https:\/\/doi.org\/10.1007\/s11192-018-2767-x","journal-title":"Scientometrics"},{"issue":"2","key":"4581_CR30","doi-asserted-by":"publisher","first-page":"1213","DOI":"10.1007\/s11192-019-03217-6","volume":"121","author":"I Heibi","year":"2019","unstructured":"Heibi, I., Peroni, S., & Shotton, D. (2019). Software review: COCI, the opencitations index of crossref open DOI-to-DOI citations. Scientometrics, 121(2), 1213\u20131228. https:\/\/doi.org\/10.1007\/s11192-019-03217-6","journal-title":"Scientometrics"},{"issue":"1","key":"4581_CR31","doi-asserted-by":"publisher","first-page":"414","DOI":"10.1162\/qss_a_00022","volume":"1","author":"G Hendricks","year":"2020","unstructured":"Hendricks, G., Tkaczyk, D., Lin, J., & Feeney, P. (2020). Crossref: The sustainable source of community-owned scholarly metadata. Quantitative Science Studies, 1(1), 414\u2013427. https:\/\/doi.org\/10.1162\/qss_a_00022","journal-title":"Quantitative Science Studies"},{"key":"4581_CR32","doi-asserted-by":"publisher","DOI":"10.1101\/750075","author":"CK Huang","year":"2019","unstructured":"Huang, C. K., Neylon, C., Brookes-Kenworthy, C., Hosking, R., Montgomery, L., Wilson, K., & Ozaygen, A. (2019). Comparison of bibliographic data sources: Implications for the robustness of university rankings. BioRxiv. https:\/\/doi.org\/10.1101\/750075","journal-title":"BioRxiv"},{"issue":"24","key":"4581_CR33","doi-asserted-by":"publisher","first-page":"3303","DOI":"10.1093\/bioinformatics\/btp585","volume":"25","author":"A Ib\u00e1\u00f1ez","year":"2009","unstructured":"Ib\u00e1\u00f1ez, A., Larra\u00f1aga, P., & Bielza, C. (2009). Predicting citation count of bioinformatics papers within four years of publication. Bioinformatics, 25(24), 3303\u20133309. https:\/\/doi.org\/10.1093\/bioinformatics\/btp585","journal-title":"Bioinformatics"},{"key":"4581_CR80","unstructured":"Japkowicz, N. (2000). The class imbalance problem: Significance and strategies. In proceedings of the 2000 international conference on artificial intelligence (ICAI), 111\u2013117."},{"key":"4581_CR34","doi-asserted-by":"publisher","DOI":"10.1177\/0165551520962775","author":"R Kleminski","year":"2020","unstructured":"Kleminski, R., Kazienko, P., & Kajdanowicz, T. (2020). Analysis of direct citation, co-citation and bibliographic coupling in scientific topic identification. Journal of Information Science. https:\/\/doi.org\/10.1177\/0165551520962775","journal-title":"Journal of Information Science"},{"key":"4581_CR35","volume-title":"The manufacture of knowledge: An essay on the constructivist and contextual nature of science","author":"K Knorr-Cetina","year":"1981","unstructured":"Knorr-Cetina, K. (1981). The manufacture of knowledge: An essay on the constructivist and contextual nature of science. Pergamon Press."},{"issue":"4","key":"4581_CR36","doi-asserted-by":"publisher","first-page":"522","DOI":"10.2307\/2071782","volume":"20","author":"K Knorr-Cetina","year":"1991","unstructured":"Knorr-Cetina, K. (1991). Merton sociology of science: The first and the last sociology of science. Contemporary Sociology, 20(4), 522\u2013526.","journal-title":"Contemporary Sociology"},{"issue":"5\u20136","key":"4581_CR37","doi-asserted-by":"publisher","first-page":"563","DOI":"10.1007\/BF02017071","volume":"15","author":"E Kranakis","year":"1989","unstructured":"Kranakis, E., & Leydesdorff, L. (1989). Teletraffic conferences: Studying a field of engineering science. Scientometrics, 15(5\u20136), 563\u2013591. https:\/\/doi.org\/10.1007\/BF02017071","journal-title":"Scientometrics"},{"key":"4581_CR81","doi-asserted-by":"publisher","unstructured":"Krawczyk, B. (2016). Learning from imbalanced data: open challenges and future directions. Prog Artif Intell 5, 221\u2013232. https:\/\/doi.org\/10.1007\/s13748-016-0094-0","DOI":"10.1007\/s13748-016-0094-0"},{"key":"4581_CR38","unstructured":"L. 240\/2010. (2011). Rules concerning the organization of the universities, academic employees and recruitment procedures, empowering the government to foster the quality and efficiency of the university system (Norme in materia di organizzazione delle universit\u00e0, di personale accademico e reclutamento, nonch\u00e9 delega al Governo per incentivare la qualit\u00e0 e l\u2019efficienza del sistema universitario), Gazzetta Ufficiale Serie Generale n.10 del 14\/01\/2011\u2014Suppl. Ordinario n.11. Retrived from https:\/\/www.gazzettaufficiale.it\/eli\/id\/2011\/01\/14\/011G0009\/sg"},{"key":"4581_CR39","volume-title":"Laboratory life: The social construction of scientific facts. SAGE library of social research","author":"B Latour","year":"1979","unstructured":"Latour, B., & Woolgar, S. (1979). Laboratory life: The social construction of scientific facts. SAGE library of social research. SAGE Publications, Inc."},{"issue":"5\u20136","key":"4581_CR40","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1007\/BF02017058","volume":"15","author":"L Leydesdorff","year":"1989","unstructured":"Leydesdorff, L. (1989). The relations between qualitative theory and scientometric methods in science and technology studies: Introduction to the topical issue. Scientometrics, 15(5\u20136), 333\u2013347. https:\/\/doi.org\/10.1007\/BF02017058","journal-title":"Scientometrics"},{"issue":"3","key":"4581_CR41","doi-asserted-by":"publisher","first-page":"918","DOI":"10.1162\/qss_e_00061","volume":"1","author":"L Leydesdorff","year":"2020","unstructured":"Leydesdorff, L., R\u00e0fols, I., & Milojevi\u0107, S. (2020). Bridging the divide between qualitative and quantitative science studies (Issue). Quantitative Science Studies, 1(3), 918\u20131348. https:\/\/doi.org\/10.1162\/qss_e_00061","journal-title":"Quantitative Science Studies"},{"issue":"5\u20136","key":"4581_CR42","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1007\/BF02017059","volume":"15","author":"T Luukkonen","year":"1989","unstructured":"Luukkonen, T. (1989). Publish in a visible journal of perish? Assessing citation performance of nordic cancer research. Scientometrics, 15(5\u20136), 349\u2013367. https:\/\/doi.org\/10.1007\/BF02017059","journal-title":"Scientometrics"},{"issue":"3","key":"4581_CR43","doi-asserted-by":"publisher","first-page":"1041","DOI":"10.1162\/qss_a_00062","volume":"1","author":"N Marres","year":"2020","unstructured":"Marres, N., & de Rijcke, S. (2020). From indicators to indicating interdisciplinarity: A participatory mapping methodology for research communities in-the-making. Quantitative Science Studies, 1(3), 1041\u20131055. https:\/\/doi.org\/10.1162\/qss_a_00062","journal-title":"Quantitative Science Studies"},{"key":"4581_CR44","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-020-03690-4","author":"A Mart\u00edn-Mart\u00edn","year":"2020","unstructured":"Mart\u00edn-Mart\u00edn, A., Thelwall, M., Orduna-Malea, E., & Delgado L\u00f3pez-C\u00f3zar, E. (2020). Google scholar, microsoft academic, scopus, dimensions, web of science, and opencitations\u2019 COCI: A multidisciplinary comparison of coverage via citations. Scientometrics. https:\/\/doi.org\/10.1007\/s11192-020-03690-4","journal-title":"Scientometrics"},{"key":"4581_CR45","doi-asserted-by":"publisher","first-page":"5633","DOI":"10.1007\/s11192-022-04485-5","volume":"127","author":"MC Martini","year":"2022","unstructured":"Martini, M. C., Pelle, E., Poggi, F., & Sciandra, A. (2022). The role of citation networks to explain academic promotions. An empirical analysis of the Italian national scientific qualification. Scientometrics, 127, 5633\u20135659. https:\/\/doi.org\/10.1007\/s11192-022-04485-5","journal-title":"Scientometrics"},{"key":"4581_CR46","volume-title":"The sociology of science: Theoretical and empirical investigations","author":"RK Merton","year":"1973","unstructured":"Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. University of Chicago press."},{"issue":"3","key":"4581_CR47","doi-asserted-by":"publisher","first-page":"693","DOI":"10.1016\/j.joi.2014.06.003","volume":"8","author":"S Milojevi\u0107","year":"2014","unstructured":"Milojevi\u0107, S., Sugimoto, C. R., Larivi\u00e8re, V., Thelwall, M., & Ding, Y. (2014). The role of handbooks in knowledge creation and diffusion: A case of science and technology studies. Journal of Informetrics, 8(3), 693\u2013709. https:\/\/doi.org\/10.1016\/j.joi.2014.06.003","journal-title":"Journal of Informetrics"},{"issue":"3","key":"4581_CR48","doi-asserted-by":"publisher","first-page":"1292","DOI":"10.1111\/caje.12460","volume":"53","author":"R Miniaci","year":"2020","unstructured":"Miniaci, R., & Pezzoni, M. (2020). Social connections and editorship in economics. Canadian Journal of Economics\/revue Canadienne D\u2019\u00e9conomique, 53(3), 1292\u20131317. https:\/\/doi.org\/10.1111\/caje.12460","journal-title":"Canadian Journal of Economics\/revue Canadienne D\u2019\u00e9conomique"},{"issue":"2","key":"4581_CR49","doi-asserted-by":"publisher","first-page":"539","DOI":"10.1007\/s11192-018-2988-z","volume":"118","author":"AG Nuzzolese","year":"2019","unstructured":"Nuzzolese, A. G., Ciancarini, P., Gangemi, A., Peroni, S., Poggi, F., & Presutti, V. (2019). Do altmetrics work for assessing research quality? Scientometrics, 118(2), 539\u2013562. https:\/\/doi.org\/10.1007\/s11192-018-2988-z","journal-title":"Scientometrics"},{"issue":"1","key":"4581_CR50","doi-asserted-by":"publisher","first-page":"253","DOI":"10.1007\/s11192-020-03397-6","volume":"123","author":"S Peroni","year":"2020","unstructured":"Peroni, S., Ciancarini, P., Gangemi, A., Nuzzolese, A. G., Poggi, F., & Presutti, V. (2020). The practice of self-citations: A longitudinal study. Scientometrics, 123(1), 253\u2013282. https:\/\/doi.org\/10.1007\/s11192-020-03397-6","journal-title":"Scientometrics"},{"issue":"1","key":"4581_CR51","doi-asserted-by":"publisher","first-page":"428","DOI":"10.1162\/qss_a_00023","volume":"1","author":"S Peroni","year":"2020","unstructured":"Peroni, S., & Shotton, D. (2020). Opencitations, an infrastructure organization for open scholarship. Quantitative Science Studies, 1(1), 428\u2013444. https:\/\/doi.org\/10.1162\/qss_a_00023","journal-title":"Quantitative Science Studies"},{"key":"4581_CR52","doi-asserted-by":"publisher","DOI":"10.7717\/peerj-cs.199","volume":"5","author":"F Poggi","year":"2019","unstructured":"Poggi, F., Ciancarini, P., Gangemi, A., Nuzzolese, A. G., Peroni, S., & Presutti, V. (2019). Predicting the results of evaluation procedures of academics. PeerJ Computer Science, 5, e199. https:\/\/doi.org\/10.7717\/peerj-cs.199","journal-title":"PeerJ Computer Science"},{"key":"4581_CR53","doi-asserted-by":"publisher","first-page":"8","DOI":"10.3389\/frma.2017.00008","volume":"2","author":"J Portenoy","year":"2017","unstructured":"Portenoy, J., Hullman, J., & West, J. D. (2017). Leveraging citation networks to visualize scholarly influence over time. Frontiers in Research Metrics and Analytics, 2, 8. https:\/\/doi.org\/10.3389\/frma.2017.00008","journal-title":"Frontiers in Research Metrics and Analytics"},{"issue":"3","key":"4581_CR54","doi-asserted-by":"publisher","first-page":"160","DOI":"10.18352\/lq.8110","volume":"22","author":"N Rettberg","year":"2012","unstructured":"Rettberg, N., & Schmidt, B. (2012). OpenAIRE - building a collaborative open access infrastructure for European researchers. LIBER Quarterly, 22(3), 160. https:\/\/doi.org\/10.18352\/lq.8110","journal-title":"LIBER Quarterly"},{"issue":"2","key":"4581_CR55","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2019.102168","volume":"57","author":"P Savov","year":"2020","unstructured":"Savov, P., Jatowt, A., & Nielek, R. (2020). Identifying breakthrough scientific papers. Information Processing & Management, 57(2), 102168. https:\/\/doi.org\/10.1016\/j.ipm.2019.102168","journal-title":"Information Processing & Management"},{"issue":"1","key":"4581_CR56","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1023\/B:SCIE.0000027310.68393.bc","volume":"60","author":"H Small","year":"2004","unstructured":"Small, H. (2004). On the shoulders of Robert Merton: Towards a normative theory of citation. Scientometrics, 60(1), 71\u201379. https:\/\/doi.org\/10.1023\/B:SCIE.0000027310.68393.bc","journal-title":"Scientometrics"},{"issue":"1","key":"4581_CR57","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/0378-8733(89)90016-6","volume":"11","author":"K Stephenson","year":"1989","unstructured":"Stephenson, K., & Zelen, M. (1989). Rethinking centrality: Methods and examples. Social Networks, 11(1), 1\u201337. https:\/\/doi.org\/10.1016\/0378-8733(89)90016-6","journal-title":"Social Networks"},{"issue":"9","key":"4581_CR58","doi-asserted-by":"publisher","first-page":"1825","DOI":"10.1016\/j.respol.2018.06.014","volume":"47","author":"M Teplitskiy","year":"2018","unstructured":"Teplitskiy, M., Acuna, D., Elamrani-Raoult, A., K\u00f6rding, K., & Evans, J. (2018). The sociology of scientific validity: How professional networks shape judgement in peer review. Research Policy, 47(9), 1825\u20131841. https:\/\/doi.org\/10.1016\/j.respol.2018.06.014","journal-title":"Research Policy"},{"issue":"1","key":"4581_CR60","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-019-41695-z","volume":"9","author":"VA Traag","year":"2019","unstructured":"Traag, V. A., Waltman, L., & Van Eck, N. J. (2019). From Louvain to Leiden: Guaranteeing well-connected communities. Scientific Reports, 9(1), 1\u201312. https:\/\/doi.org\/10.1038\/s41598-019-41695-z","journal-title":"Scientific Reports"},{"key":"4581_CR61","doi-asserted-by":"publisher","DOI":"10.7717\/peerj.5707","volume":"6","author":"JR Tregellas","year":"2018","unstructured":"Tregellas, J. R., Smucny, J., Rojas, D. C., & Legget, K. T. (2018). Predicting academic career outcomes by predoctoral publication record. PeerJ, 6, e5707. https:\/\/doi.org\/10.7717\/peerj.5707","journal-title":"PeerJ"},{"issue":"2","key":"4581_CR62","doi-asserted-by":"publisher","first-page":"523","DOI":"10.1007\/s11192-009-0146-3","volume":"84","author":"NJ van Eck","year":"2010","unstructured":"van Eck, N. J., & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics, 84(2), 523\u2013538. https:\/\/doi.org\/10.1007\/s11192-009-0146-3","journal-title":"Scientometrics"},{"key":"4581_CR63","unstructured":"Visser, M., van Eck, N. J., & Waltman, L. (2020). Large-scale comparison of bibliographic data sources: Scopus, web of Science, dimensions, crossref, and microsoft academic. Retrived from ArXiv:2005.10732 [Cs]. http:\/\/arxiv.org\/abs\/2005.10732"},{"issue":"1","key":"4581_CR64","doi-asserted-by":"publisher","first-page":"396","DOI":"10.1162\/qss_a_00021","volume":"1","author":"K Wang","year":"2020","unstructured":"Wang, K., Shen, Z., Huang, C., Wu, C.-H., Dong, Y., & Kanakia, A. (2020). Microsoft academic graph: When experts are not enough. Quantitative Science Studies, 1(1), 396\u2013413. https:\/\/doi.org\/10.1162\/qss_a_00021","journal-title":"Quantitative Science Studies"},{"issue":"1","key":"4581_CR65","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1023\/B:SCIE.0000027313.91401.9b","volume":"60","author":"HD White","year":"2004","unstructured":"White, H. D. (2004). Reward, persuasion, and the Sokal Hoax: A study in citation identities. Scientometrics, 60(1), 93\u2013120. https:\/\/doi.org\/10.1023\/B:SCIE.0000027313.91401.9b","journal-title":"Scientometrics"},{"key":"4581_CR66","first-page":"87","volume-title":"Handbook of science and technology studies","author":"S Wyatt","year":"2017","unstructured":"Wyatt, S., Milojevi\u0107, S., Park, H. W., & Leydesdorff, L. (2017). The intellectual and practical contributions of scientometrics. In U. Felt, R. Fouch\u00e9, & L. Smith-Doerr (Eds.), Handbook of science and technology studies (pp. 87\u2013112). MIT Press."},{"issue":"7","key":"4581_CR67","doi-asserted-by":"publisher","first-page":"1313","DOI":"10.1002\/asi.22680","volume":"63","author":"E Yan","year":"2012","unstructured":"Yan, E., & Ding, Y. (2012). Scholarly network similarities: How bibliographic coupling networks, citation networks, cocitation networks, topical networks, coauthorship networks, and coword networks relate to each other. Journal of the American Society for Information Science and Technology, 63(7), 1313\u20131326. https:\/\/doi.org\/10.1002\/asi.22680","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"8","key":"4581_CR68","doi-asserted-by":"publisher","first-page":"1498","DOI":"10.1002\/asi.21556","volume":"62","author":"E Yan","year":"2011","unstructured":"Yan, E., & Sugimoto, C. R. (2011). Institutional interactions: Exploring social, cognitive, and geographic relationships between institutions as demonstrated through citation networks. Journal of the American Society for Information Science and Technology, 62(8), 1498\u20131514. https:\/\/doi.org\/10.1002\/asi.21556","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"1","key":"4581_CR69","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1007\/s11192-014-1358-8","volume":"101","author":"G Zhang","year":"2014","unstructured":"Zhang, G., Liu, L., Feng, Y., Shao, Z., & Li, Y. (2014). Cext-N index: A network node centrality measure for collaborative relationship distribution. Scientometrics, 101(1), 291\u2013307. https:\/\/doi.org\/10.1007\/s11192-014-1358-8","journal-title":"Scientometrics"},{"issue":"2","key":"4581_CR70","doi-asserted-by":"publisher","first-page":"1097","DOI":"10.1007\/s11192-019-03311-9","volume":"122","author":"Y Zhu","year":"2019","unstructured":"Zhu, Y., Yan, E., Peroni, S., & Che, C. (2019). Nine million book items and eleven million citations: A study of book-based scholarly communication using opencitations. Scientometrics, 122(2), 1097\u20131112. https:\/\/doi.org\/10.1007\/s11192-019-03311-9","journal-title":"Scientometrics"},{"issue":"2","key":"4581_CR71","doi-asserted-by":"publisher","first-page":"264","DOI":"10.1257\/app.20120337","volume":"7","author":"N Zinovyeva","year":"2015","unstructured":"Zinovyeva, N., & Bagues, M. (2015). The role of connections in academic promotions. American Economic Journal: Applied Economics, 7(2), 264\u2013292. https:\/\/doi.org\/10.1257\/app.20120337","journal-title":"American Economic Journal: Applied Economics"}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-022-04581-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11192-022-04581-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-022-04581-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,1]],"date-time":"2023-02-01T13:20:51Z","timestamp":1675257651000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11192-022-04581-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,21]]},"references-count":72,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,1]]}},"alternative-id":["4581"],"URL":"https:\/\/doi.org\/10.1007\/s11192-022-04581-6","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,21]]},"assertion":[{"value":"8 March 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 October 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 November 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors did not receive support from any organization for the submitted work. The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}