{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T20:04:52Z","timestamp":1760385892448,"version":"3.37.3"},"reference-count":54,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2022,3,12]],"date-time":"2022-03-12T00:00:00Z","timestamp":1647043200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,3,12]],"date-time":"2022-03-12T00:00:00Z","timestamp":1647043200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004271","name":"Sapienza Universit\u00e0 di Roma","doi-asserted-by":"publisher","award":["PH11715C8239C105","RM11916B8853C925"],"award-info":[{"award-number":["PH11715C8239C105","RM11916B8853C925"]}],"id":[{"id":"10.13039\/501100004271","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In this paper we answer the question of how evaluation should be by proposing a good evaluation of research practices. A good evaluation of research practices, intended as social practices \u00e0 la MacIntyre, should take into account the stable motivations and the traits of the characters (i.e. the<jats:italic>virtues<\/jats:italic>) of researchers. We also show that a good evaluation is also<jats:italic>just<\/jats:italic>, beyond the sense of<jats:italic>fairness<\/jats:italic>, as working on good research practices implies keep into account a<jats:italic>broader sense of justice<\/jats:italic>. After that, we propose the development of a knowledge base for the assessment of \u201cgood\u201d evaluations of research practices to implement a questionnaire for the assessment of researchers\u2019 virtues. Although the latter is a challenging task, the use of ontologies and taxonomic knowledge, and the reasoning algorithms that can draw inferences on the basis of such knowledge represents a way for testing the consistency of the information reported in the questionnaire and to analyse correctly and coherently how the data is gathered through it. Finally, we describe the potential application usefulness of our proposal for the reform of current research assessment systems.<\/jats:p>","DOI":"10.1007\/s11192-022-04329-2","type":"journal-article","created":{"date-parts":[[2022,3,12]],"date-time":"2022-03-12T04:35:25Z","timestamp":1647059725000},"page":"7127-7146","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["How should evaluation be? Is a good evaluation of research also just? Towards the implementation of good evaluation"],"prefix":"10.1007","volume":"127","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4825-0071","authenticated-orcid":false,"given":"Cinzia","family":"Daraio","sequence":"first","affiliation":[]},{"given":"Alessio","family":"Vaccari","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,3,12]]},"reference":[{"key":"4329_CR2","volume-title":"Passion and value in Hume\u2019s treatise","author":"PS Ardal","year":"1966","unstructured":"Ardal, P. S. (1966). Passion and value in Hume\u2019s treatise. Edinburgh University Press."},{"key":"4329_CR3","doi-asserted-by":"crossref","unstructured":"Aristotle (2000, revised edition 2014). Nicomachean Ethics, R. Crisp (Eds.), Cambridge University Press.","DOI":"10.1017\/CBO9780511802058"},{"volume-title":"The description logic handbook: Theory, implementation and applications","year":"2003","key":"4329_CR4","unstructured":"Baader, F., Calvanese, D., McGuinness, D., Patel-Schneider, P., & Nardi, D. (Eds.). (2003). The description logic handbook: Theory, implementation and applications. Cambridge University Press."},{"key":"4329_CR5","doi-asserted-by":"publisher","DOI":"10.4159\/9780674020382","volume-title":"A progress of sentiments: Reflections on Hume\u2019s treatise","author":"A Baier","year":"1991","unstructured":"Baier, A. (1991). A progress of sentiments: Reflections on Hume\u2019s treatise. Harvard University Press."},{"key":"4329_CR6","volume-title":"The oxford handbook of compassion science","author":"CD Batson","year":"2017","unstructured":"Batson, C. D. (2017). The empathy-altruism hypothesis: What and so what? In E. M. Sepp\u00e4l\u00e4, E. Simon-Thomas, S. L. Brown, M. C. Worline, C. D. Cameron, & J. R. Doty (Eds.), The oxford handbook of compassion science. Oxford: Oxford University Press."},{"key":"4329_CR7","doi-asserted-by":"crossref","unstructured":"Borodin, A. V., & Zavyalova, Y. V. (2016) An ontology-based semantic design of the survey questionnaires. In 2016 19th Conference of open innovations association (FRUCT) (pp. 10\u201315). IEEE.","DOI":"10.23919\/FRUCT.2016.7892177"},{"issue":"1","key":"4329_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s11948-016-9782-1","volume":"23","author":"M Burget","year":"2017","unstructured":"Burget, M., Bardone, E., & Pedaste, M. (2017). Definitions and conceptual dimensions of responsible research and innovation: A literature review. Science and Engineering Ethics, 23(1), 1\u201319.","journal-title":"Science and Engineering Ethics"},{"key":"4329_CR9","doi-asserted-by":"publisher","DOI":"10.1093\/acprof:oso\/9780199268443.001.0001","volume-title":"Hume\u2019s morality: Feeling and fabrication","author":"R Cohon","year":"2008","unstructured":"Cohon, R. (2008). Hume\u2019s morality: Feeling and fabrication. Oxford University Press."},{"key":"4329_CR1","doi-asserted-by":"publisher","unstructured":"Curry, S., de Rijcke, S., Hatch, A., Pillay, D. G., van der Weijden, I., & Wilsdon, J. (2020). The changing role of funders in responsible research assessment: progress, obstacles and the way ahead. Working Paper. Research on Research Institute (RoRI). https:\/\/doi.org\/10.6084\/m9.figshare.13227914.v1.","DOI":"10.6084\/m9.figshare.13227914.v1"},{"key":"4329_CR10","unstructured":"European Commission (2021). Towards a reform of the research assessment system: scoping report. November 2021, Bruxelles, ISBN 978\u201392\u201376\u201343463\u20134"},{"key":"4329_CR11","doi-asserted-by":"publisher","DOI":"10.1515\/9780804778121","volume-title":"The evaluation society","author":"P Dahler-Larsen","year":"2011","unstructured":"Dahler-Larsen, P. (2011). The evaluation society. Stanford Univ. Press."},{"key":"4329_CR12","unstructured":"Daraio C., Vaccari, A. (2021), An Ontology-based semantic design for good evaluations of research practices, 18th International Conference on Scientometrics & Informetrics (ISSI2021), 12\u201315 July 2021 Proceedings, edited by W. Gl\u00e4nzel, S. Heeffer, P.-S. Chi & R. Rousseau, ISBN 9789080328228, July 2021, pp. 305\u2013316."},{"key":"4329_CR13","doi-asserted-by":"publisher","first-page":"1053","DOI":"10.1007\/s11192-020-03658-4","volume":"125","author":"C Daraio","year":"2020","unstructured":"Daraio, C., & Vaccari, A. (2020). Using normative ethics for building a good evaluation of research practices: Towards the assessment of researcher\u2019s virtues. Scientometrics, 125, 1053\u20131075.","journal-title":"Scientometrics"},{"issue":"2015","key":"4329_CR14","first-page":"83","volume":"1","author":"P Donatelli","year":"2015","unstructured":"Donatelli, P. (2015). La giustizia fra deontologismo e consequenzialismo. Parolechiave, 1(2015), 83\u201396.","journal-title":"Parolechiave"},{"key":"4329_CR15","volume-title":"Model driven engineering and ontology development","author":"D Ga\u0161evi\u0107","year":"2009","unstructured":"Ga\u0161evi\u0107, D., Djuri\u0107, D., & Deved\u017ei\u0107, V. (2009). Model driven engineering and ontology development (2nd ed.). Springer.","edition":"2"},{"key":"4329_CR16","volume-title":"The three careers of an academic","author":"J Gl\u00e4ser","year":"2015","unstructured":"Gl\u00e4ser, J., & Laudel, G. (2015). The three careers of an academic. Center for Technology and Society."},{"key":"4329_CR17","first-page":"2021","volume":"22","author":"J Grant","year":"2021","unstructured":"Grant, J. (2021). Academic incentives and research impact: Developing reward and recognition systems to better people\u2019s lives. Paradigm Project, AcademyHealth, 22, 2021\u20132002.","journal-title":"Paradigm Project, AcademyHealth"},{"key":"4329_CR18","unstructured":"Hochschild, J. L. (2009). Conducting intensive interviews and elite interviews. In Workshop on interdisciplinary standards for systematic qualitative research (pp. 124\u2013127). Washington, DC: National Science Foundation."},{"issue":"1","key":"4329_CR19","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1007\/s41463-018-0035-0","volume":"3","author":"S Hormann","year":"2018","unstructured":"Hormann, S. (2018). Exploring resilience: In the face of trauma. Humanistic Management Journal, 3(1), 91\u2013104.","journal-title":"Humanistic Management Journal"},{"issue":"7366","key":"4329_CR20","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/477529a","volume":"477","author":"JP Ioannidis","year":"2011","unstructured":"Ioannidis, J. P. (2011). Fund people not projects. Nature, 477(7366), 529\u2013531.","journal-title":"Nature"},{"key":"4329_CR21","volume-title":"Doing interviews","author":"S Kvale","year":"2008","unstructured":"Kvale, S. (2008). Doing interviews. Sage."},{"issue":"2","key":"4329_CR22","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1007\/BF00138348","volume":"16","author":"RC Larson","year":"1983","unstructured":"Larson, R. C., & Berliner, L. (1983). On evaluating evaluations. Policy Sciences, 16(2), 147\u2013163.","journal-title":"Policy Sciences"},{"key":"4329_CR23","volume-title":"After virtue","author":"A MacIntyre","year":"1985","unstructured":"MacIntyre, A. (1985). After virtue. Duckworth."},{"issue":"3","key":"4329_CR24","doi-asserted-by":"publisher","first-page":"331","DOI":"10.5840\/monist197962319","volume":"62","author":"J McDowell","year":"1979","unstructured":"McDowell, J. (1979). Virtue and reason. The Monist, 62(3), 331\u2013350.","journal-title":"The Monist"},{"key":"4329_CR25","volume-title":"Mill: Texts, commentaries","author":"JS Mill","year":"1997","unstructured":"Mill, J. S., & Ryan, A. (1997). Mill: Texts, commentaries. Norton Critical Edition."},{"volume-title":"Cognitive interviewing methodology","year":"2014","key":"4329_CR26","unstructured":"Miller, K., Chepp, V., Willson, S., & Padilla, J. L. (Eds.). (2014). Cognitive interviewing methodology. John Wiley & Sons."},{"issue":"1","key":"4329_CR27","doi-asserted-by":"publisher","first-page":"1","DOI":"10.29024\/sar.8","volume":"2","author":"HF Moed","year":"2020","unstructured":"Moed, H. F. (2020). Appropriate use of metrics in research assessment of autonomous academic institutions. Scholarly Assessment Reports, 2(1), 1.","journal-title":"Scholarly Assessment Reports"},{"key":"4329_CR28","volume-title":"Existentialists and mystics: Writings on philosophy and literature","author":"I Murdoch","year":"1998","unstructured":"Murdoch, I. (1998). Existentialists and mystics: Writings on philosophy and literature. Allen Lane\/the Penguin Press."},{"key":"4329_CR29","volume-title":"Reasons and persons","author":"D Parfit","year":"1984","unstructured":"Parfit, D. (1984). Reasons and persons. Oxford University Press."},{"issue":"2","key":"4329_CR30","first-page":"563","volume":"16","author":"SE Rabionet","year":"2011","unstructured":"Rabionet, S. E. (2011). How I learned to design and conduct semi-structured interviews: An ongoing and continuous journey. Qualitative Report, 16(2), 563\u2013566.","journal-title":"Qualitative Report"},{"key":"4329_CR31","doi-asserted-by":"crossref","unstructured":"Rawls, A. (1971). Theory of justice. Original ed. Cambridge, Mass. Web.","DOI":"10.4159\/9780674042605"},{"key":"4329_CR32","unstructured":"Saenen, B., Hatch, A., Curry, S., Proudman, V., Lakoduk, A. (2021). Reimagining academic career assessment: Stories of innovation and change. Case study report. DORA, EUA, SPARC Europe."},{"key":"4329_CR33","first-page":"85","volume":"12","author":"MJ Sandel","year":"2010","unstructured":"Sandel, M. J. (2010). Justice: What\u2019s the right thing to do. The Hedgehog Review, 12, 85\u201390.","journal-title":"The Hedgehog Review"},{"key":"4329_CR34","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1007\/978-3-030-02511-3_12","volume-title":"Springer handbook of science and technology indicators","author":"A Schubert","year":"2019","unstructured":"Schubert, A., & Schubert, G. (2019). All along the h-index-related literature: A guided tour. In W. Gl\u00e4nzel, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 301\u2013334). Springer."},{"key":"4329_CR35","unstructured":"Science Europe (2020). Position statement and recommendations on research assessment processes, July 2020. https:\/\/www.scienceeurope.org\/media\/3twjxim0\/se-position-statement-research-assessment-processes.pdf."},{"key":"4329_CR36","volume-title":"The idea of justice","author":"A Sen","year":"2010","unstructured":"Sen, A. (2010). The idea of justice. Penguin."},{"key":"4329_CR37","unstructured":"Sherimon, P.C., Vinu, P.V., Krishnan, R., Takroni, Y., (2013). Developing a survey questionnaire ontology for the decision support system in the domain of hypertension, IEEE South East conference, April 4\u20137, Florida, U.S."},{"key":"4329_CR38","doi-asserted-by":"crossref","unstructured":"Sherimon, P. C., Vinu, P. V., Krishnan, R., Takroni, Y., AlKaabi, Y., & AlFars, Y. (2014). Adaptive questionnaire ontology in gathering patient medical history in diabetes domain. In Proceedings of the first international conference on advanced data and information engineering (DaEng-2013) (pp. 453\u2013460). Springer.","DOI":"10.1007\/978-981-4585-18-7_51"},{"key":"4329_CR39","volume-title":"One world. L\u2019etica della globalizzazione","author":"P Singer","year":"2003","unstructured":"Singer, P. (2003). One world. L\u2019etica della globalizzazione. Einaudi."},{"key":"4329_CR40","unstructured":"Snow, N. (2014). Virtue intelligence, unpublished paper presented for the conference \u201cCan virtue be measured?\u201d held by the jubilee centre for character & value, 9th\u201311th January 2014, https:\/\/www.jubileecentre.ac.uk\/485\/conferences\/can-virtue-be-measured."},{"key":"4329_CR41","doi-asserted-by":"publisher","DOI":"10.4324\/9780203880579","volume-title":"Virtue as social intelligence: An empirically grounded theory","author":"N Snow","year":"2010","unstructured":"Snow, N. (2010). Virtue as social intelligence: An empirically grounded theory. Routledge."},{"key":"4329_CR42","first-page":"130","volume-title":"Aristotelian ethics in contemporary perspective","author":"N Snow","year":"2012","unstructured":"Snow, N. (2012). Notes toward an empirical psychology of virtue: Exploring the personality scaffolding of virtue. In J. Peters (Ed.), Aristotelian ethics in contemporary perspective (pp. 130\u2013144). Routledge."},{"key":"4329_CR43","doi-asserted-by":"publisher","DOI":"10.1093\/0199253889.001.0001","volume-title":"Virtue ethics: A pluralistic view","author":"C Swanton","year":"2003","unstructured":"Swanton, C. (2003). Virtue ethics: A pluralistic view. Clarendon Press."},{"key":"4329_CR44","unstructured":"Sweeney, D. (2021). The science of 21st century science, Keynote at ISSI2021 Conference, 12 July 2021."},{"key":"4329_CR45","unstructured":"Talbert, M. (2019). \u201cMoral responsibility\u201d, the stanford encyclopedia of philosophy (Winter 2019 Edition), Zalta, Edward N. (Eds.). Retrieved June 17, 2021 from https:\/\/plato.stanford.edu\/archives\/win2019\/entries\/moral-responsibility\/."},{"key":"4329_CR46","doi-asserted-by":"crossref","first-page":"541","DOI":"10.1002\/0470013494.ch26","volume-title":"Handbook of cognition and emotion","author":"JP Tangney","year":"1999","unstructured":"Tangney, J. P. (1999). The self-conscious emotions: Shame, guilt, embarrassment and pride. In T. Dalgleish & M. J. Powers (Eds.), Handbook of cognition and emotion (pp. 541\u2013568). Wiley."},{"key":"4329_CR47","doi-asserted-by":"publisher","DOI":"10.1093\/acprof:oso\/9780198729525.001.0001","volume-title":"Reflecting subjects: Passion, sympathy, and society in Hume\u2019s philosophy","author":"J Taylor","year":"2015","unstructured":"Taylor, J. (2015). Reflecting subjects: Passion, sympathy, and society in Hume\u2019s philosophy. Oxford University Press."},{"key":"4329_CR48","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1007\/978-3-030-02511-3_14","volume-title":"Springer handbook of science and technology indicators","author":"L Wildgaard","year":"2019","unstructured":"Wildgaard, L. (2019). An overview of author-level indicators of research performance. In W. Gl\u00e4nzel, H. F. Moed, U. Schmoch, & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 361\u2013396). Springer International Publishing."},{"key":"4329_CR49","volume-title":"Truth and truthfulness: An essay in genealogy","author":"B Williams","year":"2002","unstructured":"Williams, B. (2002). Truth and truthfulness: An essay in genealogy. Princeton University Press."},{"key":"4329_CR50","volume-title":"Cognitive interviewing: A tool for improving questionnaire design","author":"GB Willis","year":"2004","unstructured":"Willis, G. B. (2004). Cognitive interviewing: A tool for improving questionnaire design. Sage publications."},{"key":"4329_CR51","doi-asserted-by":"publisher","DOI":"10.13140\/RG.2.1.4929.1363","author":"J Wilsdon","year":"2015","unstructured":"Wilsdon, J., et al. (2015). The metric tide: Report of the independent review of the role of metrics. Research Assessment and Management. https:\/\/doi.org\/10.13140\/RG.2.1.4929.1363","journal-title":"Research Assessment and Management"},{"issue":"3","key":"4329_CR52","doi-asserted-by":"publisher","first-page":"306","DOI":"10.1177\/0165551509360123","volume":"36","author":"DC Wimalasuriya","year":"2010","unstructured":"Wimalasuriya, D. C., & Dou, D. (2010). Ontology-based information extraction: An introduction and a survey of current approaches. Journal of Information Science, 36(3), 306\u2013323.","journal-title":"Journal of Information Science"},{"key":"4329_CR53","volume-title":"Writing up qualitative research","author":"HF Wolcott","year":"2008","unstructured":"Wolcott, H. F. (2008). Writing up qualitative research. Sage Publications."},{"issue":"3","key":"4329_CR54","first-page":"48","volume":"9","author":"P Wouters","year":"2013","unstructured":"Wouters, P., Gl\u00e4nzel, W., Gl\u00e4ser, J., & Rafols, I. (2013). The dilemmas of performance indicators of individual researchers\u2013An urgent debate in bibliometrics. ISSI Newsletter, 9(3), 48\u201353.","journal-title":"ISSI Newsletter"}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-022-04329-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11192-022-04329-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-022-04329-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,18]],"date-time":"2023-11-18T19:12:11Z","timestamp":1700334731000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11192-022-04329-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,12]]},"references-count":54,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["4329"],"URL":"https:\/\/doi.org\/10.1007\/s11192-022-04329-2","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"type":"print","value":"0138-9130"},{"type":"electronic","value":"1588-2861"}],"subject":[],"published":{"date-parts":[[2022,3,12]]},"assertion":[{"value":"31 October 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 February 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 March 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 July 2022","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Update","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Missing Open Access funding information has been added in the Funding Note.","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The first author (Cinzia Daraio) is a member of the Board of <i>Scientometrics<\/i>.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}