{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,23]],"date-time":"2026-07-23T18:49:31Z","timestamp":1784832571630,"version":"3.55.0"},"reference-count":57,"publisher":"MIT Press","issue":"4","license":[{"start":{"date-parts":[[2022,11,15]],"date-time":"2022-11-15T00:00:00Z","timestamp":1668470400000},"content-version":"vor","delay-in-days":318,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010685","name":"H2020 Science with and for Society","doi-asserted-by":"publisher","award":["824612"],"award-info":[{"award-number":["824612"]}],"id":[{"id":"10.13039\/100010685","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,12,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The need to reform research assessment processes related to career advancement at research institutions has become increasingly recognized in recent years, especially to better foster open and responsible research practices. Current assessment criteria are believed to focus too heavily on inappropriate criteria related to productivity and quantity as opposed to quality, collaborative open research practices, and the socioeconomic impact of research. Evidence of the extent of these issues is urgently needed to inform actions for reform, however. We analyze current practices as revealed by documentation on institutional review, promotion, and tenure (RPT) processes in seven countries (Austria, Brazil, Germany, India, Portugal, the United Kingdom and the United States). Through systematic coding and analysis of 143 RPT policy documents from 107 institutions for the prevalence of 17 criteria (including those related to qualitative or quantitative assessment of research, service to the institution or profession, and open and responsible research practices), we compare assessment practices across a range of international institutions to significantly broaden this evidence base. Although the prevalence of indicators varies considerably between countries, overall we find that currently open and responsible research practices are minimally rewarded and problematic practices of quantification continue to dominate.<\/jats:p>","DOI":"10.1162\/qss_a_00224","type":"journal-article","created":{"date-parts":[[2022,11,15]],"date-time":"2022-11-15T15:10:20Z","timestamp":1668525020000},"page":"888-911","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":52,"title":["Indicators of research quality, quantity, openness, and responsibility in institutional review, promotion, and tenure policies across seven countries"],"prefix":"10.1162","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2091-0402","authenticated-orcid":true,"given":"Nancy","family":"Pontika","sequence":"first","affiliation":[{"name":"Knowledge Media Institute, Open University, Milton Keynes, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7331-4751","authenticated-orcid":true,"given":"Thomas","family":"Klebel","sequence":"additional","affiliation":[{"name":"Open and Reproducible Research Group, Know-Center GmbH, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6610-8853","authenticated-orcid":true,"given":"Antonia","family":"Correia","sequence":"additional","affiliation":[{"name":"University of Minho, Braga, Portugal"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9254-3675","authenticated-orcid":true,"given":"Hannah","family":"Metzler","sequence":"additional","affiliation":[{"name":"Complexity Science Hub Vienna, Vienna, Austria"},{"name":"Medical University of Vienna, Vienna, Austria"},{"name":"Institute for Globally Distributed Open Research and Education, Gothenburg, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1161-7359","authenticated-orcid":true,"given":"Petr","family":"Knoth","sequence":"additional","affiliation":[{"name":"Knowledge Media Institute, Open University, Milton Keynes, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4470-7027","authenticated-orcid":true,"given":"Tony","family":"Ross-Hellauer","sequence":"additional","affiliation":[{"name":"Open and Reproducible Research Group, Know-Center GmbH, Graz, Austria"},{"name":"Open and Reproducible Research Group, Graz University of Technology, Graz, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2022,12,20]]},"reference":[{"issue":"1","key":"2023020920354755600_bib1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1214\/09-STS285","article-title":"Citation statistics","volume":"24","author":"Adler","year":"2009","journal-title":"Statistical Science"},{"key":"2023020920354755600_bib2","doi-asserted-by":"publisher","DOI":"10.7910\/DVN\/VY4TJE","volume-title":"Terms and concepts found in tenure and promotion guidelines from the US and Canada","author":"Alperin","year":"2019"},{"key":"2023020920354755600_bib3","doi-asserted-by":"publisher","DOI":"10.17613\/ye06-n045","article-title":"The value of data and other non-traditional scholarly outputs in academic review, promotion, and tenure in Canada and the United States","volume-title":"Open handbook of linguistic data management","author":"Alperin","year":"2020"},{"key":"2023020920354755600_bib4","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1201\/9781420011319.ch1","article-title":"Correspondence analysis and related methods in practice","volume-title":"Multiple correspondence analysis and related methods","author":"Blasius","year":"2006"},{"key":"2023020920354755600_bib5","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1007\/978-3-319-10720-2_2","article-title":"The rocky road to tenure\u2014Career paths in academia","volume-title":"Academic work and careers in Europe: Trends, challenges, perspectives","author":"Brechelmacher","year":"2015"},{"issue":"1","key":"2023020920354755600_bib6","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1108\/RSR-05-2013-0023","article-title":"Citation searching for tenure and promotion: An overview of issues and tools","volume":"42","author":"Brown","year":"2014","journal-title":"Reference Services Review"},{"issue":"4","key":"2023020920354755600_bib7","doi-asserted-by":"publisher","first-page":"e0230416","DOI":"10.1371\/journal.pone.0230416","article-title":"The citation advantage of linking publications to research data","volume":"15","author":"Colavizza","year":"2020","journal-title":"PLOS ONE"},{"key":"2023020920354755600_bib8","article-title":"Publishing in open access journals in the social sciences and humanities: Who\u2019s doing it and why","volume-title":"Pushing the edge: Proceedings of the Fourteenth National Conference of the Association of College and Research Libraries","author":"Coonin","year":"2009"},{"issue":"3","key":"2023020920354755600_bib9","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1016\/j.joi.2007.02.001","article-title":"The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level","volume":"1","author":"Costas","year":"2007","journal-title":"Journal of Informetrics"},{"issue":"7691","key":"2023020920354755600_bib10","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1038\/d41586-018-01642-w","article-title":"Let\u2019s move beyond the rhetoric: It\u2019s time to change how we judge research","volume":"554","author":"Curry","year":"2018","journal-title":"Nature"},{"key":"2023020920354755600_bib11","doi-asserted-by":"publisher","DOI":"10.2777\/75255","volume-title":"Evaluation of research careers fully acknowledging open science practices: Rewards, incentives and\/or recognition for researchers practicing open science","author":"European Commission, Directorate General for Research and Innovation","year":"2017"},{"key":"2023020920354755600_bib12","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1017\/S0003975613000167","article-title":"Impact factor fetishism","volume":"2","author":"Fleck","year":"2013","journal-title":"European Journal of Sociology"},{"key":"2023020920354755600_bib13","doi-asserted-by":"publisher","first-page":"680023","DOI":"10.3389\/frma.2021.680023","article-title":"Mis-measuring our universities: Why global university rankings don\u2019t add up","volume":"6","author":"Gadd","year":"2021","journal-title":"Frontiers in Research Metrics and Analytics"},{"issue":"1","key":"2023020920354755600_bib14","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1353\/rhe.2014.0045","article-title":"Evincing the ratchet: A thematic analysis of the promotion and tenure guidelines at a striving university","volume":"38","author":"Gardner","year":"2014","journal-title":"Review of Higher Education"},{"key":"2023020920354755600_bib15","volume-title":"Ca: Simple, multiple and joint correspondence analysis","author":"Greenacre","year":"2018"},{"issue":"5","key":"2023020920354755600_bib16","doi-asserted-by":"publisher","first-page":"607","DOI":"10.1108\/AJIM-01-2017-0006","article-title":"Recognition and reward in the academy: Valuing publication oeuvres in biomedicine, economics and history","volume":"69","author":"Hammarfelt","year":"2017","journal-title":"Aslib Journal of Information Management"},{"issue":"3","key":"2023020920354755600_bib17","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1093\/reseval\/rvx018","article-title":"Indicators as judgment devices: An empirical study of citizen bibliometrics in research evaluation","volume":"26","author":"Hammarfelt","year":"2017","journal-title":"Research Evaluation"},{"issue":"7548","key":"2023020920354755600_bib18","doi-asserted-by":"publisher","first-page":"429","DOI":"10.1038\/520429a","article-title":"Bibliometrics: The Leiden Manifesto for research metrics","volume":"520","author":"Hicks","year":"2015","journal-title":"Nature"},{"issue":"11","key":"2023020920354755600_bib19","doi-asserted-by":"publisher","first-page":"e2000995","DOI":"10.1371\/journal.pbio.2000995","article-title":"Current incentives for scientists lead to underpowered studies with erroneous conclusions","volume":"14","author":"Higginson","year":"2016","journal-title":"PLOS Biology"},{"issue":"8","key":"2023020920354755600_bib20","doi-asserted-by":"publisher","first-page":"e124","DOI":"10.1371\/journal.pmed.0020124","article-title":"Why most published research findings are false","volume":"2","author":"Ioannidis","year":"2005","journal-title":"PLOS Medicine"},{"key":"2023020920354755600_bib21","volume-title":"Assessing the future landscape of scholarly communication: An exploration of faculty values and needs in seven disciplines","author":"King","year":"2010"},{"issue":"21","key":"2023020920354755600_bib22","doi-asserted-by":"publisher","first-page":"550","DOI":"10.21105\/joss.00550","article-title":"The drake R package: A pipeline toolkit for reproducibility and high-performance computing","volume":"3","author":"Landau","year":"2018","journal-title":"Journal of Open Source Software"},{"issue":"6","key":"2023020920354755600_bib23","doi-asserted-by":"publisher","first-page":"820","DOI":"10.1177\/030631201031006002","article-title":"The decision-making constraints and processes of grant peer review, and their effects on the review outcome","volume":"31","author":"Langfeldt","year":"2001","journal-title":"Social Studies of Science"},{"key":"2023020920354755600_bib24","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1016\/j.cpa.2014.02.006","article-title":"Journal ranking effects on junior academics: Identity fragmentation and politicization","volume":"26","author":"Malsch","year":"2015","journal-title":"Critical Perspectives on Accounting"},{"key":"2023020920354755600_bib25","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1145\/2047594.2047632","article-title":"Computing faculty tenure and promotion requirements at USA and Canadian post-secondary institutions","volume-title":"Proceedings of the 2011 Conference on Information Technology Education","author":"McGill","year":"2011"},{"key":"2023020920354755600_bib26","doi-asserted-by":"publisher","first-page":"e16800","DOI":"10.7554\/eLife.16800","article-title":"How open science helps researchers succeed","volume":"5","author":"McKiernan","year":"2016","journal-title":"eLife"},{"key":"2023020920354755600_bib27","doi-asserted-by":"publisher","first-page":"e47338","DOI":"10.7554\/eLife.47338","article-title":"Use of the journal impact factor in academic review, promotion, and tenure evaluations","volume":"8","author":"McKiernan","year":"2019","journal-title":"eLife"},{"issue":"4","key":"2023020920354755600_bib28","doi-asserted-by":"publisher","first-page":"1250","DOI":"10.1111\/ecin.12131","article-title":"Open access journals and academics\u2019 behavior","volume":"52","author":"Migheli","year":"2014","journal-title":"Economic Inquiry"},{"key":"2023020920354755600_bib29","volume-title":"Open access in European universities: Results from the 2016\/2017 EUA Institutional Survey","author":"Morais","year":"2018"},{"key":"2023020920354755600_bib30","volume-title":"The evolution of responsible research and innovation in Europe: The MoRRI Indicators Report D4.3","author":"MoRRI","year":"2018"},{"issue":"3","key":"2023020920354755600_bib31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v020.i03","article-title":"Correspondence analysis in R, with two- and three-dimensional graphics: The ca package","volume":"20","author":"Nenadic","year":"2007","journal-title":"Journal of Statistical Software"},{"issue":"3","key":"2023020920354755600_bib32","doi-asserted-by":"publisher","first-page":"e0228914","DOI":"10.1371\/journal.pone.0228914","article-title":"Why we publish where we do: Faculty publishing values and their relationship to review, promotion and tenure expectations","volume":"15","author":"Niles","year":"2020","journal-title":"PLOS ONE"},{"issue":"6","key":"2023020920354755600_bib33","doi-asserted-by":"publisher","first-page":"751","DOI":"10.1093\/scipol\/scs093","article-title":"Responsible research and innovation: From science in society to science for society, with society","volume":"39","author":"Owen","year":"2012","journal-title":"Science and Public Policy"},{"key":"2023020920354755600_bib100","article-title":"Ouvrir La Science (blog)","author":"Paris Call on Research Assessment","year":"2022"},{"issue":"5","key":"2023020920354755600_bib34","doi-asserted-by":"publisher","first-page":"640","DOI":"10.1177\/0165551515587855","article-title":"How library and information science faculty perceive and engage with open access","volume":"41","author":"Peekhaus","year":"2015","journal-title":"Journal of Information Science"},{"issue":"1","key":"2023020920354755600_bib35","doi-asserted-by":"publisher","first-page":"51","DOI":"10.54886\/ibersid.v10i1.4252","article-title":"Produ\u00e7\u00e3o da ci\u00eancia da informa\u00e7\u00e3o na Web of Science entre 1994 e 2013 e a lista Qualis\/Capes da \u00c1rea","volume":"10","author":"Pinto","year":"2016","journal-title":"Ibersid: Revista de Sistemas de Informaci\u00f3n y Documentaci\u00f3n"},{"key":"2023020920354755600_bib36","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2809563.2809571","article-title":"Fostering open science to research using a taxonomy and an eLearning portal","volume-title":"Proceedings of the 15th International Conference on Knowledge Technologies and Data-Driven Business","author":"Pontika","year":"2015"},{"key":"2023020920354755600_bib37","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.5552197","article-title":"ON-MERRIT D6.1 Investigating institutional structures of reward & recognition in open science & RRI (1.0)","author":"Pontika","year":"2021","journal-title":"Zenodo"},{"key":"2023020920354755600_bib38","doi-asserted-by":"publisher","first-page":"99","DOI":"10.12688\/openreseurope.14921.1","article-title":"How do career promotion policies affect research publications and open access? [version 1; peer review: awaiting peer review]","volume":"2","author":"Pontika","year":"2022","journal-title":"Open Research Europe"},{"key":"2023020920354755600_bib39","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.6323765","article-title":"Data for \u201cIndicators of research quality, quantity, openness and responsibility in institutional promotion, review and tenure policies across seven countries\u201d","author":"Pontika","year":"2022","journal-title":"Zenodo"},{"key":"2023020920354755600_bib40","volume-title":"R: A language and environment for statistical computing","author":"R Core Team","year":"2021"},{"key":"2023020920354755600_bib41","volume-title":"Process towards an agreement on reforming research assessment","author":"Research and Innovation","year":"2022"},{"key":"2023020920354755600_bib42","doi-asserted-by":"publisher","first-page":"m2081","DOI":"10.1136\/bmj.m2081","article-title":"Academic criteria for promotion and tenure in biomedical sciences faculties: Cross sectional analysis of international sample of universities","volume":"369","author":"Rice","year":"2020","journal-title":"British Medical Journal"},{"issue":"6","key":"2023020920354755600_bib43","doi-asserted-by":"publisher","first-page":"604","DOI":"10.1016\/j.acalib.2014.07.013","article-title":"Awareness and attitudes about open access publishing: A glance at generational differences","volume":"40","author":"Rodriguez","year":"2014","journal-title":"The Journal of Academic Librarianship"},{"issue":"1","key":"2023020920354755600_bib44","doi-asserted-by":"publisher","first-page":"211032","DOI":"10.1098\/rsos.211032","article-title":"Dynamics of cumulative advantage and threats to equity in open science: A scoping review","volume":"9","author":"Ross-Hellauer","year":"2022","journal-title":"Royal Society Open Science"},{"key":"2023020920354755600_bib45","first-page":"48","volume-title":"Research assessment in the transition to open science","author":"Saenen","year":"2019"},{"issue":"1","key":"2023020920354755600_bib46","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1080\/10437797.2003.10779120","article-title":"Assessing publication for tenure","volume":"39","author":"Seipel","year":"2003","journal-title":"Journal of Social Work Education"},{"issue":"Suppl. 2","key":"2023020920354755600_bib47","doi-asserted-by":"publisher","first-page":"S17","DOI":"10.1016\/j.radi.2020.02.004","article-title":"The increasing significance of impact within the Research Excellence Framework (REF)","volume":"26","author":"Sutton","year":"2020","journal-title":"Radiography"},{"key":"2023020920354755600_bib49","doi-asserted-by":"publisher","DOI":"10.4337\/9781784718862.00006","article-title":"Why responsible innovation","volume-title":"The international handbook on responsible innovation: A global resource","author":"von Schomberg","year":"2019"},{"issue":"1","key":"2023020920354755600_bib50","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1186\/1472-6920-10-21","article-title":"Authors\u2019 opinions on publication in relation to annual performance assessment","volume":"10","author":"Walker","year":"2010","journal-title":"BMC Medical Education"},{"issue":"12","key":"2023020920354755600_bib51","doi-asserted-by":"publisher","first-page":"2419","DOI":"10.1002\/asi.22708","article-title":"The Leiden ranking 2011\/2012: Data collection, indicators, and interpretation","volume":"63","author":"Waltman","year":"2012","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"2","key":"2023020920354755600_bib52","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1016\/j.joi.2014.01.006","article-title":"Unpacking the Matthew Effect in citations","volume":"8","author":"Wang","year":"2014","journal-title":"Journal of Informetrics"},{"key":"2023020920354755600_bib53","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24277-4","volume-title":"ggplot2: Elegant graphics for data analysis","author":"Wickham","year":"2016","edition":"2nd ed."},{"issue":"43","key":"2023020920354755600_bib54","doi-asserted-by":"publisher","first-page":"1686","DOI":"10.21105\/joss.01686","article-title":"Welcome to the Tidyverse","volume":"4","author":"Wickham","year":"2019","journal-title":"Journal of Open Source Software"},{"key":"2023020920354755600_bib55","doi-asserted-by":"publisher","DOI":"10.13140\/RG.2.1.4929.1363","volume-title":"The metric tide: Report of the independent review of the role of metrics in research assessment and management","author":"Wilsdon","year":"2015"},{"key":"2023020920354755600_bib56","doi-asserted-by":"publisher","first-page":"462","DOI":"10.1038\/d41586-021-01759-5","article-title":"Impact factor abandoned by Dutch university in hiring and promotion decisions","volume":"595","author":"Woolston","year":"2021","journal-title":"Nature"},{"issue":"2","key":"2023020920354755600_bib57","doi-asserted-by":"publisher","first-page":"204","DOI":"10.1080\/00221546.2009.11772139","article-title":"Learning from the experience of others: The evolution of faculty tenure and promotion rules in comprehensive institutions","volume":"80","author":"Youn","year":"2009","journal-title":"Journal of Higher Education"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/3\/4\/888\/2070770\/qss_a_00224.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/3\/4\/888\/2070770\/qss_a_00224.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,9]],"date-time":"2023-02-09T15:39:23Z","timestamp":1675957163000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/3\/4\/888\/113769\/Indicators-of-research-quality-quantity-openness"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":57,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,12,20]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00224","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00224\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00224\/v3\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00224\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00224\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00224\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00224\/v2\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00224\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00224\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00224\/v3\/decision1","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}