{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,2]],"date-time":"2025-12-02T15:19:53Z","timestamp":1764688793634},"reference-count":50,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2010,11,11]],"date-time":"2010-11-11T00:00:00Z","timestamp":1289433600000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2011,3]]},"DOI":"10.1007\/s11192-010-0312-7","type":"journal-article","created":{"date-parts":[[2010,11,10]],"date-time":"2010-11-10T15:42:39Z","timestamp":1289403759000},"page":"575-592","source":"Crossref","is-referenced-by-count":50,"title":["Overturning some assumptions about the effects of evaluation systems on publication performance"],"prefix":"10.1007","volume":"86","author":[{"given":"Carmen","family":"Osuna","sequence":"first","affiliation":[]},{"given":"Laura","family":"Cruz-Castro","sequence":"additional","affiliation":[]},{"given":"Luis","family":"Sanz-Men\u00e9ndez","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2010,11,11]]},"reference":[{"issue":"1","key":"312_CR1","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1080\/10967490801887889","volume":"11","author":"LB Andersen","year":"2008","unstructured":"Andersen, L. B., & Pallesen, T. (2008). \u201cNot Just for the Money?\u201d How financial incentives affect the number of publications at Danish Research Institutions. International Public Management Journal, 11(1), 28\u201347.","journal-title":"International Public Management Journal"},{"issue":"1\u20132","key":"312_CR2","first-page":"1","volume":"13","author":"T Braun","year":"1988","unstructured":"Braun, T., & Schubert, A. (1988). World flash on basic research: Scientometric versus socio-economic indicators. Scatter plots for 51 countries. 1978\u20131980. Scientometrics, 13(1\u20132), 1\u20139.","journal-title":"Scientometrics"},{"key":"312_CR3","doi-asserted-by":"crossref","DOI":"10.1142\/0106","volume-title":"Scientometrics indicators. A 32-country comparative evaluation of publishing performance and citation impact","author":"T Braun","year":"1985","unstructured":"Braun, T., Gl\u00e4nzel, W., & Schubert, A. (1985). Scientometrics indicators. A 32-country comparative evaluation of publishing performance and citation impact. Singapore, Philadelphia: World Scientific."},{"issue":"1","key":"312_CR4","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1016\/S0048-7333(02)00007-0","volume":"32","author":"L Butler","year":"2003","unstructured":"Butler, L. (2003a). Explaining Australia\u2019s increased share of ISI publications. The effects of a funding formula based on publication counts. Research Policy, 32(1), 143\u2013155.","journal-title":"Research Policy"},{"issue":"1","key":"312_CR5","doi-asserted-by":"crossref","first-page":"39","DOI":"10.3152\/147154403781776780","volume":"12","author":"L Butler","year":"2003","unstructured":"Butler, L. (2003b). Modifying publication practices in response to funding formulas. Research Evaluation, 12(1), 39\u201346.","journal-title":"Research Evaluation"},{"key":"312_CR6","doi-asserted-by":"crossref","first-page":"98","DOI":"10.4337\/9781781957059.00014","volume-title":"Learning from science and technology policy evaluation","author":"DFJ Campbell","year":"2003","unstructured":"Campbell, D. F. J. (2003). The evaluation of university research in the United Kingdom and the Netherlands, Germany and Austria. In Ph. Shapira & S. Kuhlman (Eds.), Learning from science and technology policy evaluation (pp. 98\u2013131). Cheltenham (UK): Edward Elgar."},{"issue":"4","key":"312_CR7","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1016\/S0149-7189(96)00025-0","volume":"19","author":"DT Campbell","year":"1996","unstructured":"Campbell, D. T. (1996). Regression artifacts in time-series and longitudinal data. Evaluation and Program Planning, 19(4), 377\u2013389.","journal-title":"Evaluation and Program Planning"},{"issue":"1","key":"312_CR8","doi-asserted-by":"crossref","first-page":"32","DOI":"10.2307\/3052794","volume":"3","author":"DT Campbell","year":"1968","unstructured":"Campbell, D. T., & Ross, H. L. (1968). The Connecticut crackdown on speeding. Time-series data in quasi-experimental analysis. Law and Society Review, 3(1), 32\u201354.","journal-title":"Law and Society Review"},{"key":"312_CR9","volume-title":"Quasi-experimentation: Design and analysis issues for field settings","author":"TD Cook","year":"1979","unstructured":"Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin Co."},{"issue":"4","key":"312_CR10","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1016\/j.respol.2007.12.007","volume":"37","author":"GA Crespi","year":"2008","unstructured":"Crespi, G. A., & Geuna, A. (2008). An empirical study of scientific production: A cross country analysis (1981\u20132002). Research Policy, 37(4), 565\u2013579.","journal-title":"Research Policy"},{"key":"312_CR11","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1007\/978-1-4020-6746-4_10","volume-title":"The changing governance of the sciences. The advent of the research evaluation systems","author":"L Cruz-Castro","year":"2007","unstructured":"Cruz-Castro, L., & Sanz-Men\u00e9ndez, L. (2007). Research evaluation in transition: individual versus organisational assessment in Spain. In R. Whitley & J. Glaser (Eds.), The changing governance of the sciences. The advent of the research evaluation systems (pp. 205\u2013224). Dordrecht (NL): Springer."},{"key":"312_CR12","doi-asserted-by":"crossref","unstructured":"Cruz-Castro, L., Sanz-Men\u00e9ndez, L. & Mart\u00ednez, C. (2010). Research centers in transition: patterns of convergence and diversity. The Journal of Technology Transfer, doi: 10.1007\/s10961-010-9168-5 (forthcoming).","DOI":"10.1007\/s10961-010-9168-5"},{"key":"312_CR13","unstructured":"Georghiou, L., Howells, J., Rigby, J. Glynn, S., Butler, J., Cameron, H., et al. (2000). Impact of the research assessment exercise and the future of quality assurance in the light of changes in the research landscape. A report produced by PREST, University of Manchester for HEFCE Higher Education Funding Council for England. April 2000. http:\/\/www.mbs.ac.uk\/research\/innovation\/publications-archive\/reports.aspx . Accessed 10 May 2010."},{"issue":"4","key":"312_CR14","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1023\/B:MINE.0000005155.70870.bd","volume":"41","author":"A Geuna","year":"2003","unstructured":"Geuna, A., & Martin, B. (2003). University research evaluation and funding: An international comparison. Minerva, 41(4), 277\u2013304.","journal-title":"Minerva"},{"key":"312_CR15","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1007\/978-1-4020-6746-4_12","volume-title":"The changing governance of the sciences. The advent of the research evaluation systems","author":"J Gl\u00e4ser","year":"2007","unstructured":"Gl\u00e4ser, J. (2007). The social orders of research evaluation systems. In R. Whitley & J. Glaser (Eds.), The changing governance of the sciences. The advent of the research evaluation systems (pp. 245\u2013266). Dordrecht (NL): Springer."},{"issue":"3","key":"312_CR16","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1016\/0048-7333(94)00777-5","volume":"24","author":"I G\u00f3mez","year":"1995","unstructured":"G\u00f3mez, I., Fern\u00e1ndez, M. T., Zulueta, M. A., & Cam\u00ed, J. (1995). Analysis of biomedical research in Spain. Research Policy, 24(3), 459\u2013471.","journal-title":"Research Policy"},{"key":"312_CR17","first-page":"275","volume-title":"Radiograf\u00eda de la investigaci\u00f3n p\u00fablica en Espa\u00f1a","author":"I G\u00f3mez","year":"2006","unstructured":"G\u00f3mez, I., Sancho, R., Bordons, M., & Fern\u00e1ndez, M. T. (2006). La I+D en Espa\u00f1a a trav\u00e9s de publicaciones y patentes. In J. Sebasti\u00e1n & E. Mu\u00f1oz (Eds.), Radiograf\u00eda de la investigaci\u00f3n p\u00fablica en Espa\u00f1a (pp. 275\u2013302). Madrid: Biblioteca Nueva."},{"issue":"4","key":"312_CR18","doi-asserted-by":"crossref","first-page":"393","DOI":"10.1007\/s10734-008-9154-0","volume":"57","author":"D Hicks","year":"2009","unstructured":"Hicks, D. (2009). Evolving regimes of multi-university research evaluation. Higher Education, 57(4), 393\u2013404.","journal-title":"Higher Education"},{"issue":"3","key":"312_CR19","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1023\/B:SCIE.0000018541.29653.f5","volume":"59","author":"P Ingwersen","year":"2004","unstructured":"Ingwersen, P., & Jacobs, D. (2004). South African research in selected scientific areas: Status 1981\u20132000. Scientometrics, 59(3), 405\u2013423.","journal-title":"Scientometrics"},{"issue":"1","key":"312_CR20","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1007\/BF02126647","volume":"36","author":"E Jim\u00e9nez-Contreras","year":"1996","unstructured":"Jim\u00e9nez-Contreras, E., & Ferreiro-Al\u00e1ez, L. (1996). Publishing abroad: Fair trade or short sell for non-english-speaking authors: A Spanish Study. Scientometrics, 36(1), 81\u201395.","journal-title":"Scientometrics"},{"issue":"1","key":"312_CR21","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/S0048-7333(02)00008-2","volume":"32","author":"E Jim\u00e9nez-Contreras","year":"2003","unstructured":"Jim\u00e9nez-Contreras, E., Moya-Aneg\u00f3n, F., & L\u00f3pez-C\u00f3zar, E. (2003). The evolution of research activity in Spain. The impact of National Commission for the Evaluation of Research Activity (CNEAI). Research Policy, 32(1), 123\u2013142.","journal-title":"Research Policy"},{"issue":"7","key":"312_CR22","doi-asserted-by":"crossref","first-page":"489","DOI":"10.3152\/147154306781778777","volume":"33","author":"G Laudel","year":"2006","unstructured":"Laudel, G. (2006). The art of getting funding: How scientists adapt to their funding conditions. Science and Public Policy, 33(7), 489\u2013504.","journal-title":"Science and Public Policy"},{"issue":"4","key":"312_CR23","doi-asserted-by":"crossref","first-page":"469","DOI":"10.1023\/A:1027381906977","volume":"46","author":"I Liefner","year":"2003","unstructured":"Liefner, I. (2003). Funding, resources allocation, and performance in higher education systems. Higher Education, 46(4), 469\u2013489.","journal-title":"Higher Education"},{"issue":"3\u20134","key":"312_CR24","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1007\/BF02026042","volume":"10","author":"A M\u00e9ndez","year":"1986","unstructured":"M\u00e9ndez, A., & G\u00f3mez, I. (1986). The Spanish scientific productivity through eight international databases. Scientometrics, 10(3\u20134), 207\u2013219.","journal-title":"Scientometrics"},{"issue":"1","key":"312_CR25","doi-asserted-by":"crossref","first-page":"153","DOI":"10.1007\/s11192-008-0108-1","volume":"74","author":"H Moed","year":"2008","unstructured":"Moed, H. (2008). UK research assessment exercises: Informed judgements on research quality or quantity? Scientometric, 74(1), 153\u2013161.","journal-title":"Scientometric"},{"issue":"3","key":"312_CR26","doi-asserted-by":"crossref","first-page":"381","DOI":"10.1007\/BF02017338","volume":"33","author":"HF Moed","year":"1995","unstructured":"Moed, H. F., Debruin, R. E., & Van Leeuwen, T. N. (1995). New bibliometric tools for the assessment of national research performance: Data base description, overview of indicators and first applications. Scientometrics, 33(3), 381\u2013422.","journal-title":"Scientometrics"},{"issue":"1","key":"312_CR27","doi-asserted-by":"crossref","first-page":"60","DOI":"10.3152\/147154499781777711","volume":"8","author":"HF Moed","year":"1999","unstructured":"Moed, H. F., Leeuwen, T. N., & Visser, M. S. (1999). Trends in publication output and impact of universities in the Netherlands. Research Evaluation, 8(1), 60\u201367.","journal-title":"Research Evaluation"},{"issue":"4","key":"312_CR28","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1016\/S0149-7189(00)00029-X","volume":"23","author":"LB Mohr","year":"2000","unstructured":"Mohr, L. B. (2000). Regression artifacts and other customs of dubious desert. Evaluation and Program Planning, 23(4), 397\u2013409.","journal-title":"Evaluation and Program Planning"},{"issue":"2","key":"312_CR29","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1007\/BF02464780","volume":"46","author":"F Moya-Aneg\u00f3n","year":"1999","unstructured":"Moya-Aneg\u00f3n, F., & Herrero-Solana, V. (1999). Science in America Latina: A comparison of bibliometric and scientific-technical indicators. Scientometrics, 46(2), 299\u2013320.","journal-title":"Scientometrics"},{"key":"312_CR30","volume-title":"Indicadores bibliom\u00e9tricos de la actividad cient\u00edfica Espa\u00f1ola (1990\u20132004)","author":"F Moya-Aneg\u00f3n","year":"2007","unstructured":"Moya-Aneg\u00f3n, F., et al. (2007). Indicadores bibliom\u00e9tricos de la actividad cient\u00edfica Espa\u00f1ola (1990\u20132004). Madrid: FECYT."},{"issue":"3","key":"312_CR31","doi-asserted-by":"crossref","first-page":"543","DOI":"10.1007\/s11192-007-1878-6","volume":"76","author":"C \u00d6nder","year":"2008","unstructured":"\u00d6nder, C., Sevkli, M., Altinok, T., & Tavuk\u00e7uoglu, C. (2008). Institutional change and scientific research: A preliminary bibliometric analysis of institutional influences on Turkey\u2019s recent social science publications. Scientometrics, 76(3), 543\u2013560.","journal-title":"Scientometrics"},{"issue":"3","key":"312_CR32","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1023\/B:SCIE.0000034384.35498.7d","volume":"60","author":"O Persson","year":"2004","unstructured":"Persson, O., Gl\u00e4nzel, W., & Danell, R. (2004). Inflationary bibliometric values: The role of scientific collaboration and the need for relative indicators in evaluative studies. Scientometrics, 60(3), 421\u2013432.","journal-title":"Scientometrics"},{"issue":"9","key":"312_CR33","doi-asserted-by":"crossref","first-page":"1845","DOI":"10.1002\/asi.21104","volume":"60","author":"A Rodriguez-Navarro","year":"2009","unstructured":"Rodriguez-Navarro, A. (2009). Sound research, unimportant discoveries: Research, universities, and formal evaluation of research in Spain. Journal of the American Society for Information Science and Technology, 60(9), 1845\u20131858.","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"4\/5\/6","key":"312_CR34","first-page":"255","volume":"10","author":"L Sanz-Men\u00e9ndez","year":"1995","unstructured":"Sanz-Men\u00e9ndez, L. (1995a). Policy choices, institutional constraints and policy learning: The Spanish science and technology policy in the eighties. International Journal of Technology Management, 10(4\/5\/6), 255\u2013274.","journal-title":"International Journal of Technology Management"},{"issue":"1","key":"312_CR35","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1093\/rev\/5.1.79","volume":"5","author":"L Sanz-Men\u00e9ndez","year":"1995","unstructured":"Sanz-Men\u00e9ndez, L. (1995b). Research actors and the state: Research evaluation and evaluation of science and technology policies in Spain. Research Evaluation, 5(1), 79\u201388.","journal-title":"Research Evaluation"},{"key":"312_CR36","volume-title":"Estado, ciencia y tecnolog\u00eda en Espa\u00f1a (1939\u20131997)","author":"L Sanz-Men\u00e9ndez","year":"1997","unstructured":"Sanz-Men\u00e9ndez, L. (1997). Estado, ciencia y tecnolog\u00eda en Espa\u00f1a (1939\u20131997). Madrid: Alianza Editorial."},{"issue":"Mayo","key":"312_CR37","first-page":"9","volume":"557","author":"L Sanz Men\u00e9ndez","year":"1992","unstructured":"Sanz Men\u00e9ndez, L., & Pfretzchner, J. (1992). Pol\u00edtica cient\u00edfica y gesti\u00f3n de la investigaci\u00f3n: El CSIC (1986\u20131990) en el sistema espa\u00f1ol de ciencia y tecnolog\u00eda. Arbor, 557(Mayo), 9\u201351.","journal-title":"Arbor"},{"issue":"3","key":"312_CR38","doi-asserted-by":"crossref","first-page":"227","DOI":"10.3145\/epi.2006.may.09","volume":"15","author":"G Scimago","year":"2006","unstructured":"Scimago, G. (2006). Producci\u00f3n ISI y tramos de investigaci\u00f3n: C\u00f3mo combinarlos en un nuevo indicador. El profesional de la informaci\u00f3n, 15(3), 227\u2013228.","journal-title":"El profesional de la informaci\u00f3n"},{"issue":"4","key":"312_CR39","doi-asserted-by":"crossref","first-page":"510","DOI":"10.3145\/epi.2007.sep.14","volume":"16","author":"G Scimago","year":"2007","unstructured":"Scimago, G. (2007). Producci\u00f3n ISI y tramos de investigaci\u00f3n: C\u00f3mo combinarlos en un nuevo indicador (II). El profesional de la informaci\u00f3n, 16(4), 510\u2013511.","journal-title":"El profesional de la informaci\u00f3n"},{"key":"312_CR40","volume-title":"Experimental and quasi-experimental designs for generalized causal inference","author":"WR Shadish","year":"2002","unstructured":"Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, New York: Houghton Mifflin Co."},{"issue":"2","key":"312_CR41","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1007\/s11192-008-0212-2","volume":"74","author":"RD Shelton","year":"2008","unstructured":"Shelton, R. D. (2008). Relations between national research investment and publication output: Application to an American paradox. Scientometrics, 74(2), 191\u2013205.","journal-title":"Scientometrics"},{"key":"312_CR50","unstructured":"SRI International (1988). Research activity in Spain, Portugal and Greece. A 1988 Bibliometric model assessment. (Report prepared by Ailes, Coward and Fresne), Science and Technology Policy Programme, SRI International, Arlington (Vi), mimeo."},{"issue":"3","key":"312_CR42","first-page":"30","volume":"33","author":"AA Talib","year":"2001","unstructured":"Talib, A. A. (2001). The continuing behavioural modification of academics since the 1992 research assessment exercise. Higher Education Review, 33(3), 30\u201346.","journal-title":"Higher Education Review"},{"issue":"1","key":"312_CR43","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1111\/1468-2273.00173","volume":"55","author":"J Taylor","year":"2001","unstructured":"Taylor, J. (2001). The impact of performance indicators on the work of university academics: Evidence from Australian universities. Higher Education Quarterly, 55(1), 42\u201361.","journal-title":"Higher Education Quarterly"},{"issue":"5","key":"312_CR44","doi-asserted-by":"crossref","first-page":"290","DOI":"10.1093\/spp\/24.5.290","volume":"24","author":"AFJ Raan Van","year":"1997","unstructured":"Van Raan, A. F. J. (1997). Science as an international enterprise. Science and Public Policy, 24(5), 290\u2013300.","journal-title":"Science and Public Policy"},{"issue":"4","key":"312_CR45","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1023\/A:1002995623272","volume":"33","author":"D Westerheijden","year":"1997","unstructured":"Westerheijden, D. (1997). A solid base for decisions: Use of the VSNU research evaluation in Dutch Universities. Higher Education, 33(4), 397\u2013413.","journal-title":"Higher Education"},{"issue":"6","key":"312_CR46","doi-asserted-by":"crossref","first-page":"1015","DOI":"10.1016\/S0048-7333(02)00112-9","volume":"32","author":"R Whitley","year":"2003","unstructured":"Whitley, R. (2003). Competition and pluralism in public sciences: The impact of institutional frameworks on the organization of academic science. Research Policy, 32(6), 1015\u20131039.","journal-title":"Research Policy"},{"key":"312_CR47","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/978-1-4020-6746-4_1","volume-title":"The changing governance of the sciences. The advent of the research evaluation systems","author":"R Whitley","year":"2007","unstructured":"Whitley, R. (2007). The changing governance of the public sciences: The consequences of research evaluation systems for knowledge production in different countries and scientific fields. In R. Whitley & J. Glaser (Eds.), The changing governance of the sciences. The advent of the research evaluation systems (pp. 3\u201327). Dordrecht (NL): Springer."},{"key":"312_CR48","volume-title":"The changing governance of the sciences. The advent of the research evaluation systems","year":"2007","unstructured":"Whitley, R., & Glaser, J. (Eds.). (2007). The changing governance of the sciences. The advent of the research evaluation systems. Dordrecht (NL): Springer."},{"issue":"1","key":"312_CR49","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1002\/(SICI)1097-4571(1998)49:1<30::AID-ASI5>3.0.CO;2-3","volume":"49","author":"M Zitt","year":"1998","unstructured":"Zitt, M., Perrot, F., & Barr\u00e9, R. (1998). The transition from \u201cnational\u201d to \u201ctransnational\u201d model and related measures of countries\u2019 performance. Journal of the American Society of Information Science, 49(1), 30\u201342.","journal-title":"Journal of the American Society of Information Science"}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-010-0312-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s11192-010-0312-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-010-0312-7","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,4]],"date-time":"2023-06-04T05:01:55Z","timestamp":1685854915000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s11192-010-0312-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,11,11]]},"references-count":50,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2011,3]]}},"alternative-id":["312"],"URL":"https:\/\/doi.org\/10.1007\/s11192-010-0312-7","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[2010,11,11]]}}}