{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,17]],"date-time":"2026-01-17T22:41:37Z","timestamp":1768689697705,"version":"3.49.0"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2013,4,13]],"date-time":"2013-04-13T00:00:00Z","timestamp":1365811200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2014,1]]},"DOI":"10.1007\/s11192-013-1020-x","type":"journal-article","created":{"date-parts":[[2013,4,12]],"date-time":"2013-04-12T04:51:37Z","timestamp":1365742297000},"page":"211-219","source":"Crossref","is-referenced-by-count":53,"title":["How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons"],"prefix":"10.1007","volume":"98","author":[{"given":"Lutz","family":"Bornmann","sequence":"first","affiliation":[]},{"given":"Werner","family":"Marx","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2013,4,13]]},"reference":[{"issue":"3","key":"1020_CR1","doi-asserted-by":"crossref","first-page":"227","DOI":"10.3152\/095820211X13118583635792","volume":"20","author":"S Bell","year":"2011","unstructured":"Bell, S., Shaw, B., & Boaz, A. (2011). Real-world approaches to assessing the impact of environmental research on policy. Research Evaluation, 20(3), 227\u2013237.","journal-title":"Research Evaluation"},{"key":"1020_CR2","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1002\/aris.2011.1440450112","volume":"45","author":"L Bornmann","year":"2011","unstructured":"Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199\u2013245.","journal-title":"Annual Review of Information Science and Technology"},{"issue":"8","key":"1020_CR3","doi-asserted-by":"crossref","first-page":"673","DOI":"10.1038\/embor.2012.99","volume":"13","author":"L Bornmann","year":"2012","unstructured":"Bornmann, L. (2012). Measuring the societal impact of research. EMBO Reports, 13(8), 673\u2013676.","journal-title":"EMBO Reports"},{"issue":"2","key":"1020_CR4","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1002\/asi.22803","volume":"64","author":"L Bornmann","year":"2013","unstructured":"Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society of Information Science and Technology, 64(2), 217\u2013233.","journal-title":"Journal of the American Society of Information Science and Technology"},{"issue":"1","key":"1020_CR5","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1108\/00220410810844150","volume":"64","author":"L Bornmann","year":"2008","unstructured":"Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45\u201380. doi: 10.1108\/00220410810844150 .","journal-title":"Journal of Documentation"},{"issue":"3","key":"1020_CR6","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1016\/j.joi.2007.03.001","volume":"1","author":"L Bornmann","year":"2007","unstructured":"Bornmann, L., Mutz, R., & Daniel, H.-D. (2007). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1(3), 226\u2013238. doi: 10.1016\/j.joi.2007.03.001 .","journal-title":"Journal of Informetrics"},{"issue":"3","key":"1020_CR7","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1016\/j.joi.2011.01.006","volume":"5","author":"L Bornmann","year":"2011","unstructured":"Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H. D. (2011). A meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5(3), 346\u2013359. doi: 10.1016\/j.joi.2011.01.006 .","journal-title":"Journal of Informetrics"},{"issue":"3","key":"1020_CR8","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1007\/s11162-011-9232-5","volume":"53","author":"NA Bowman","year":"2012","unstructured":"Bowman, N. A. (2012). Effect sizes and statistical methods for meta-analysis in higher education. Research in Higher Education, 53(3), 375\u2013382. doi: 10.1007\/s11162-011-9232-5 .","journal-title":"Research in Higher Education"},{"issue":"11","key":"1020_CR9","doi-asserted-by":"crossref","first-page":"964","DOI":"10.1038\/embor.2012.147","volume":"13","author":"P Brown","year":"2012","unstructured":"Brown, P. (2012). Nothing but the truth. EMBO Reports, 13(11), 964\u2013967.","journal-title":"EMBO Reports"},{"key":"1020_CR10","volume-title":"Assessing Europe\u2019s university-based research","author":"European Commission","year":"2010","unstructured":"Commission, European. (2010). Assessing Europe\u2019s university-based research. Publications Office of the European Union: Expert group on assessment of university-based research. Brussels."},{"key":"1020_CR11","volume-title":"Informing research choices: Indicators and judgment: The expert panel on science performance and research funding","author":"Council of Canadian Academies","year":"2012","unstructured":"Council of Canadian Academies. (2012). Informing research choices: Indicators and judgment: The expert panel on science performance and research funding. Ottawa: Council of Canadian Academies."},{"issue":"3","key":"1020_CR13","doi-asserted-by":"crossref","first-page":"175","DOI":"10.3152\/095820211X13118583635918","volume":"20","author":"C Donovan","year":"2011","unstructured":"Donovan, C. (2011). State of the art in assessing research impact: Introduction to a special issue. Research Evaluation, 20(3), 175\u2013179.","journal-title":"Research Evaluation"},{"issue":"2","key":"1020_CR14","doi-asserted-by":"crossref","first-page":"131","DOI":"10.3152\/095820211X12941371876544","volume":"20","author":"E Erno-Kjolhede","year":"2011","unstructured":"Erno-Kjolhede, E., & Hansson, F. (2011). Measuring research performance during a changing relationship between science and society. Research Evaluation, 20(2), 131\u2013143. doi: 10.3152\/095820211x12941371876544 .","journal-title":"Research Evaluation"},{"key":"1020_CR15","doi-asserted-by":"crossref","first-page":"3","DOI":"10.3102\/0013189X005010003","volume":"5","author":"GV Glass","year":"1976","unstructured":"Glass, G. V. (1976). Primary, secondary, and meta-analysis. Educational Researcher, 5, 3\u20138.","journal-title":"Educational Researcher"},{"key":"1020_CR16","unstructured":"Godin, B., & Dore, C. (2005). Measuring the impacts of science; beyond the economic dimension, INRS Urbanisation, Culture et Soci\u00e9t\u00e9. Paper presented at the HIST Lecture, Helsinki Institute for Science and Technology Studies, Helsinki, Finland. http:\/\/www.csiic.ca\/PDF\/Godin_Dore_Impacts.pdf . Accessed 6 April 2013."},{"issue":"2","key":"1020_CR17","doi-asserted-by":"crossref","first-page":"151","DOI":"10.3152\/030234209X406818","volume":"36","author":"B Gregersen","year":"2009","unstructured":"Gregersen, B., Linde, L. T., & Rasmussen, J. G. (2009). Linking between Danish universities and society. Science and Public Policy, 36(2), 151\u2013156. doi: 10.3152\/030234209x406818 .","journal-title":"Science and Public Policy"},{"key":"1020_CR18","volume-title":"Research Excellence Framework. Second consultation on the assessment and funding of research","author":"Higher Education Funding Council for England","year":"2009","unstructured":"Higher Education Funding Council for England. (2009). Research Excellence Framework. Second consultation on the assessment and funding of research. Higher Education Funding Council for England (HEFCE): Bristol."},{"key":"1020_CR19","volume-title":"Decisions on assessing research impact","author":"Higher Education Funding Council for England","year":"2011","unstructured":"Higher Education Funding Council for England. (2011). Decisions on assessing research impact. Higher Education Funding Council for England (HEFCE): Bristol."},{"issue":"3","key":"1020_CR20","doi-asserted-by":"crossref","first-page":"213","DOI":"10.3727\/194982410X12895770314078","volume":"12","author":"JB Holbrook","year":"2010","unstructured":"Holbrook, J. B. (2010). The use of societal impacts considerations in grant proposal peer review: A comparison of five models. Technology & Innovation, 12(3), 213\u2013224. doi: 10.3727\/194982410x12895770314078 .","journal-title":"Technology & Innovation"},{"key":"1020_CR21","volume-title":"Comparative Assessment of Peer Review (CAPR). EU\/US workshop on peer review: Assessing \u201cbroader impact\u201d in research grant applications","author":"JB Holbrook","year":"2010","unstructured":"Holbrook, J. B., & Frodeman, R. (2010). Comparative Assessment of Peer Review (CAPR). EU\/US workshop on peer review: Assessing \u201cbroader impact\u201d in research grant applications. Brussels: European Commission, Directorate-General for Research and Innovation."},{"issue":"3","key":"1020_CR22","doi-asserted-by":"crossref","first-page":"239","DOI":"10.3152\/095820211X12941371876788","volume":"20","author":"JB Holbrook","year":"2011","unstructured":"Holbrook, J. B., & Frodeman, R. (2011). Peer review and the ex ante assessment of societal impacts. Research Evaluation, 20(3), 239\u2013246.","journal-title":"Research Evaluation"},{"key":"1020_CR23","volume-title":"Research with an impact evaluation practises in public research organisations","author":"K L\u00e4hteenm\u00e4ki-Smith","year":"2006","unstructured":"L\u00e4hteenm\u00e4ki-Smith, K., Hyytinen, K., Kutinlahti, P., & Konttinen, J. (2006). Research with an impact evaluation practises in public research organisations. Kemistintie: VTT Technical Research Centre of Finland."},{"issue":"3","key":"1020_CR24","doi-asserted-by":"crossref","first-page":"1290","DOI":"10.3102\/0034654309334143","volume":"79","author":"HW Marsh","year":"2009","unstructured":"Marsh, H. W., Bornmann, L., Mutz, R., Daniel, H. D., & O\u2019Mara, A. (2009). Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Review of Educational Research, 79(3), 1290\u20131326. doi: 10.3102\/0034654309334143 .","journal-title":"Review of Educational Research"},{"issue":"3","key":"1020_CR25","doi-asserted-by":"crossref","first-page":"247","DOI":"10.3152\/095820211X13118583635693","volume":"20","author":"BR Martin","year":"2011","unstructured":"Martin, B. R. (2011). The research excellence framework and the \u2018impact agenda\u2019: Are we creating a Frankenstein monster? Research Evaluation, 20(3), 247\u2013254.","journal-title":"Research Evaluation"},{"issue":"1","key":"1020_CR26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/S0272-7358(96)00042-6","volume":"17","author":"GE Matt","year":"1997","unstructured":"Matt, G. E., & Navarro, A. M. (1997). What meta-analyses have and have not taught us about psychotherapy effects: A review and future directions. Clinical Psychology Review, 17(1), 1\u201332.","journal-title":"Clinical Psychology Review"},{"key":"1020_CR27","volume-title":"The sociology of science: Theoretical and empirical investigations","author":"RK Merton","year":"1973","unstructured":"Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago: University of Chicago Press."},{"issue":"6053","key":"1020_CR28","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1126\/science.334.6053.169","volume":"334","author":"J Mervis","year":"2011","unstructured":"Mervis, J. (2011). Beyond the data. Science, 334(6053), 169\u2013171. doi: 10.1126\/science.334.6053.169 .","journal-title":"Science"},{"key":"1020_CR29","volume-title":"Encyclopedia of sustainability science and technology","year":"2012","unstructured":"Meyers, R. A. (Ed.). (2012). Encyclopedia of sustainability science and technology. New York: Springer."},{"issue":"1","key":"1020_CR30","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1186\/1478-4505-8-30","volume":"8","author":"S Mostert","year":"2010","unstructured":"Mostert, S., Ellenbroek, S., Meijer, I., van Ark, G., & Klasen, E. (2010). Societal output and use of research performed by health research groups. Health Research Policy and Systems, 8(1), 30.","journal-title":"Health Research Policy and Systems"},{"key":"1020_CR31","doi-asserted-by":"crossref","first-page":"588","DOI":"10.1186\/1471-2458-11-588","volume":"11","author":"T Niederkrotenthaler","year":"2011","unstructured":"Niederkrotenthaler, T., Dorner, T. E., & Maier, M. (2011). Development of a practical tool to measure the impact of publications on the society based on focus group discussions with scientists. BMC Public Health, 11, 588.","journal-title":"BMC Public Health"},{"issue":"2","key":"1020_CR32","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1017\/S1062798704000195","volume":"12","author":"JC Petit","year":"2004","unstructured":"Petit, J. C. (2004). Why do we need fundamental research? European Review, 12(2), 191\u2013207.","journal-title":"European Review"},{"issue":"7431","key":"1020_CR33","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1038\/493159a","volume":"493","author":"H Piwowar","year":"2013","unstructured":"Piwowar, H. (2013). Altmetrics: Value all research products. Nature, 493(7431), 159.","journal-title":"Nature"},{"issue":"1","key":"1020_CR34","doi-asserted-by":"crossref","first-page":"50","DOI":"10.7326\/0003-4819-154-1-201101040-00007","volume":"154","author":"KA Robinson","year":"2011","unstructured":"Robinson, K. A., & Goodman, S. N. (2011). A systematic examination of the citation of prior research in reports of randomized, controlled trials. Annals of Internal Medicine, 154(1), 50\u201355.","journal-title":"Annals of Internal Medicine"},{"key":"1020_CR12","volume-title":"Research quality framework: Assessing the quality and impact of research in Australia. The recommended RQF (report by the RQF Development Advisory Group)","author":"RQF development advisory group","year":"2006","unstructured":"RQF development advisory group. (2006). Research quality framework: Assessing the quality and impact of research in Australia. The recommended RQF (report by the RQF Development Advisory Group). Canberra, Australia: Department of Education, Science and Training."},{"key":"1020_CR35","volume-title":"Measuring the impact of research\u2014the context for metric development","author":"L Rymer","year":"2011","unstructured":"Rymer, L. (2011). Measuring the impact of research\u2014the context for metric development. Turner: The Group of Eight."},{"issue":"3","key":"1020_CR36","doi-asserted-by":"crossref","first-page":"509","DOI":"10.1016\/S0048-7333(00)00091-3","volume":"30","author":"AJ Salter","year":"2001","unstructured":"Salter, A. J., & Martin, B. R. (2001). The economic benefits of publicly funded basic research: A critical review. Research Policy, 30(3), 509\u2013532.","journal-title":"Research Policy"},{"key":"1020_CR37","unstructured":"Smith, C.H.L. (1997). What\u2019s the use of basic science? Retrieved 6 April 2013, from http:\/\/wwwnew.jinr.ru\/section.asp?sd_id=94 ."},{"issue":"5","key":"1020_CR38","doi-asserted-by":"crossref","first-page":"348","DOI":"10.7326\/0003-4819-157-5-201209040-00007","volume":"157","author":"C Smith-Spangler","year":"2012","unstructured":"Smith-Spangler, C., Brandeau, M. L., Hunter, G. E., Bavinger, J. C., Pearson, M., Eschbach, P. J., et al. (2012). Are organic foods safer or healthier than conventional alternatives? A systematic review. Annals of Internal Medicine, 157(5), 348\u2013366. doi: 10.7326\/0003-4819-157-5-201209040-00007 .","journal-title":"Annals of Internal Medicine"},{"issue":"1","key":"1020_CR39","doi-asserted-by":"crossref","first-page":"11","DOI":"10.3152\/147154400781777449","volume":"9","author":"B Meulen van der","year":"2000","unstructured":"van der Meulen, B., & Rip, A. (2000). Evaluation of societal quality of public sector research in the Netherlands. Research Evaluation, 9(1), 11\u201325.","journal-title":"Research Evaluation"},{"key":"1020_CR40","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511541391","volume-title":"Real science. What it is, and what it means","author":"J Ziman","year":"2000","unstructured":"Ziman, J. (2000). Real science. What it is, and what it means. Cambridge: Cambridge University Press."}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-013-1020-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s11192-013-1020-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-013-1020-x","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,7,12]],"date-time":"2019-07-12T09:14:46Z","timestamp":1562922886000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s11192-013-1020-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,4,13]]},"references-count":40,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2014,1]]}},"alternative-id":["1020"],"URL":"https:\/\/doi.org\/10.1007\/s11192-013-1020-x","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,4,13]]}}}