{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T14:23:20Z","timestamp":1768487000142,"version":"3.49.0"},"reference-count":51,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[1996,7,1]],"date-time":"1996-07-01T00:00:00Z","timestamp":836179200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[1996,7]]},"DOI":"10.1007\/bf02129599","type":"journal-article","created":{"date-parts":[[2005,9,14]],"date-time":"2005-09-14T15:42:30Z","timestamp":1126712550000},"page":"343-362","source":"Crossref","is-referenced-by-count":236,"title":["The use of multiple indicators in the assessment of basic research"],"prefix":"10.1007","volume":"36","author":[{"given":"B. R.","family":"Martin","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","reference":[{"key":"BF02129599_CR1","doi-asserted-by":"crossref","first-page":"1229","DOI":"10.1126\/science.132.3435.1229","volume":"132","author":"J. H. Westbrook","year":"1960","unstructured":"E.g.J. H. Westbrook, Identifying significant research,Science, 132 (1960) 1229\u20131234;N. Wade, Citation analysis: a new tool for science administrators,Science, 188 (1975) 429\u2013432.","journal-title":"Science"},{"key":"BF02129599_CR2","first-page":"67","volume-title":"Science and Technology Policy in the 1980s and Beyond","author":"J. Irvine","year":"1984","unstructured":"J. Irvine, B. R. Martin, What direction for basic scientific research?, Chapter 5 inM. Gibbons, P. Gummett, B. M. Udgaonkar (eds.),Science and Technology Policy in the 1980s and Beyond, London, Longman, 1984, pp. 67\u201398."},{"key":"BF02129599_CR3","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/0048-7333(83)90005-7","volume":"12","author":"B. R. Martin","year":"1983","unstructured":"B. R. Martin, J. Irvine, Assessing basic research: some partial indicators of scientific progress in radio astronomy,Research Policy, 12 (1983) 61\u201390.","journal-title":"Research Policy"},{"key":"BF02129599_CR4","unstructured":"E.g.Anon, Is your lab well cited?,Nature, 227 (1970) 219;Anon, More games with numbers,Nature, 2228 (1970) 698\u2013699."},{"key":"BF02129599_CR5","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1177\/030631278001000202","volume":"10","author":"D. Lindsey","year":"1980","unstructured":"E.g.D. Lindsey, Production and citation measures in the sociology of science: the problem of multiple authorship,Social Studies of Science, 10 (1980) 145\u2013162.","journal-title":"Social Studies of Science"},{"key":"BF02129599_CR6","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1016\/0048-7333(84)90014-3","volume":"13","author":"B. R. Martin","year":"1984","unstructured":"The sixth, CERN, was left to a subsequent study two years later. The results were published in a series of three articles:B. R. Martin, J. Irvine, CERN: past peformance and future prospects-I-CERN's position in World High-Energy Physics,Research Policy, 13 (1984) 183\u2013210;J. Irvine, B. R. Martin, CERN: past performance and future prospects-II-The scientific performance of the CERN accelerators,Research Policy, 13 (1984) 247\u2013284; andB. R. Martin, J. Irvine, CERN: Past performance and future prospects-III-CERN and the future of world high-energy physics,Research Policy, 13 (1984) 311\u2013342.","journal-title":"Research Policy"},{"key":"BF02129599_CR7","first-page":"83","volume":"6","author":"J. Irvine","year":"1980","unstructured":"J. Irvine, B. R. Martin, A methodology for assessing the scientific performance of research groups,Scientia Yugoslavia, 6 (1980) 83\u201395.","journal-title":"Scientia Yugoslavia"},{"key":"BF02129599_CR8","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/0048-7333(83)90005-7","volume":"12","author":"Martin Irvine","year":"1983","unstructured":"Martin, Irvine, \u2014\u2014op. cit., note 3. This article appeared in April 1983 even though it had been accepted for publication in September 1980.","journal-title":"Research Policy"},{"key":"BF02129599_CR9","volume-title":"Investing in the Future: An International Comparison of Government Funding of Academic and Related Research","author":"J. Irvine","year":"1990","unstructured":"J. Irvine, B. R. Martin, P. A. Isard,Investing in the Future: An International Comparison of Government Funding of Academic and Related Research, Aldershot and Brookfield, Vermont, Edward Elgar, 1990."},{"key":"BF02129599_CR10","first-page":"67","volume-title":"Science and Technology Policy in the 1980s and Beyond","author":"Irvine Martin","year":"1984","unstructured":"Irvine, Martin, \u2014\u2014op. cit., note 2."},{"key":"BF02129599_CR11","first-page":"1","volume-title":"Science Policy & Research Management: Can Research Indicators Be Used?","author":"H. F. Hansen","year":"1995","unstructured":"H. F. Hansen, B. H. J\u00f8rgensen,Science Policy & Research Management: Can Research Indicators Be Used?, Institute of Political Science, University of Copenhagen, Copenhagen, 1995, p. 1."},{"key":"BF02129599_CR12","unstructured":"R. N. Kostoff,The Handbook of Research Impact Assessment (Fifth Edition), DTIC Report Number ADA296021, 1995."},{"key":"BF02129599_CR13","unstructured":"For example, the evaluation of government-funded applied research in Norway employed a combination of peer review and \u2018customer review\u2019-seeJ. Irvine, B. R. Martin, M. Schwarz, K. Pavitt, R. Rothwell,Government Support for Industrial Research in Norway: A SPRU Report, Oslo: Universitetsforlaget Norwegian Official Publication NOU 30B, 1981."},{"key":"BF02129599_CR14","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/0048-7333(83)90005-7","volume":"12","author":"Martin Irvine","year":"1983","unstructured":"See Fig. 1 on p. 64 inMartin, Irvine, \u2014\u2014op. cit., note 3.","journal-title":"Research Policy"},{"key":"BF02129599_CR15","unstructured":"A good example here would be popular books by scientists such asStephen Hawking."},{"key":"BF02129599_CR16","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1016\/0048-7333(83)90005-7","volume":"12","author":"B. R. Martin","year":"1983","unstructured":"Ibid., note 3, p. 64.","journal-title":"Research Policy"},{"key":"BF02129599_CR17","unstructured":"Kostoff,op. cit., note 12R. N. Kostoff,The Handbook of Research Impact Assessment (Fifth Edition), DTIC Report Number ADA296021, 1995, p. 8."},{"key":"BF02129599_CR18","first-page":"75","volume":"12","author":"Martin Irvine","year":"1983","unstructured":"Martin, Irvine, \u2014\u2014op. cit., note 3, p. 75.","journal-title":"Research Policy"},{"key":"BF02129599_CR19","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1111\/j.1467-9310.1992.tb00785.x","volume":"22","author":"R. Miller","year":"1992","unstructured":"R. Miller, The influence of primary task on R&D laboratory evaluation: a comparative bibliometric analysis,R&D Management, 22 (1992) 3\u201320.","journal-title":"R&D Management"},{"key":"BF02129599_CR20","unstructured":"For an example of how the educational technological outputs from basic research may be assessed, see the references cited in note 21.J. Irvine, B. R. Martin, The Economic Effects of Big Science: The Case of Radio Astronomy,Proceedings of the International Colloquium on the Economic Effects of Space and Other Advanced Technologies, Strasbourg, 28\u201330 April 1980, Paris, European Space Agency, ESA SP-151, 1980;"},{"key":"BF02129599_CR21","series-title":"ESA SP-151","volume-title":"The Economic Effects of Big Science: The Case of Radio Astronomy","author":"J. Irvine","year":"1980","unstructured":"J. Irvine, B. R. Martin,The Economic Effects of Big Science: The Case of Radio Astronomy,Proceedings of the International Colloquium on the Economic Effects of Space and Other Advanced Technologies, Strasbourg, 28\u201330 April 1980 Paris, European Space Agency, ESA SP-151, 1980; andB. R. Martin, J. Irvine, Spin-Off from Basic Science: The Case of Radio Astronomy,Physics in Technology, 12 (1981) 204\u2013212."},{"key":"BF02129599_CR22","unstructured":"Over 150 scientists were interviewed in the \u2018Big Science Project\u2019."},{"key":"BF02129599_CR23","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/0048-7333(83)90005-7","volume":"12","author":"Martin Irvine","year":"1983","unstructured":"This section draws heavily onMartin, Irvine,op. cit., note 3 61\u201390.","journal-title":"Research Policy"},{"key":"BF02129599_CR24","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1016\/0048-7333(83)90005-7","volume":"12","author":"B. R. Martin","year":"1983","unstructured":"See, for example, the discussion in Ibid. p. 67.","journal-title":"Research Policy"},{"key":"BF02129599_CR25","unstructured":"T. Luukkonen, The cognitive and social foundation of citation studies-why we still lack a theory of citation, submitted toScience, Technology and Human Values (1995)."},{"key":"BF02129599_CR26","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1177\/030631295025003003","volume":"25","author":"W. R. Shadish","year":"1995","unstructured":"W. R. Shadish, D. Tolliver, M. Gray, S. K. Sen Gupta, Author judgements about works they cite: three studies from psychology journals,Social Studies of Science, 25 (1995) 447\u2013498 \u2014 quote on p. 481.","journal-title":"Social Studies of Science"},{"key":"BF02129599_CR27","first-page":"3","volume-title":"Science Policy & Research Management: Can Research Indicators Be Used?","author":"H. F. Hansen","year":"1995","unstructured":"See also the related distinction between \u2018quality\u2019 and \u2018relevance\u2019 inHansen, J\u00f8rgensen,op. cit., note 11, p. 3."},{"key":"BF02129599_CR28","first-page":"70","volume":"12","author":"Martin Irvine","year":"1983","unstructured":"Martin, Irvine,op. cit., note 3, p. 70.","journal-title":"Research Policy"},{"key":"BF02129599_CR29","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1016\/0048-7333(83)90005-7","volume":"12","author":"B. R. Martin","year":"1983","unstructured":"Ibid..","journal-title":"Research Policy"},{"key":"BF02129599_CR30","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1016\/0048-7333(83)90005-7","volume":"12","author":"B. R. Martin","year":"1983","unstructured":"Ibid..","journal-title":"Research Policy"},{"key":"BF02129599_CR31","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1016\/0048-7333(83)90005-7","volume":"12","author":"B. R. Martin","year":"1983","unstructured":"Ibid..","journal-title":"Research Policy"},{"key":"BF02129599_CR32","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1016\/0048-7333(84)90014-3","volume":"13","author":"Martin Irvine","year":"1984","unstructured":"Examples of this in the field of experimental high-energy physics can be found inMartin, Irvine,op. cit., note 6.","journal-title":"Research Policy"},{"key":"BF02129599_CR33","volume-title":"The Structure of Scientific Revolutions","author":"T. S. Kuhn","year":"1970","unstructured":"T. S. Kuhn,The Structure of Scientific Revolutions, Chicago, University of Chicago Press, 1970."},{"key":"BF02129599_CR34","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/0048-7333(83)90005-7","volume":"12","author":"Martin Irvine","year":"1983","unstructured":"Martin, Irvine,op. cit., note 3,idem.,op. cit., note 6 CERN: past performance and future prospects-I-CERN's position in World High-Energy Physics,Research Policy, 13 (1984) 183\u2013210.","journal-title":"Research Policy"},{"key":"BF02129599_CR35","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1016\/0048-7333(83)90005-7","volume":"12","author":"Martin Irvine","year":"1983","unstructured":"Martin, Irvine,op. cit., note 3.","journal-title":"Research Policy"},{"key":"BF02129599_CR36","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1177\/030631283013001004","volume":"13","author":"J. Irvine","year":"1983","unstructured":"J. Irvine, B. R. Martin, Assessing basic research: The case of the Isaac Newton Telescope,Social Studies of Science, 13 (1983) 49\u201386.","journal-title":"Social Studies of Science"},{"key":"BF02129599_CR37","doi-asserted-by":"crossref","first-page":"408","DOI":"10.1007\/BF02192823","volume":"XIX","author":"B. R. Martin","year":"1981","unstructured":"B. R. Martin, J. Irvine, Internal criteria for scientific choice: an evaluation of the research performance of electron high-energy physics accelerators,Minerva, XIX (1981) 408\u2013432.","journal-title":"Minerva"},{"key":"BF02129599_CR38","doi-asserted-by":"crossref","first-page":"183","DOI":"10.1016\/0048-7333(84)90014-3","volume":"13","author":"Irvine Martin","year":"1984","unstructured":"Martin, Irvine op. cit., note 6.","journal-title":"Research Policy"},{"key":"BF02129599_CR39","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1177\/016555159402000102","volume":"20","author":"L. M. Baird","year":"1994","unstructured":"L. M. Baird, C. Oppenheim, Do citations matter?,Journal of Information Science, 20 (1994) 2\u201315 (quote on p. 13).","journal-title":"Journal of Information Science"},{"key":"BF02129599_CR40","unstructured":"Kostoff,op. cit., note 12The Handbook of Research Impact Assessment (Fifth Edition), DTIC Report Number ADA296021, 1995. p. 37."},{"key":"BF02129599_CR41","unstructured":"Ibid.R. N. Kostoff,The Handbook of Research Impact Assessment (Fifth Edition), DTIC Report Number ADA296021, 1995 p. 118"},{"key":"BF02129599_CR42","unstructured":"A. H. Rubenstein, E. Geisler, Evaluating the outputs and impacts of R&D\/innovation,International Journal of Technology Management, 6 (1991)."},{"key":"BF02129599_CR43","unstructured":"Not all scientometric analysts are guilty of this. For example, the ISI analysts who periodically publish lists of leading research institutes inScience Watch normally use three indicators-papers, citations and citations per paper."},{"key":"BF02129599_CR44","volume-title":"Academic Research Performance Indicators: An Assessment of the Possibilities","author":"B. R. Martin","year":"1992","unstructured":"Full details of the study and the results can be found inB. R. Martin, J. E. F. Skea,Academic Research Performance Indicators: An Assessment of the Possibilities, Brighton, SPRU, 1992."},{"key":"BF02129599_CR45","volume-title":"Academic Research Performance Indicators: An Assessment of the Possibilities","author":"B. R. Martin","year":"1992","unstructured":"See Table 12 inibid.."},{"key":"BF02129599_CR46","unstructured":"Kostoff,op cit, note 12R. N. Kostoff,The Handbook of Research Impact Assessment (Fifth Edition), DTIC Report Number ADA296021, 1995, p. 8."},{"key":"BF02129599_CR47","first-page":"5","volume-title":"Science Policy & Research Management: Can Research Indicators Be Used?","author":"H. F. Hansen","year":"1995","unstructured":"Hansen, J\u00f8rgensen,op. cit., note 11, p. 5."},{"key":"BF02129599_CR48","first-page":"75","volume-title":"Academic Research Performance Indicators: An Assessment of the Possibilities","author":"Martin Skea","year":"1992","unstructured":"Martin, Skea,op. cit., note 44,. p. 75."},{"key":"BF02129599_CR49","first-page":"75","volume":"1","author":"J. P. Greve de","year":"1989","unstructured":"Ibid. p. 75.","journal-title":"Higher Education Management"},{"key":"BF02129599_CR50","first-page":"83","volume":"1","author":"J. P. Greve de","year":"1989","unstructured":"J. P. de Greve, A. Frijdal, Evaluation of scientific research prolife analysis \u2014 a mixed method,Higher Education Management, 1 (1989) 83\u201390.","journal-title":"Higher Education Management"},{"key":"BF02129599_CR51","unstructured":"Kostoff,op. cit., note 12The Handbook of Research Impact Assessment (Fifth Edition), DTIC Report Number ADA296021, 1995. p. 9."}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/BF02129599.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/BF02129599\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/BF02129599","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,5,14]],"date-time":"2019-05-14T15:30:46Z","timestamp":1557847846000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/BF02129599"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[1996,7]]},"references-count":51,"journal-issue":{"issue":"3","published-print":{"date-parts":[[1996,7]]}},"alternative-id":["BF02129599"],"URL":"https:\/\/doi.org\/10.1007\/bf02129599","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[1996,7]]}}}