{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T05:19:28Z","timestamp":1782364768456,"version":"3.54.5"},"reference-count":22,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,7,31]],"date-time":"2020-07-31T00:00:00Z","timestamp":1596153600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,7,31]],"date-time":"2020-07-31T00:00:00Z","timestamp":1596153600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2020,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>While bibliometric analysis is normally able to rely on complete publication sets this is not universally the case. For example, Australia (in ERA) and the UK (in the RAE\/REF) use institutional research assessment that may rely on small or fractional parts of researcher output. Using the Category Normalised Citation Impact (CNCI) for the publications of ten universities with similar output (21,000\u201328,000 articles and reviews) indexed in the <jats:italic>Web of Science<\/jats:italic> for 2014\u20132018, we explore the extent to which a \u2018sample\u2019 of institutional data can accurately represent the averages and\/or the correct relative status of the population CNCIs. Starting with full institutional data, we find a high variance in average CNCI across 10,000 institutional samples of fewer than 200 papers, which we suggest may be an analytical minimum although smaller samples may be acceptable for qualitative review. When considering the \u2018top\u2019 CNCI paper in researcher sets represented by DAIS-ID clusters, we find that samples of 1000 papers provide a good guide to relative (but not absolute) institutional citation performance, which is driven by the abundance of high performing individuals. However, such samples may be perturbed by scarce \u2018highly cited\u2019 papers in smaller or less research-intensive units. We draw attention to the significance of this for assessment processes and the further evidence that university rankings are innately unstable and generally unreliable.<\/jats:p>","DOI":"10.1007\/s11192-020-03647-7","type":"journal-article","created":{"date-parts":[[2020,7,31]],"date-time":"2020-07-31T13:05:04Z","timestamp":1596200704000},"page":"777-794","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":172,"title":["Sample size in bibliometric analysis"],"prefix":"10.1007","volume":"125","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9971-2731","authenticated-orcid":false,"given":"Gordon","family":"Rogers","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0347-3527","authenticated-orcid":false,"given":"Martin","family":"Szomszor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0325-4431","authenticated-orcid":false,"given":"Jonathan","family":"Adams","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,7,31]]},"reference":[{"key":"3647_CR1","doi-asserted-by":"publisher","first-page":"565","DOI":"10.2307\/4264","volume":"49","author":"J Adams","year":"1980","unstructured":"Adams, J. (1980). The role of competition in the population dynamics of a freshwater flatworm Bdellocephala punctata (Turbellaria, Tricladida). Journal of Animal Ecology, 49, 565\u2013579.","journal-title":"Journal of Animal Ecology"},{"key":"3647_CR2","doi-asserted-by":"publisher","first-page":"2","DOI":"10.3389\/frma.2020.00002","volume":"5","author":"J Adams","year":"2020","unstructured":"Adams, J., Gurney, K. A., Loach, T., & Szomszor, M. (2020). Evolving document patterns in UK research assessment cycles. Frontiers in Research Metrics and Analytics, 5, 2.","journal-title":"Frontiers in Research Metrics and Analytics"},{"issue":"1","key":"3647_CR3","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1007\/s11192-016-1842-4","volume":"107","author":"MDC Calatrava Moreno","year":"2016","unstructured":"Calatrava Moreno, M. D. C., Auzinger, T., & Werthner, H. (2016). On the uncertainty of interdisciplinarity measurements due to incomplete bibliographic data. Scientometrics, 107(1), 213\u2013232. https:\/\/doi.org\/10.1007\/s11192-016-1842-4.","journal-title":"Scientometrics"},{"issue":"3","key":"3647_CR4","doi-asserted-by":"publisher","first-page":"789","DOI":"10.22197\/rbdpp.v3i3.108","volume":"3","author":"B Capparelli","year":"2017","unstructured":"Capparelli, B., & Giacomolli, N. J. (2017). The evaluation of impact factor in the scientific publication of criminal procedure. Revista Brasileira de Direito Processual Penal, 3(3), 789\u2013806. https:\/\/doi.org\/10.22197\/rbdpp.v3i3.108.","journal-title":"Revista Brasileira de Direito Processual Penal"},{"key":"3647_CR5","unstructured":"ERA. (2018). Excellence in research for Australia: Submission Guidelines, p. 72, \u00a9 Commonwealth of Australia 2017. ISBN: 978-0-9943687-4-4 (online)."},{"issue":"4","key":"3647_CR6","doi-asserted-by":"publisher","first-page":"895","DOI":"10.1016\/j.joi.2015.09.005","volume":"9","author":"R Fairclough","year":"2015","unstructured":"Fairclough, R., & Thelwall, M. (2015). More precise methods for national research citation impact comparisons. Journal of Informetrics, 9(4), 895\u2013906. https:\/\/doi.org\/10.1016\/j.joi.2015.09.005.","journal-title":"Journal of Informetrics"},{"issue":"4","key":"3647_CR7","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1007\/BF02019306","volume":"1","author":"E Garfield","year":"1979","unstructured":"Garfield, E. (1979). Is citation analysis a legitimate evaluation tool? Scientometrics, 1(4), 359\u2013375. https:\/\/doi.org\/10.1007\/BF02019306.","journal-title":"Scientometrics"},{"issue":"1","key":"3647_CR8","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1007\/s11192-013-1022-8","volume":"97","author":"W Gl\u00e4nzel","year":"2013","unstructured":"Gl\u00e4nzel, W. (2013). High-end performance or outlier? Evaluating the tail of scientometric distributions. Scientometrics, 97(1), 13\u201323. https:\/\/doi.org\/10.1007\/s11192-013-1022-8.","journal-title":"Scientometrics"},{"issue":"1","key":"3647_CR9","doi-asserted-by":"publisher","first-page":"381","DOI":"10.1007\/s11192-012-0898-z","volume":"96","author":"W Gl\u00e4nzel","year":"2013","unstructured":"Gl\u00e4nzel, W., & Moed, H. F. (2013). Opinion paper: Thoughts and facts on bibliometric indicators. Scientometrics, 96(1), 381\u2013394. https:\/\/doi.org\/10.1007\/s11192-012-0898-z.","journal-title":"Scientometrics"},{"issue":"1","key":"3647_CR10","doi-asserted-by":"publisher","first-page":"19","DOI":"10.3152\/147154404781776554","volume":"13","author":"J Glaser","year":"2004","unstructured":"Glaser, J., Spurling, T. H., & Butler, L. (2004). Intraorganisational evaluation: Are there \u2018least evaluable units\u2019. Research Evaluation, 13(1), 19\u201332.","journal-title":"Research Evaluation"},{"key":"3647_CR11","unstructured":"HEFCE. (2014). REF2014: Assessment criteria and level definitions. http:\/\/www.ref.ac.uk\/2014\/panels\/assessmentcriteriaandleveldefinitions\/. Last accessed April 06, 2020."},{"issue":"5","key":"3647_CR12","doi-asserted-by":"publisher","first-page":"1030","DOI":"10.1002\/asi.22621","volume":"63","author":"M Levin","year":"2012","unstructured":"Levin, M., Krawczyk, S., Bethard, S., & Jurafsky, D. (2012). Citation-based bootstrapping for large-scale author disambiguation. Journal of the American Society for Information Science and Technology, 63(5), 1030\u20131047.","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"3","key":"3647_CR13","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1007\/BF02016935","volume":"8","author":"H Moed","year":"1985","unstructured":"Moed, H., Burger, W., Frankfort, J., & Van Raan, A. (1985). The application of bibliometric indicators: Important field- and time-dependent factors to be considered. Scientometrics, 8(3), 177\u2013203. https:\/\/doi.org\/10.1007\/BF02016935.","journal-title":"Scientometrics"},{"issue":"4","key":"3647_CR14","doi-asserted-by":"publisher","first-page":"101075","DOI":"10.1016\/j.joi.2020.101075","volume":"14","author":"RWK Potter","year":"2020","unstructured":"Potter, R. W. K., Szomszor, M., & Adams, J. (2020). Interpreting CNCIs on a country-scale: The effect of domestic and international collaboration type. Journal of Informetrics, 14(4), 101075.","journal-title":"Journal of Informetrics"},{"key":"3647_CR15","unstructured":"REF. (2019). Guidance on submissions. Research excellence framework 2019\/01. https:\/\/www.ref.ac.uk\/publications\/guidance-on-submissions-201901\/. Last accessed April 15, 2020."},{"issue":"9","key":"3647_CR16","doi-asserted-by":"publisher","first-page":"628","DOI":"10.1002\/(SICI)1097-4571(199210)43:9<628::AID-ASI5>3.0.CO;2-0","volume":"43","author":"PO Seglen","year":"1992","unstructured":"Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43(9), 628\u2013638.","journal-title":"Journal of the American Society for Information Science"},{"issue":"1","key":"3647_CR17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1002\/(sici)1097-4571(199401)45:1<1::aid-asi1>3.0.co;2-y","volume":"45","author":"PO Seglen","year":"1994","unstructured":"Seglen, P. O. (1994). Causal relationship between article citedness and journal impact. Journal of the American Society for Information Science, 45(1), 1\u201311. https:\/\/doi.org\/10.1002\/(sici)1097-4571(199401)45:1%3c1:aid-asi1%3e3.0.co;2-y.","journal-title":"Journal of the American Society for Information Science"},{"issue":"2","key":"3647_CR18","doi-asserted-by":"publisher","first-page":"653","DOI":"10.1007\/s11192-018-2995-0","volume":"118","author":"Z Shen","year":"2019","unstructured":"Shen, Z., Yang, L., Di, Z., & Wu, J. (2019). Large enough sample size to rank two groups of data reliably according to their means. Scientometrics, 118(2), 653\u2013671. https:\/\/doi.org\/10.1007\/s11192-018-2995-0.","journal-title":"Scientometrics"},{"issue":"4","key":"3647_CR19","doi-asserted-by":"publisher","first-page":"265","DOI":"10.1002\/asi.4630240406","volume":"24","author":"H Small","year":"1973","unstructured":"Small, H. (1973). Co-citation in the scientific literature: A new measure of the relationship between two documents. Journal of the American Society for Information Science, 24(4), 265\u2013269.","journal-title":"Journal of the American Society for Information Science"},{"issue":"1","key":"3647_CR20","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1016\/j.joi.2015.12.001","volume":"10","author":"M Thelwall","year":"2016","unstructured":"Thelwall, M. (2016). The precision of the arithmetic mean, geometric mean and percentiles for citation data: An experimental simulation modelling approach. Journal of Informetrics, 10(1), 110\u2013123. https:\/\/doi.org\/10.1016\/j.joi.2015.12.001.","journal-title":"Journal of Informetrics"},{"issue":"2","key":"3647_CR21","doi-asserted-by":"publisher","first-page":"365","DOI":"10.1016\/j.joi.2016.02.007","volume":"10","author":"L Waltman","year":"2016","unstructured":"Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365\u2013391. https:\/\/doi.org\/10.1016\/j.joi.2016.02.007.","journal-title":"Journal of Informetrics"},{"key":"3647_CR22","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1002\/asi.4630320302","volume":"32","author":"HD White","year":"1981","unstructured":"White, H. D., & Griffith, B. C. (1981). Author co-citation: A literature measure of intellectual structure. Journal of the American Society for Information Science, 32, 163\u2013171.","journal-title":"Journal of the American Society for Information Science"}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03647-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11192-020-03647-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03647-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,31]],"date-time":"2021-07-31T00:48:25Z","timestamp":1627692505000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11192-020-03647-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,31]]},"references-count":22,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,10]]}},"alternative-id":["3647"],"URL":"https:\/\/doi.org\/10.1007\/s11192-020-03647-7","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,31]]},"assertion":[{"value":"11 May 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 July 2020","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"The authors are employees of the Institute for Scientific Information (ISI), which is a part of Clarivate, the owners of the <i>Web of Science<\/i> Group.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of intrerest"}}]}}