{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,18]],"date-time":"2026-02-18T11:11:58Z","timestamp":1771413118791,"version":"3.50.1"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2020,5,22]],"date-time":"2020-05-22T00:00:00Z","timestamp":1590105600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,5,22]],"date-time":"2020-05-22T00:00:00Z","timestamp":1590105600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2020,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Percentiles are statistics pointing to the standing of a paper\u2019s citation impact relative to other papers in a given citation distribution. Percentile Ranks (<jats:italic>PR<\/jats:italic>s) often play an important role in evaluating the impact of researchers, institutions, and similar\u00a0lines of study. Because<jats:italic>PR<\/jats:italic>s are so important for the assessment of scholarly impact, and because citations differ greatly across time and fields, various percentile approaches have been proposed to time- and field-normalize citations. Unfortunately, current popular methods often face significant problems in time- and field-normalization, including when papers are assigned to multiple fields or have been published by more than one unit (e.g., researchers or countries). They also face problems for estimating citation counts for pre-defined<jats:italic>PR<\/jats:italic>s (e.g., the 90th<jats:italic>PR<\/jats:italic>). We offer a series of guidelines and procedures that, we argue, address these problems and others and provide a superior means to make the use of percentile methods more accurate and informative. In particular, we introduce two approaches,<jats:italic>CP<\/jats:italic>-<jats:italic>IN<\/jats:italic>and<jats:italic>CP<\/jats:italic>-<jats:italic>EX<\/jats:italic>, that should be preferred in bibliometric studies because they consider the complete citation distribution\u00a0and can be accurately interpreted. Both approaches are based on cumulative frequencies in percentages (<jats:italic>CP<\/jats:italic>s). The paper further shows how bar graphs and beamplots can present<jats:italic>PR<\/jats:italic>s in a more meaningful and accurate manner.<\/jats:p>","DOI":"10.1007\/s11192-020-03512-7","type":"journal-article","created":{"date-parts":[[2020,5,22]],"date-time":"2020-05-22T05:02:34Z","timestamp":1590123754000},"page":"1457-1478","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":36,"title":["An evaluation of percentile measures of citation impact, and a proposal for making them better"],"prefix":"10.1007","volume":"124","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0810-7091","authenticated-orcid":false,"given":"Lutz","family":"Bornmann","sequence":"first","affiliation":[]},{"given":"Richard","family":"Williams","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,5,22]]},"reference":[{"key":"3512_CR1","volume-title":"Profiles, not metrics","author":"J Adams","year":"2019","unstructured":"Adams, J., McVeigh, M., Pendlebury, D., & Szomszor, M. (2019). Profiles, not metrics. Philadelphia, PA: Clarivate Analytics."},{"key":"3512_CR2","unstructured":"Barrett, P. (2003). Percentiles and textbook definitions\u2014Confused or what? Retrieved November 11, 2019, from https:\/\/www.pbarrett.net\/techpapers\/percentiles.pdf."},{"issue":"3","key":"3512_CR3","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1002\/asi.22792","volume":"64","author":"L Bornmann","year":"2013","unstructured":"Bornmann, L. (2013). How to analyze percentile citation impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes, and top-cited papers. Journal of the American Society for Information Science and Technology,64(3), 587\u2013595. https:\/\/doi.org\/10.1002\/asi.22792.","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"2","key":"3512_CR4","doi-asserted-by":"publisher","first-page":"166","DOI":"10.1093\/reseval\/rvu002","volume":"23","author":"L Bornmann","year":"2014","unstructured":"Bornmann, L. (2014). How are excellent (highly cited) papers defined in bibliometrics? A quantitative analysis of the literature. Research Evaluation,23(2), 166\u2013173. https:\/\/doi.org\/10.1093\/reseval\/rvu002.","journal-title":"Research Evaluation"},{"issue":"2","key":"3512_CR5","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1016\/j.joi.2011.11.006","volume":"6","author":"L Bornmann","year":"2012","unstructured":"Bornmann, L., de Moya Aneg\u00f3n, F., & Leydesdorff, L. (2012). The new excellence indicator in the world report of the SCImago institutions rankings 2011. Journal of Informetrics,6(2), 333\u2013335. https:\/\/doi.org\/10.1016\/j.joi.2011.11.006.","journal-title":"Journal of Informetrics"},{"issue":"1","key":"3512_CR6","doi-asserted-by":"publisher","first-page":"385","DOI":"10.1007\/s11192-018-2658-1","volume":"115","author":"L Bornmann","year":"2018","unstructured":"Bornmann, L., & Haunschild, R. (2018). Plots for visualizing paper impact and journal impact of single researchers in a single graph. Scientometrics,115(1), 385\u2013394. https:\/\/doi.org\/10.1007\/s11192-018-2658-1.","journal-title":"Scientometrics"},{"issue":"1","key":"3512_CR7","doi-asserted-by":"publisher","first-page":"158","DOI":"10.1016\/j.joi.2012.10.001","volume":"7","author":"L Bornmann","year":"2013","unstructured":"Bornmann, L., Leydesdorff, L., & Mutz, R. (2013a). The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits. Journal of Informetrics,7(1), 158\u2013165. https:\/\/doi.org\/10.1016\/j.joi.2012.10.001.","journal-title":"Journal of Informetrics"},{"issue":"4","key":"3512_CR8","doi-asserted-by":"publisher","first-page":"933","DOI":"10.1016\/j.joi.2013.09.003","volume":"7","author":"L Bornmann","year":"2013","unstructured":"Bornmann, L., Leydesdorff, L., & Wang, J. (2013b). Which percentile-based approach should be preferred for calculating normalized citation impact values? An empirical comparison of five approaches including a newly developed citation-rank approach (P100). Journal of Informetrics,7(4), 933\u2013944. https:\/\/doi.org\/10.1016\/j.joi.2013.09.003.","journal-title":"Journal of Informetrics"},{"issue":"2","key":"3512_CR9","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1007\/s11192-019-03018-x","volume":"120","author":"L Bornmann","year":"2019","unstructured":"Bornmann, L., & Marewski, J. N. (2019). Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation. Scientometrics,120(2), 419\u2013459.","journal-title":"Scientometrics"},{"issue":"1","key":"3512_CR10","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1002\/asi.22996","volume":"65","author":"L Bornmann","year":"2014","unstructured":"Bornmann, L., & Marx, W. (2014a). Distributions instead of single numbers: Percentiles and beam plots for the assessment of single researchers. Journal of the American Society of Information Science and Technology,65(1), 206\u2013208.","journal-title":"Journal of the American Society of Information Science and Technology"},{"issue":"1","key":"3512_CR11","doi-asserted-by":"publisher","first-page":"487","DOI":"10.1007\/s11192-013-1161-y","volume":"98","author":"L Bornmann","year":"2014","unstructured":"Bornmann, L., & Marx, W. (2014b). How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations. Scientometrics,98(1), 487\u2013509. https:\/\/doi.org\/10.1007\/s11192-013-1161-y.","journal-title":"Scientometrics"},{"issue":"2","key":"3512_CR12","doi-asserted-by":"publisher","first-page":"408","DOI":"10.1016\/j.joi.2015.01.006","volume":"9","author":"L Bornmann","year":"2015","unstructured":"Bornmann, L., & Marx, W. (2015). Methods for the generation of normalized citation impact scores in bibliometrics: Which method best reflects the judgements of experts? Journal of Informetrics,9(2), 408\u2013418.","journal-title":"Journal of Informetrics"},{"issue":"9","key":"3512_CR13","doi-asserted-by":"publisher","first-page":"1939","DOI":"10.1002\/asi.23152","volume":"65","author":"L Bornmann","year":"2014","unstructured":"Bornmann, L., & Mutz, R. (2014). From P100 to P100\u2032: A new citation-rank approach. Journal of the Association For Information Science and Technology,65(9), 1939\u20131943. https:\/\/doi.org\/10.1002\/asi.23152.","journal-title":"Journal of the Association For Information Science and Technology"},{"key":"3512_CR14","unstructured":"Cox, N. J. (2005). Calculating percentile ranks or plotting positions. Retrieved May 30, 2019, from http:\/\/www.stata.com\/support\/faqs\/stat\/pcrank.html."},{"issue":"4","key":"3512_CR15","doi-asserted-by":"publisher","first-page":"289","DOI":"10.2307\/2685780","volume":"54","author":"DP Doane","year":"2000","unstructured":"Doane, D. P., & Tracy, R. L. (2000). Using beam and fulcrum displays to explore data. American Statistician,54(4), 289\u2013290. https:\/\/doi.org\/10.2307\/2685780.","journal-title":"American Statistician"},{"key":"3512_CR16","doi-asserted-by":"publisher","DOI":"10.1108\/S1876-0562(2005)05","volume-title":"Power laws in the information production process: Lotkaian informetrics","author":"L Egghe","year":"2005","unstructured":"Egghe, L. (2005). Power laws in the information production process: Lotkaian informetrics. Kidlington: Elsevier Academic Press."},{"key":"3512_CR17","first-page":"120","volume-title":"The evaluation of scientific research","author":"D Evered","year":"1989","unstructured":"Evered, D., Hamett, S., & Narin, F. (1989). The impact of different modes of research funding. In D. Evered & S. Hamett (Eds.), The evaluation of scientific research (pp. 120\u2013140). Chichester: Wiley."},{"issue":"1","key":"3512_CR18","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1007\/s11192-007-1934-2","volume":"77","author":"M Gauffriau","year":"2008","unstructured":"Gauffriau, M., Larsen, P. O., Maye, I., Roulin-Perriard, A., & von Ins, M. (2008). Comparisons of results of publication counting using different methods. Scientometrics,77(1), 147\u2013176. https:\/\/doi.org\/10.1007\/s11192-007-1934-2.","journal-title":"Scientometrics"},{"key":"3512_CR19","doi-asserted-by":"crossref","first-page":"1539","DOI":"10.1061\/taceat.0002563","volume":"77","author":"A Hazen","year":"1914","unstructured":"Hazen, A. (1914). Storage to be provided in impounding reservoirs for municipal water supply. Transactions of American Society of Civil Engineers,77, 1539\u20131640.","journal-title":"Transactions of American Society of Civil Engineers"},{"issue":"7548","key":"3512_CR20","doi-asserted-by":"publisher","first-page":"429","DOI":"10.1038\/520429a","volume":"520","author":"D Hicks","year":"2015","unstructured":"Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden manifesto for research metrics. Nature,520(7548), 429\u2013431.","journal-title":"Nature"},{"issue":"9","key":"3512_CR21","doi-asserted-by":"publisher","first-page":"e1002542","DOI":"10.1371\/journal.pbio.1002542","volume":"14","author":"JPA Ioannidis","year":"2016","unstructured":"Ioannidis, J. P. A., Boyack, K., & Wouters, P. F. (2016). Citation metrics: A primer on how (not) to normalize. PLoS Biology,14(9), e1002542. https:\/\/doi.org\/10.1371\/journal.pbio.1002542.","journal-title":"PLoS Biology"},{"key":"3512_CR22","volume-title":"Encyclopedia of survey research methods","year":"2008","unstructured":"Lavrakas, P. J. (Ed.). (2008). Encyclopedia of survey research methods. Thousand Oaks, CA: Sage."},{"issue":"11","key":"3512_CR23","doi-asserted-by":"publisher","first-page":"2133","DOI":"10.1002\/asi.21609","volume":"62","author":"L Leydesdorff","year":"2011","unstructured":"Leydesdorff, L., & Bornmann, L. (2011). Integrated impact indicators (I3) compared with impact factors (IFs): An alternative research design with policy implications. Journal of the American Society of Information Science and Technology,62(11), 2133\u20132146.","journal-title":"Journal of the American Society of Information Science and Technology"},{"issue":"9","key":"3512_CR24","doi-asserted-by":"publisher","first-page":"1901","DOI":"10.1002\/asi.22641","volume":"63","author":"L Leydesdorff","year":"2012","unstructured":"Leydesdorff, L., & Bornmann, L. (2012). Percentile ranks and the integrated impact indicator (I3). Journal of the American Society for Information Science and Technology,63(9), 1901\u20131902. https:\/\/doi.org\/10.1002\/asi.22641.","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"3","key":"3512_CR25","doi-asserted-by":"publisher","first-page":"1669","DOI":"10.1007\/s11192-019-03099-8","volume":"119","author":"L Leydesdorff","year":"2019","unstructured":"Leydesdorff, L., Bornmann, L., & Adams, J. (2019). The integrated impact indicator (I3) revisited: A non-parametric alternative to the journal impact factor. Scientometrics,119(3), 1669\u20131694.","journal-title":"Scientometrics"},{"issue":"4","key":"3512_CR26","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1109\/TEM.1983.6448622","volume":"30","author":"PR McAllister","year":"1983","unstructured":"McAllister, P. R., Narin, F., & Corrigan, J. G. (1983). Programmatic evaluation and comparison based on standardized citation scores. IEEE Transactions on Engineering Management,30(4), 205\u2013211.","journal-title":"IEEE Transactions on Engineering Management"},{"issue":"2","key":"3512_CR27","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1093\/spp\/14.2.99","volume":"14","author":"F Narin","year":"1987","unstructured":"Narin, F. (1987). Bibliometric techniques in the evaluation of research programs. Science and Public Policy,14(2), 99\u2013106. https:\/\/doi.org\/10.1093\/spp\/14.2.99.","journal-title":"Science and Public Policy"},{"issue":"5\u20136","key":"3512_CR28","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1007\/BF02017249","volume":"9","author":"A Schubert","year":"1986","unstructured":"Schubert, A., & Braun, T. (1986). Relative indicators and relational charts for comparative assessment of publication output and citation impact. Scientometrics,9(5\u20136), 281\u2013291.","journal-title":"Scientometrics"},{"key":"3512_CR29","volume-title":"Stata statistical software: Release 15","author":"StataCorp.","year":"2017","unstructured":"StataCorp. (2017). Stata statistical software: Release 15. College Station, TX: Stata Corporation."},{"issue":"3","key":"3512_CR30","doi-asserted-by":"publisher","first-page":"906","DOI":"10.1016\/j.joi.2018.07.005","volume":"12","author":"I Tahamtan","year":"2018","unstructured":"Tahamtan, I., & Bornmann, L. (2018). Creativity in science and the link to cited references: Is the creative potential of papers reflected in their cited references? Journal of Informetrics,12(3), 906\u2013930. https:\/\/doi.org\/10.1016\/j.joi.2018.07.005.","journal-title":"Journal of Informetrics"},{"issue":"2","key":"3512_CR31","doi-asserted-by":"publisher","first-page":"365","DOI":"10.1016\/j.joi.2016.02.007","volume":"10","author":"L Waltman","year":"2016","unstructured":"Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics,10(2), 365\u2013391.","journal-title":"Journal of Informetrics"},{"issue":"12","key":"3512_CR32","doi-asserted-by":"publisher","first-page":"2419","DOI":"10.1002\/asi.22708","volume":"63","author":"L Waltman","year":"2012","unstructured":"Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E. C. M., Tijssen, R. J. W., van Eck, N. J., et al. (2012). The Leiden Ranking 2011\/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology,63(12), 2419\u20132432.","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"2","key":"3512_CR33","doi-asserted-by":"publisher","first-page":"372","DOI":"10.1002\/asi.22775","volume":"64","author":"L Waltman","year":"2013","unstructured":"Waltman, L., & Schreiber, M. (2013). On the calculation of percentile-based bibliometric indicators. Journal of the American Society for Information Science and Technology,64(2), 372\u2013379.","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"4","key":"3512_CR34","doi-asserted-by":"publisher","first-page":"872","DOI":"10.1016\/j.joi.2015.08.001","volume":"9","author":"L Waltman","year":"2015","unstructured":"Waltman, L., & van Eck, N. J. (2015). Field-normalized citation impact indicators and the choice of an appropriate counting method. Journal of Informetrics,9(4), 872\u2013894.","journal-title":"Journal of Informetrics"},{"issue":"1","key":"3512_CR35","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/j.joi.2010.08.001","volume":"5","author":"L Waltman","year":"2011","unstructured":"Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. J. (2011). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics,5(1), 37\u201347. https:\/\/doi.org\/10.1016\/j.joi.2010.08.001.","journal-title":"Journal of Informetrics"},{"key":"3512_CR36","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1007\/978-3-319-10377-8_12","volume-title":"Measuring scholarly impact: Methods and practice","author":"R Williams","year":"2014","unstructured":"Williams, R., & Bornmann, L. (2014). The substantive and practical significance of citation impact differences between institutions: Guidelines for the analysis of percentiles using effect sizes and confidence intervals. In Y. Ding, R. Rousseau, & D. Wolfram (Eds.), Measuring scholarly impact: Methods and practice (pp. 259\u2013281). Heidelberg: Springer."}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03512-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11192-020-03512-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-020-03512-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,6]],"date-time":"2024-08-06T05:46:31Z","timestamp":1722923191000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11192-020-03512-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,22]]},"references-count":36,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,8]]}},"alternative-id":["3512"],"URL":"https:\/\/doi.org\/10.1007\/s11192-020-03512-7","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,5,22]]},"assertion":[{"value":"13 January 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 May 2020","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}