{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,1]],"date-time":"2025-10-01T15:30:28Z","timestamp":1759332628326,"version":"3.37.3"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2019,12,14]],"date-time":"2019-12-14T00:00:00Z","timestamp":1576281600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,12,14]],"date-time":"2019-12-14T00:00:00Z","timestamp":1576281600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2020,2]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Although there are now several bibliographic databases of research publications, such as Google Scholar, Pubmed, Scopus, and the Web of Science (WoS), and some also include counts of citations, there is at present no similarly comprehensive database of the rapidly growing number of clinical practice guidelines (CPGs), with their references, which sometimes number in the hundreds. CPGs have been shown to be useful for the evaluation of clinical (as opposed to basic) biomedical research, which often suffers from relatively low counts of citations in the serial literature. The objectives were to introduce a new citation database, clinical impact<jats:sup>\u00ae<\/jats:sup>, and demonstrate how it can be used to evaluate research impact of clinical research publications by exploring the characteristics of CPG citations of two sets of papers, as well as show temporal variation of clinical impact<jats:sup>\u00ae<\/jats:sup> and the WoS. The paper includes the methodology used to retain the data and also the rationale adopted to achieve data quality. The analysis showed that although CPGs tend preferentially to cite papers from their own country, this is not always the case. It also showed that cited papers tend to have a more clinical research level than uncited papers. An analysis of diachronous citations in both clinical impact<jats:sup>\u00ae<\/jats:sup> and the WoS showed that although the WoS citations showed a decreasing trend after a peak at 2\u20133\u00a0years after publication, this was less clear for CPG citations and a longer timescale would be needed to evaluate their impact on these documents.<\/jats:p>","DOI":"10.1007\/s11192-019-03318-2","type":"journal-article","created":{"date-parts":[[2019,12,14]],"date-time":"2019-12-14T12:05:42Z","timestamp":1576325142000},"page":"1221-1235","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["A new database of the references on international clinical practice guidelines: a facility for the evaluation of clinical research"],"prefix":"10.1007","volume":"122","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8637-7981","authenticated-orcid":false,"given":"Magnus","family":"Eriksson","sequence":"first","affiliation":[]},{"given":"Annika","family":"Billhult","sequence":"additional","affiliation":[]},{"given":"Tommy","family":"Billhult","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1967-6345","authenticated-orcid":false,"given":"Elena","family":"Pallari","sequence":"additional","affiliation":[]},{"given":"Grant","family":"Lewison","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2019,12,14]]},"reference":[{"issue":"3","key":"3318_CR1","doi-asserted-by":"publisher","first-page":"1711","DOI":"10.1007\/s11192-016-2143-7","volume":"109","author":"G Abramo","year":"2016","unstructured":"Abramo, G., D\u2019Angelo, C. A., & Soldatenkova, A. (2016). The dispersion of the citation distribution of top scientists\u2019 publications. Scientometrics,109(3), 1711\u20131724. https:\/\/doi.org\/10.1007\/s11192-016-2143-7.","journal-title":"Scientometrics"},{"key":"3318_CR2","doi-asserted-by":"publisher","first-page":"1793","DOI":"10.1007\/s11192-017-2453-4","volume":"112","author":"B Alvarez-Bornstein","year":"2017","unstructured":"Alvarez-Bornstein, B., Morillo, F., & Bordons, M. (2017). Funding acknowledgments in the web of science: Completeness and accuracy of collected data. Scientometrics,112, 1793\u20131812. https:\/\/doi.org\/10.1007\/s11192-017-2453-4.","journal-title":"Scientometrics"},{"issue":"1","key":"3318_CR3","doi-asserted-by":"publisher","first-page":"206","DOI":"10.1007\/s11192-009-0023-0","volume":"83","author":"AD Anastasiadis","year":"2010","unstructured":"Anastasiadis, A. D., de Albuquerque, M. P., de Albuquerque, M. P., & Mussi, D. B. (2010). Tsallis q-exponential describes the distribution of scientific citations\u2014A new characterization of the impact. Scientometrics,83(1), 206\u2013218. https:\/\/doi.org\/10.1007\/s11192-009-0023-0.","journal-title":"Scientometrics"},{"key":"3318_CR4","unstructured":"Anon 1997 The development of national clinical guidelines and integrated care pathways and audit of practices against these standards: A collaborative project by all Scottish clinical genetics services. Journal of Medical Genetics 34(S1): 515."},{"issue":"2","key":"3318_CR104","doi-asserted-by":"publisher","first-page":"1199","DOI":"10.1007\/s11192-017-2490-z","volume":"113","author":"V Bakare","year":"2017","unstructured":"Bakare, V., & Lewison, G. (2017). Country over-citation ratios. Scientometrics, 113(2), 1199\u20131207.","journal-title":"Scientometrics"},{"issue":"2","key":"3318_CR5","doi-asserted-by":"publisher","first-page":"65","DOI":"10.5530\/jscires.6.2.12","volume":"6","author":"M Begum","year":"2017","unstructured":"Begum, M., & Lewison, G. (2017). Web of science research funding information: Methodology for its use in analysis and evaluation. Journal of Scientometric Research,6(2), 65\u201373.","journal-title":"Journal of Scientometric Research"},{"issue":"4","key":"3318_CR6","doi-asserted-by":"publisher","first-page":"e0154197","DOI":"10.1371\/journal.pone.0154197","volume":"11","author":"M Begum","year":"2016","unstructured":"Begum, M., Lewison, G., Wright, J. S. F., Pallari, E., & Sullivan, R. (2016). European non-communicable respiratory disease research, 2002\u201313: Bibliometric study of outputs and funding. PLoS ONE,11(4), e0154197. https:\/\/doi.org\/10.1371\/journal.pone.0154197.","journal-title":"PLoS ONE"},{"issue":"6","key":"3318_CR7","doi-asserted-by":"publisher","first-page":"370","DOI":"10.1016\/j.acalib.2012.08.002","volume":"38","author":"EML Bergman","year":"2012","unstructured":"Bergman, E. M. L. (2012). Finding citations to social work literature: The relative benefits of using web of science, scopus or google scholar. The Journal of Academic Librarianship,38(6), 370\u2013379. https:\/\/doi.org\/10.1016\/j.acalib.2012.08.002.","journal-title":"The Journal of Academic Librarianship"},{"issue":"10","key":"3318_CR8","doi-asserted-by":"publisher","first-page":"514","DOI":"10.1016\/S1070-3241(16)30465-5","volume":"25","author":"FA Cluzeau","year":"1999","unstructured":"Cluzeau, F. A., & Littlejohns, P. (1999). Apprising clinical practice guidelines in England and Wales: The development of a methodologic framework and its application to policy. The Joint Commission Journal Quality Improvement,25(10), 514\u2013521.","journal-title":"The Joint Commission Journal Quality Improvement"},{"key":"3318_CR106","unstructured":"Dawson, G., Lucocq, B., Cottrell, R., & Lewison, G. (1998). Mapping the landscape: National Biomedical Research Outputs 1988\u201395 London: The Wellcome Trust. Policy Report no. 9. ISBN: 1869835956."},{"issue":"3159","key":"3318_CR9","doi-asserted-by":"publisher","first-page":"108","DOI":"10.1126\/science.122.3159.108","volume":"122","author":"E Garfield","year":"1955","unstructured":"Garfield, E. (1955). Citation indexes for science\u2014New dimension in documentation through association of ideas. Science,122(3159), 108\u2013111. https:\/\/doi.org\/10.1126\/science.122.3159.108.","journal-title":"Science"},{"issue":"4060","key":"3318_CR10","doi-asserted-by":"publisher","first-page":"471","DOI":"10.1126\/science.178.4060.471","volume":"178","author":"E Garfield","year":"1972","unstructured":"Garfield, E. (1972). Citation analysis as a tool in journal evaluation\u2014Journals can be ranked by frequency and impact of citations for science policy studies. Science,178(4060), 471\u2013479. https:\/\/doi.org\/10.1126\/science.178.4060.471.","journal-title":"Science"},{"issue":"1","key":"3318_CR11","doi-asserted-by":"publisher","first-page":"33","DOI":"10.3152\/147154499781777658","volume":"8","author":"J Grant","year":"1999","unstructured":"Grant, J. (1999). Evaluating the outcomes of biomedical research on healthcare. Research Evaluation,8(1), 33\u201338.","journal-title":"Research Evaluation"},{"issue":"7242","key":"3318_CR12","doi-asserted-by":"publisher","first-page":"1107","DOI":"10.1136\/bmj.320.7242.1107","volume":"320","author":"J Grant","year":"2000","unstructured":"Grant, J., Cottrell, R., Cluzeau, F., & Fawcett, G. (2000). Evaluating, \u201cpayback\u201d on biomedical research from papers cited in clinical guidelines: Applied bibliometric study. BMJ,320(7242), 1107\u20131111. https:\/\/doi.org\/10.1136\/bmj.320.7242.1107.","journal-title":"BMJ"},{"key":"3318_CR101","unstructured":"Ingwersen, P., Larsen, B., & Wormell, I. (2000). Applying diachronic citation analysis to research program evaluations. In Web of Knowledge - A Festschrift in Honor of. Eugene Garfield : ASIST Monograph Series (pp. 373\u2013387)."},{"key":"3318_CR13","unstructured":"Jeschin, D., Lewison, G., & Anderson, J. (1995). A bibliometric database for tracking acknowledgements of research funding. In M. E. D. Koenig & A. Bookstein (Eds.), proceedings of the fifth biennial conference of the international society for scientometrics and informetrics. Learned Information Inc., Medford NJ, USA. ISBN: 1-57387-010-2."},{"issue":"1","key":"3318_CR100","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1007\/BF02458392","volume":"43","author":"RN Kostoff","year":"1998","unstructured":"Kostoff, R. N. (1998). The use and misuse of citation analysis in research evaluation. Scientometrics, 43(1), 27\u201343.","journal-title":"Scientometrics"},{"key":"3318_CR14","unstructured":"Lewison, G. (2007). The references on UK cancer clinical guidelines.In D. Torres-Salinas, & H. F. Moed (Eds.), proceedings of the 11th biennial conference of the international society for scientometrics and informetrics (pp. 489\u2013498). Madrid, Spain."},{"issue":"1\u20132","key":"3318_CR15","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1007\/BF02457963","volume":"41","author":"G Lewison","year":"1998","unstructured":"Lewison, G., & Dawson, G. (1998). The effect of funding on the outputs of biomedical research. Scientometrics,41(1\u20132), 17\u201327.","journal-title":"Scientometrics"},{"issue":"2","key":"3318_CR105","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1023\/b:scie.0000027677.79173.b8","volume":"60","author":"G Lewison","year":"2004","unstructured":"Lewison, G., & Paraje, G. (2004). The classification of biomedical journals by research level. Scientometrics, 60(2), 145\u2013157. https:\/\/doi.org\/10.1023\/b:scie.0000027677.79173.b8.","journal-title":"Scientometrics"},{"issue":"12","key":"3318_CR16","doi-asserted-by":"publisher","first-page":"1944","DOI":"10.1038\/sj.bjc.6604405","volume":"98","author":"G Lewison","year":"2008","unstructured":"Lewison, G., & Sullivan, R. (2008). The impact of cancer research: How publications influence UK cancer clinical guidelines. British Journal of Cancer,98(12), 1944\u20131950. https:\/\/doi.org\/10.1038\/sj.bjc.6604405.","journal-title":"British Journal of Cancer"},{"key":"3318_CR17","doi-asserted-by":"publisher","first-page":"2151","DOI":"10.1007\/s11192-014-1507-0","volume":"102","author":"G Lewison","year":"2015","unstructured":"Lewison, G., & Sullivan, R. (2015). Conflicts of interest statements on biomedical papers. Scientometrics,102, 2151\u20132159. https:\/\/doi.org\/10.1007\/s11192-014-1507-0.","journal-title":"Scientometrics"},{"key":"3318_CR18","unstructured":"Lewison, G., & Wilcox-Jay, K. (2003). Getting biomedical research into practice: the citations from UK clinical guidelines. In G. Jiang, R. Rousseau, & Y. Wu(Eds.), proceedings of the ninth biennial conference of the international society for scientometrics and informetrics (pp. 152\u2013160). Beijing, China."},{"issue":"2","key":"3318_CR102","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1002\/asi.20743","volume":"59","author":"L Leydesdorff","year":"2008","unstructured":"Leydesdorff, L. (2008). Caveats for the use of citation indicators in research and journal evaluations. Journal of the American Society for Information Science and Technology, 59(2), 278\u2013287.","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"9","key":"3318_CR19","doi-asserted-by":"publisher","first-page":"2194","DOI":"10.1002\/asi.23372","volume":"67","author":"L Leydesdorff","year":"2016","unstructured":"Leydesdorff, L., de Moya-Anegon, F., & de Nooy, W. (2016). Aggregated journal\u2013journal citation relations in scopus and web of science matched and compared in terms of networks, maps, and interactive overlays. Journal of the Association for Information Science and Technology,67(9), 2194\u20132211. https:\/\/doi.org\/10.1002\/asi.23372.","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"11","key":"3318_CR20","first-page":"9","volume":"27","author":"M Manafy","year":"2004","unstructured":"Manafy, M. (2004). Scopus: Elsevier expands the scope of research. Econtent\u2013Digital Content Strategies and Resources,27(11), 9\u201311.","journal-title":"Econtent\u2013Digital Content Strategies and Resources"},{"issue":"4","key":"3318_CR21","doi-asserted-by":"publisher","first-page":"1160","DOI":"10.1016\/j.joi.2018.09.002","volume":"12","author":"A Martin-Martin","year":"2018","unstructured":"Martin-Martin, A., Orduna-Malea, E., Thelwall, M., & Lopez-Cozar, E. D. (2018). Google scholar, web of science, and scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics,12(4), 1160\u20131177. https:\/\/doi.org\/10.1016\/j.joi.2018.09.002.","journal-title":"Journal of Informetrics"},{"issue":"12","key":"3318_CR22","doi-asserted-by":"publisher","first-page":"2499","DOI":"10.1002\/asi.21165","volume":"60","author":"LI Meho","year":"2009","unstructured":"Meho, L. I., & Sugimoto, C. R. (2009). Assessing the scholarly impact of information studies: A tale of two citation databases\u2014Scopus and web of science. Journal of the American Society Information Science and Technology,60(12), 2499\u20132508. https:\/\/doi.org\/10.1002\/asi.21165.","journal-title":"Journal of the American Society Information Science and Technology"},{"issue":"13","key":"3318_CR23","doi-asserted-by":"publisher","first-page":"2105","DOI":"10.1002\/asi.20677","volume":"58","author":"LI Meho","year":"2007","unstructured":"Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar. Journal of the American Society Information Science Technology,58(13), 2105\u20132125. https:\/\/doi.org\/10.1002\/asi.20677.","journal-title":"Journal of the American Society Information Science Technology"},{"issue":"1","key":"3318_CR24","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1177\/14746514020020010401","volume":"2","author":"J Miller","year":"2002","unstructured":"Miller, J. (2002). The Scottish intercollegiate guidelines network (SIGN). The British Journal Diabetes Vascular Disease,2(1), 47\u201349.","journal-title":"The British Journal Diabetes Vascular Disease"},{"issue":"1","key":"3318_CR103","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1007\/s00005-009-0001-5","volume":"57","author":"HF Moed","year":"2009","unstructured":"Moed, H. F. (2009). New developments in the use of citation analysis in research evaluation. Archivum Immunologiae et Therapiae Experimentalis, 57(1), 13\u201318.","journal-title":"Archivum Immunologiae et Therapiae Experimentalis"},{"issue":"1","key":"3318_CR27","doi-asserted-by":"publisher","first-page":"UNSP e000258","DOI":"10.1136\/esmoopen-2017-000258","volume":"3","author":"E Pallari","year":"2018","unstructured":"Pallari, E., Fox, A. W., & Lewison, G. (2018b). Differential research impact in cancer practice guidelines evidence base: Lessons from ESMO NICE and SIGN. ESMO OPEN,3(1), UNSP e000258. https:\/\/doi.org\/10.1136\/esmoopen-2017-000258.","journal-title":"ESMO OPEN"},{"key":"3318_CR25","first-page":"583","volume-title":"Springer handbook of science and technology indicators","author":"E Pallari","year":"2019","unstructured":"Pallari, E., & Lewison, G. (2019). 22 How Biomedical Research Can Inform Both Clinicians and the General Public. In W. Gl\u00e4nzel, et al. (Eds.), Springer handbook of science and technology indicators (pp. 583\u2013609). Cham: Springer."},{"issue":"3","key":"3318_CR26","doi-asserted-by":"publisher","first-page":"270","DOI":"10.1093\/reseval\/rvy006","volume":"27","author":"R Pallari","year":"2018","unstructured":"Pallari, R., Lewison, G., Ciani, O., Tarricone, R., Sommariva, S., Begum, M., et al. (2018a). The impacts of diabetes research from 31 European Countries in 2002 to 2013. Research Evaluation,27(3), 270\u2013282. https:\/\/doi.org\/10.1093\/reseval\/rvy006.","journal-title":"Research Evaluation"},{"key":"3318_CR28","unstructured":"Roe, P. E., Wentworth, A., Sullivan, R., & Lewison, G.(2010). The anatomy of citations to UK cancer research papers. In proceedings of the 11th conference on S&T indicators (pp. 225\u2013226). Leiden."},{"issue":"1","key":"3318_CR29","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1007\/s11192-013-0954-3","volume":"96","author":"J Ruiz-Castillo","year":"2013","unstructured":"Ruiz-Castillo, J. (2013). The role of statistics in establishing the similarity of citation distributions in a static and a dynamic context. Scientometrics,96(1), 173\u2013181. https:\/\/doi.org\/10.1007\/s11192-013-0954-3.","journal-title":"Scientometrics"},{"key":"3318_CR30","unstructured":"Sirtes, D. (2013). Funding acknowledgements for the German Research Foundation (DFG). The dirty data of the web of science database and how to clean it up. In J. Gerroiz, E. Schiebel, C. Gumpenberger, M. Horlesberger, H. Moed (Eds.), proceedings of the 14th biennial conference of the international society for scientometrics and informetrics, (pp. 784\u2013795). Vienna."},{"key":"3318_CR31","doi-asserted-by":"publisher","first-page":"9","DOI":"10.3389\/frma.2017.00009","volume":"2","author":"A Tattersall","year":"2018","unstructured":"Tattersall, A., & Carroll, C. (2018). What can altmetric.com tell us about policy citations of research an analysis of altmetric.com data for research articles from the university of sheffield. Frontiers in Research Metrics Analytics,2, 9.","journal-title":"Frontiers in Research Metrics Analytics"},{"issue":"4","key":"3318_CR32","doi-asserted-by":"publisher","first-page":"e62395","DOI":"10.1371\/journal.pone.0062395","volume":"8","author":"NJ van Eck","year":"2013","unstructured":"van Eck, N. J., Waltman, L., van Raan, A. F. J., Klautz, R. J. M., & Peul, W. C. (2013). Citation analysis may severely underestimate the impact of clinical research as compared to basic research. PLoS ONE,8(4), e62395.","journal-title":"PLoS ONE"},{"issue":"1","key":"3318_CR33","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1023\/A:1010501820393","volume":"51","author":"AFJ van Raan","year":"2001","unstructured":"van Raan, A. F. J. (2001). Competition amongst scientists for publication status: Toward a model of scientific publication and citation distributions. Scientometrics,51(1), 347\u2013357. https:\/\/doi.org\/10.1023\/A:1010501820393.","journal-title":"Scientometrics"},{"issue":"7439","key":"3318_CR34","doi-asserted-by":"publisher","first-page":"536","DOI":"10.1136\/bmj.328.7439.536","volume":"328","author":"A Wailoo","year":"2004","unstructured":"Wailoo, A., Roberts, J., & Brazier, J. (2004). Efficiency, equity, and NICE clinical guidelines\u2014Clinical guidelines need a broader view than just the clinical. BMJ,328(7439), 536\u2013537. https:\/\/doi.org\/10.1136\/bmj.328.7439.536.","journal-title":"BMJ"}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-019-03318-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s11192-019-03318-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-019-03318-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,13]],"date-time":"2020-12-13T00:35:26Z","timestamp":1607819726000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s11192-019-03318-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,14]]},"references-count":41,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,2]]}},"alternative-id":["3318"],"URL":"https:\/\/doi.org\/10.1007\/s11192-019-03318-2","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"type":"print","value":"0138-9130"},{"type":"electronic","value":"1588-2861"}],"subject":[],"published":{"date-parts":[[2019,12,14]]},"assertion":[{"value":"29 August 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 December 2019","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"This study has no external funding. Magnus Eriksson, Annika Billhult and Tommy Billhult are shareholders of Minso Solutions AB which provides the clinical impact<sup>\u00ae<\/sup> database.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}