{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,10]],"date-time":"2026-10-10T09:44:44Z","timestamp":1791625484147,"version":"4.3.4"},"reference-count":25,"publisher":"MIT Press","issue":"3","license":[{"start":{"date-parts":[[2021,6,30]],"date-time":"2021-06-30T00:00:00Z","timestamp":1625011200000},"content-version":"vor","delay-in-days":180,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,11,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>While funders increasingly request evidence of the societal benefits of research, all academics in the UK must periodically provide this information to gain part of their block funding within the Research Excellence Framework (REF). The impact case studies produced in the UK are public and can therefore be used to gain insights into the types of sources used to justify societal impact claims. This study focuses on the URLs cited as evidence in the last public REF to help researchers and resource providers to understand what types can be used and the disciplinary differences in their uptake. Based on a new semiautomatic method to classify the URLs cited in impact case studies, the results show that there are a few key online types of source for most broad fields, but these sources differ substantially between subject areas. For example, news websites are more important in some fields than others, and YouTube is sometimes used for multimedia evidence in the arts and humanities. Knowledge of the common sources selected independently by thousands of researchers may help others to identify suitable sources for the complex task of evidencing societal impacts.<\/jats:p>","DOI":"10.1162\/qss_a_00145","type":"journal-article","created":{"date-parts":[[2021,6,30]],"date-time":"2021-06-30T14:41:54Z","timestamp":1625064114000},"page":"864-881","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":11,"title":["Which types of online evidence show the nonacademic benefits of research? Websites cited in UK impact case studies"],"prefix":"10.1162","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4827-971X","authenticated-orcid":true,"given":"Kayvan","family":"Kousha","sequence":"first","affiliation":[{"name":"Statistical Cybermetrics Research Group, University of Wolverhampton, Wolverhampton, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6065-205X","authenticated-orcid":true,"given":"Mike","family":"Thelwall","sequence":"additional","affiliation":[{"name":"Statistical Cybermetrics Research Group, University of Wolverhampton, Wolverhampton, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahshid","family":"Abdoli","sequence":"additional","affiliation":[{"name":"Statistical Cybermetrics Research Group, University of Wolverhampton, Wolverhampton, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,11,5]]},"reference":[{"issue":"1","key":"2021110817023867100_bib1","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1016\/j.joi.2019.01.008","article-title":"Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK Research Excellence Framework (REF)","volume":"13","author":"Bornmann","year":"2019","journal-title":"Journal of Informetrics"},{"issue":"1","key":"2021110817023867100_bib2","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1080\/10632921.2017.1386148","article-title":"Evidencing impact from art research: Analysis of impact case studies from the REF 2014","volume":"48","author":"Brook","year":"2018","journal-title":"Journal of Arts Management, Law, and Society"},{"key":"2021110817023867100_bib3","volume-title":"Publication patterns in research underpinning impact in REF2014: A report to HEFCE by Digital Science","author":"Digital Science","year":"2016"},{"issue":"11","key":"2021110817023867100_bib4","doi-asserted-by":"publisher","first-page":"e1002003","DOI":"10.1371\/journal.pbio.1002003","article-title":"Alternative perspectives on impact: The potential of ALMs and altmetrics to inform funders about research impact","volume":"12","author":"Dinsmore","year":"2014","journal-title":"PLOS Biology"},{"issue":"1","key":"2021110817023867100_bib5","doi-asserted-by":"publisher","first-page":"232","DOI":"10.1186\/s12916-015-0467-4","article-title":"Research impact in the community-based health sciences: An analysis of 162 case studies from the 2014 UK Research Excellence Framework","volume":"13","author":"Greenhalgh","year":"2015","journal-title":"BMC Medicine"},{"issue":"4","key":"2021110817023867100_bib6","doi-asserted-by":"publisher","first-page":"14","DOI":"10.7249\/RR1606","article-title":"100 metrics to assess and communicate the value of biomedical research: An ideas book","volume":"6","author":"Guthrie","year":"2017","journal-title":"Rand Health Quarterly"},{"key":"2021110817023867100_bib7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13063-020-04425-9","article-title":"Evidencing the impact of cancer trials: Insights from the 2014 UK Research Excellence Framework","volume":"21","author":"Hanna","year":"2020","journal-title":"Trials"},{"issue":"4","key":"2021110817023867100_bib8","doi-asserted-by":"publisher","first-page":"628","DOI":"10.1080\/03075079.2017.1393059","article-title":"Achieving wider impact in business and management: Analysing the case studies from REF 2014","volume":"44","author":"Hughes","year":"2019","journal-title":"Studies in Higher Education"},{"key":"2021110817023867100_bib9","article-title":"How was social media cited in 2014 REF Impact Case Studies?","author":"Jordan","year":"2018","journal-title":"Impact of Social Sciences blog"},{"key":"2021110817023867100_bib10","volume-title":"The nature, scale and beneficiaries of research impact: An initial analysis of Research Excellence Framework (REF) 2014 impact case studies","author":"King\u2019s College London and Digital Science","year":"2015"},{"key":"2021110817023867100_bib11","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-02511-3_31","article-title":"Web citation indicators for wider impact assessment of articles","volume-title":"Springer handbook of science and technology indicators","author":"Kousha","year":"2019"},{"issue":"1","key":"2021110817023867100_bib12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1472-6963-6-134","article-title":"Describing the impact of health research: A Research Impact Framework","volume":"6","author":"Kuruvilla","year":"2006","journal-title":"BMC Health Services Research"},{"issue":"3","key":"2021110817023867100_bib13","doi-asserted-by":"publisher","first-page":"369","DOI":"10.1177\/0165551516636291","article-title":"Beyond REF 2014: The impact of impact assessment on the future of information research","volume":"42","author":"Marcella","year":"2016","journal-title":"Journal of Information Science"},{"issue":"4","key":"2021110817023867100_bib14","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1177\/1356389017730726","article-title":"Exploring research impact in the assessment of leadership, governance and management research","volume":"23","author":"Morrow","year":"2017","journal-title":"Evaluation"},{"issue":"1","key":"2021110817023867100_bib15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/1472-6963-14-2","article-title":"A systematic review of barriers to and facilitators of the use of evidence by policymakers","volume":"14","author":"Oliver","year":"2014","journal-title":"BMC Health Services Research"},{"key":"2021110817023867100_bib16","doi-asserted-by":"publisher","DOI":"10.7249\/RR2463","volume-title":"Guidance for standardising quantitative indicators of impact within REF case studies","author":"Parks","year":"2018"},{"issue":"3","key":"2021110817023867100_bib17","doi-asserted-by":"publisher","first-page":"e0173152","DOI":"10.1371\/journal.pone.0173152","article-title":"Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements","volume":"12","author":"Ravenscroft","year":"2017","journal-title":"PLOS ONE"},{"key":"2021110817023867100_bib18","volume-title":"REF 2014: Assessment framework and guidance on submissions","author":"Research Excellence Framework","year":"2014"},{"key":"2021110817023867100_bib19","volume-title":"REF Guidance, 2019. Guidance on submissions to REF 2021","author":"Research Excellence Framework","year":"2019"},{"issue":"12","key":"2021110817023867100_bib20","doi-asserted-by":"publisher","first-page":"e0168533","DOI":"10.1371\/journal.pone.0168533","article-title":"Beyond academia\u2014Interrogating research impact in the research excellence framework","volume":"11","author":"Ter\u00e4m\u00e4","year":"2016","journal-title":"PLOS ONE"},{"issue":"5","key":"2021110817023867100_bib21","doi-asserted-by":"publisher","first-page":"e64841","DOI":"10.1371\/journal.pone.0064841","article-title":"Do altmetrics work? Twitter and ten other social web services","volume":"8","author":"Thelwall","year":"2013","journal-title":"PLOS ONE"},{"issue":"1","key":"2021110817023867100_bib22","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1108\/AJIM-09-2015-0146","article-title":"Alternative metric indicators for funding scheme evaluations","volume":"68","author":"Thelwall","year":"2015","journal-title":"Aslib Journal of Information Management"},{"issue":"1","key":"2021110817023867100_bib23","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1080\/03075079.2017.1339028","article-title":"Evidencing impact: A case study of UK academic perspectives on evidencing research impact","volume":"44","author":"Wilkinson","year":"2019","journal-title":"Studies in Higher Education"},{"key":"2021110817023867100_bib24","doi-asserted-by":"publisher","DOI":"10.4135\/9781473978782","volume-title":"The metric tide: Report of the independent review of the role of metrics in research assessment and management","author":"Wilsdon","year":"2015"},{"issue":"3","key":"2021110817023867100_bib25","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1002\/asi.24122","article-title":"Altmetric scores: An early indicator of research impact","volume":"70","author":"Wooldridge","year":"2019","journal-title":"Journal of the Association for Information Science and Technology"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/2\/3\/864\/1970772\/qss_a_00145.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/2\/3\/864\/1970772\/qss_a_00145.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,11,8]],"date-time":"2021-11-08T12:03:09Z","timestamp":1636372989000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/2\/3\/864\/102770\/Which-types-of-online-evidence-show-the"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":25,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,11,5]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00145","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00145\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00145\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00145\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00145\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00145\/v2\/decision1","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}