{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,20]],"date-time":"2026-07-20T23:00:46Z","timestamp":1784588446769,"version":"3.55.0"},"reference-count":59,"publisher":"Wiley","issue":"10","license":[{"start":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T00:00:00Z","timestamp":1747267200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000269","name":"Economic and Social Research Council","doi-asserted-by":"publisher","award":["APP43146"],"award-info":[{"award-number":["APP43146"]}],"id":[{"id":"10.13039\/501100000269","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["asistdl.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Asso for Info Science &amp;amp; Tech"],"published-print":{"date-parts":[[2025,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Academics and departments are sometimes judged by how their research has benefited society. For example, the UK's Research Excellence Framework (REF) assesses Impact Case Studies (ICSs), which are five\u2010page evidence\u2010based claims of societal impacts. This article investigates whether ChatGPT can evaluate societal impact claims and therefore potentially support expert human assessors. For this, various parts of 6220 public ICSs from REF2021 were fed to ChatGPT 4o\u2010mini along with the REF2021 evaluation guidelines, comparing ChatGPT's predictions with published departmental average ICS scores. The results suggest that the optimal strategy for high correlations with expert scores is to input the title and summary of an ICS but not the remaining text and to modify the original REF guidelines to encourage a stricter evaluation. The scores generated by this approach correlated positively with departmental average scores in all 34 Units of Assessment (UoAs), with values between 0.18 (Economics and Econometrics) and 0.56 (Psychology, Psychiatry and Neuroscience). At the departmental level, the corresponding correlations were higher, reaching 0.71 for Sport and Exercise Sciences, Leisure and Tourism. Thus, ChatGPT\u2010based ICS evaluations are simple and viable to support or cross\u2010check expert judgments, although their value varies substantially between fields.<\/jats:p>","DOI":"10.1002\/asi.25021","type":"journal-article","created":{"date-parts":[[2025,5,15]],"date-time":"2025-05-15T07:03:13Z","timestamp":1747292593000},"page":"1357-1373","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Assessing the societal influence of academic research with <scp>ChatGPT<\/scp>: Impact case study evaluations"],"prefix":"10.1002","volume":"76","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4827-971X","authenticated-orcid":false,"given":"Kayvan","family":"Kousha","sequence":"first","affiliation":[{"name":"Wolverhampton Business School University of Wolverhampton  Wolverhampton UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6065-205X","authenticated-orcid":false,"given":"Mike","family":"Thelwall","sequence":"additional","affiliation":[{"name":"Information School University of Sheffield  Sheffield UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2025,5,15]]},"reference":[{"key":"e_1_2_12_2_1","unstructured":"Adams J. Loach T. &Szomszor M.(2015).The diversity of UK research and knowledge. Analyses from the REF impact case studies. Digital Research Reports. Retrieved fromhttps:\/\/www.digital-science.com\/resource\/diversity-of-uk-research\/"},{"key":"e_1_2_12_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-020-03803-z"},{"key":"e_1_2_12_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.joi.2019.01.008"},{"key":"e_1_2_12_5_1","doi-asserted-by":"publisher","DOI":"10.1080\/10632921.2017.1386148"},{"key":"e_1_2_12_6_1","doi-asserted-by":"publisher","DOI":"10.1080\/03054985.2017.1316252"},{"key":"e_1_2_12_7_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0156978"},{"key":"e_1_2_12_8_1","volume-title":"Bibliometrics and citation analysis: From the science citation index to cybermetrics","author":"De Bellis N.","year":"2009"},{"key":"e_1_2_12_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-63627-6"},{"key":"e_1_2_12_10_1","volume-title":"Publication patterns in research underpinning impact in REF2014: A report to HEFCE by digital science","author":"Digital Science","year":"2016"},{"key":"e_1_2_12_11_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.11192"},{"key":"e_1_2_12_12_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-015-0467-4"},{"key":"e_1_2_12_13_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13063-020-04425-9"},{"key":"e_1_2_12_14_1","doi-asserted-by":"publisher","DOI":"10.1080\/03075079.2017.1393059"},{"key":"e_1_2_12_15_1","doi-asserted-by":"publisher","DOI":"10.3167\/aia.2015.220205"},{"key":"e_1_2_12_16_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0264914"},{"key":"e_1_2_12_17_1","doi-asserted-by":"publisher","DOI":"10.25304\/rlt.v28.2306"},{"key":"e_1_2_12_18_1","unstructured":"Jordan K. &Carrigan M.(2018).How was social media cited in 2014 REF impact case studies? Impact of Social Sciences Blog. Retrieved fromhttps:\/\/blogs.lse.ac.uk\/impactofsocialsciences\/2018\/06\/06\/how\u2010was\u2010social\u2010media\u2010cited\u2010in\u20102014\u2010ref\u2010impact\u2010case\u2010studies\/"},{"key":"e_1_2_12_19_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-8551.12186"},{"key":"e_1_2_12_20_1","volume-title":"The nature, scale and beneficiaries of research impact: An initial analysis of Research Excellence Framework (REF) 2014 impact case studies","author":"King's College London and Digital Science","year":"2015"},{"key":"e_1_2_12_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-024-05180-3"},{"key":"e_1_2_12_22_1","doi-asserted-by":"publisher","DOI":"10.1162\/qss_a_00145"},{"key":"e_1_2_12_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3379310.3379312"},{"key":"e_1_2_12_24_1","doi-asserted-by":"publisher","DOI":"10.1177\/1478210317742214"},{"key":"e_1_2_12_25_1","doi-asserted-by":"publisher","DOI":"10.1080\/19312458.2024.2378690"},{"key":"e_1_2_12_26_1","doi-asserted-by":"publisher","DOI":"10.1093\/reseval\/rvw024"},{"key":"e_1_2_12_27_1","doi-asserted-by":"publisher","DOI":"10.1111\/1477-9552.12242"},{"key":"e_1_2_12_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10961-017-9608-6"},{"key":"e_1_2_12_29_1","doi-asserted-by":"publisher","DOI":"10.1177\/1356389017730726"},{"key":"e_1_2_12_30_1","doi-asserted-by":"publisher","DOI":"10.7249\/RR2463"},{"key":"e_1_2_12_31_1","doi-asserted-by":"publisher","DOI":"10.1002\/berj.3572"},{"key":"e_1_2_12_32_1","unstructured":"Pollitt A. Sreenan N. Grant J. Szomszor M. Leeworthy D. &Hughes D.(2023).The impacts of research from Welsh universities: Full report. Retrieved fromhttps:\/\/www.learnedsociety.wales\/wp-content\/uploads\/2023\/10\/The-impacts-of-research-from-Welsh-universities-Final.pdf"},{"key":"e_1_2_12_33_1","doi-asserted-by":"publisher","DOI":"10.1177\/1478210315610992"},{"key":"e_1_2_12_34_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0173152"},{"key":"e_1_2_12_35_1","doi-asserted-by":"publisher","DOI":"10.12688\/f1000research.74374.2"},{"key":"e_1_2_12_36_1","unstructured":"REF. (2021a).Guidance on submissions to REF 2021. Retrieved fromhttps:\/\/2021.ref.ac.uk\/media\/1447\/ref-2019_01-guidance-on-submissions.pdf"},{"key":"e_1_2_12_37_1","unstructured":"REF. (2021b).Panel criteria and working methods on REF 2021. Retrieved fromhttps:\/\/2021.ref.ac.uk\/media\/1450\/ref-2019_02-panel-criteria-and-working-methods.pdf"},{"key":"e_1_2_12_38_1","unstructured":"REF. (2021c).Impact case study database: Securing the Legacy of the Late Spanish Filmmaker Bigas Luna. Retrieved fromhttps:\/\/results2021.ref.ac.uk\/impact\/3c81f543\u20104e15\u20104a90\u20109fae\u2010db49d6769336?page=1"},{"key":"e_1_2_12_39_1","doi-asserted-by":"publisher","DOI":"10.1057\/s41599-020-0394-7"},{"key":"e_1_2_12_40_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12955-019-1220-z"},{"key":"e_1_2_12_41_1","doi-asserted-by":"publisher","DOI":"10.1002\/jid.3255"},{"key":"e_1_2_12_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.dsx.2024.102946"},{"key":"e_1_2_12_43_1","doi-asserted-by":"publisher","DOI":"10.1093\/reseval\/rvv007"},{"key":"e_1_2_12_44_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0047279416000283"},{"key":"e_1_2_12_45_1","volume-title":"Data enhancement and analysis of the REF 2021 Impact Case Studies","author":"Stevenson C.","year":"2023"},{"key":"e_1_2_12_46_1","doi-asserted-by":"publisher","DOI":"10.1162\/qss_a_00204"},{"key":"e_1_2_12_47_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0168533"},{"key":"e_1_2_12_48_1","doi-asserted-by":"publisher","DOI":"10.2478\/jdis-2024-0013"},{"key":"e_1_2_12_49_1","doi-asserted-by":"crossref","unstructured":"Thelwall M.(2025a).Evaluating research quality with large language models: An analysis of ChatGPT's effectiveness with different settings and inputs 10(1) 1\u201325.https:\/\/doi.org\/10.2478\/jdis-2025-0011","DOI":"10.2478\/jdis-2025-0011"},{"key":"e_1_2_12_50_1","doi-asserted-by":"publisher","DOI":"10.2478\/jdis-2025-0014"},{"key":"e_1_2_12_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2025.104123"},{"key":"e_1_2_12_52_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.24751"},{"key":"e_1_2_12_53_1","unstructured":"Thelwall M. &Yaghi A.(2024).In which fields can ChatGPT detect journal article quality? An evaluation of REF2021 results. Retrieved fromhttps:\/\/arxiv.org\/abs\/2409.16695"},{"key":"e_1_2_12_54_1","doi-asserted-by":"publisher","DOI":"10.21825\/digest.86011"},{"key":"e_1_2_12_55_1","doi-asserted-by":"publisher","DOI":"10.1080\/03075079.2018.1455082"},{"key":"e_1_2_12_56_1","doi-asserted-by":"publisher","DOI":"10.1080\/03075079.2017.1339028"},{"key":"e_1_2_12_57_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0288469"},{"key":"e_1_2_12_58_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.24122"},{"key":"e_1_2_12_59_1","doi-asserted-by":"publisher","DOI":"10.2174\/1874434601711010211"},{"key":"e_1_2_12_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-021-04096-6"}],"container-title":["Journal of the Association for Information Science and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/asi.25021","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,15]],"date-time":"2025-09-15T08:20:29Z","timestamp":1757924429000},"score":1,"resource":{"primary":{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/10.1002\/asi.25021"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,15]]},"references-count":59,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2025,10]]}},"alternative-id":["10.1002\/asi.25021"],"URL":"https:\/\/doi.org\/10.1002\/asi.25021","archive":["Portico"],"relation":{},"ISSN":["2330-1635","2330-1643"],"issn-type":[{"value":"2330-1635","type":"print"},{"value":"2330-1643","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,15]]},"assertion":[{"value":"2024-10-25","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-05","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-15","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}