{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T23:03:07Z","timestamp":1771023787639,"version":"3.50.1"},"reference-count":54,"publisher":"Emerald","issue":"5","license":[{"start":{"date-parts":[[2023,2,7]],"date-time":"2023-02-07T00:00:00Z","timestamp":1675728000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JD"],"published-print":{"date-parts":[[2023,8,25]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-subheading\">Purpose<\/jats:title><jats:p>Scholars often aim to conduct high quality research and their success is judged primarily by peer reviewers. Research quality is difficult for either group to identify, however and misunderstandings can reduce the efficiency of the scientific enterprise. In response, we use a novel term association strategy to seek quantitative evidence of aspects of research that are associated with high or low quality.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Design\/methodology\/approach<\/jats:title><jats:p>We extracted the words and 2\u20135-word phrases most strongly associated with different quality scores in each of 34\u00a0Units of Assessment (UoAs) in the Research Excellence Framework (REF) 2021. We extracted the terms from 122,331 journal articles 2014\u20132020 with individual REF2021 quality scores.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Findings<\/jats:title><jats:p>The terms associating with high- or low-quality scores vary between fields but relate to writing styles, methods and topics. We show that the first-person writing style strongly associates with higher quality research in many areas because it is the norm for a set of large prestigious journals. We found methods and topics that associate with both high- and low-quality scores. Worryingly, terms associated with educational and qualitative research attract lower quality scores in multiple areas. REF experts may rarely give high scores to qualitative or educational research because the authors tend to be less competent, because it is harder to do world leading research with these themes, or because they do not value them.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Originality\/value<\/jats:title><jats:p>This is the first investigation of journal article terms associating with research quality.<\/jats:p><\/jats:sec>","DOI":"10.1108\/jd-12-2022-0261","type":"journal-article","created":{"date-parts":[[2023,2,4]],"date-time":"2023-02-04T04:20:08Z","timestamp":1675484408000},"page":"1110-1123","source":"Crossref","is-referenced-by-count":6,"title":["Terms in journal articles associating with high quality: can\u00a0qualitative research be\u00a0world-leading?"],"prefix":"10.1108","volume":"79","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6065-205X","authenticated-orcid":false,"given":"Mike","family":"Thelwall","sequence":"first","affiliation":[]},{"given":"Kayvan","family":"Kousha","sequence":"additional","affiliation":[]},{"given":"Mahshid","family":"Abdoli","sequence":"additional","affiliation":[]},{"given":"Emma","family":"Stuart","sequence":"additional","affiliation":[]},{"given":"Meiko","family":"Makita","sequence":"additional","affiliation":[]},{"given":"Paul","family":"Wilson","sequence":"additional","affiliation":[]},{"given":"Jonathan M.","family":"Levitt","sequence":"additional","affiliation":[]}],"member":"140","published-online":{"date-parts":[[2023,2,7]]},"reference":[{"issue":"1","key":"key2023082411561161000_ref001","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/jee.20031","article-title":"Confusions and conventions: qualitative research in engineering education","volume":"103","year":"2014","journal-title":"Journal of Engineering Education"},{"issue":"2","key":"key2023082411561161000_ref002","doi-asserted-by":"crossref","first-page":"233","DOI":"10.5465\/amj.2011.60262792","article-title":"The coming of age for qualitative research: embracing the diversity of qualitative methods","volume":"54","year":"2011","journal-title":"Academy of Management Journal"},{"issue":"8","key":"key2023082411561161000_ref003","doi-asserted-by":"crossref","first-page":"3157","DOI":"10.1073\/pnas.1014871108","article-title":"Understanding current causes of women's underrepresentation in science","volume":"108","year":"2011","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"9","key":"key2023082411561161000_ref004","doi-asserted-by":"crossref","first-page":"1625","DOI":"10.1080\/03075079.2016.1276549","article-title":"The Cinderella of academia: is higher education pedagogic research undervalued in UK research assessment?","volume":"43","year":"2018","journal-title":"Studies in Higher Education"},{"issue":"8746","key":"key2023082411561161000_ref005","doi-asserted-by":"crossref","first-page":"867","DOI":"10.1016\/0140-6736(91)90201-Y","article-title":"Publication bias in clinical research","volume":"337","year":"1991","journal-title":"The Lancet"},{"issue":"4","key":"key2023082411561161000_ref006","doi-asserted-by":"crossref","first-page":"45","DOI":"10.2478\/jdis-2018-0020","article-title":"The Flemish performance-based research funding system: a unique variant of the Norwegian model","volume":"3","year":"2018","journal-title":"Journal of Data and Information Science"},{"issue":"2","key":"key2023082411561161000_ref007","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1002\/leap.1417","article-title":"Questionnaires mentioned in academic research 1996-2019: rapid increase but declining citation impact","volume":"35","year":"2022","journal-title":"Learned Publishing"},{"issue":"1","key":"key2023082411561161000_ref008","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1016\/j.jclinepi.2011.05.001","article-title":"Panel discussion does not improve reliability of peer review for medical research grant proposals","volume":"65","year":"2012","journal-title":"Journal of Clinical Epidemiology"},{"issue":"6","key":"key2023082411561161000_ref009","doi-asserted-by":"crossref","first-page":"3599","DOI":"10.1002\/ece3.4993","article-title":"Gender differences in peer review outcomes and manuscript impact at six journals of ecology and evolution","volume":"9","year":"2019","journal-title":"Ecology and Evolution"},{"issue":"10","key":"key2023082411561161000_ref010","article-title":"The influence of peer reviewer expertise on the evaluation of research funding applications","volume":"11","year":"2016","journal-title":"PloS One"},{"issue":"4","key":"key2023082411561161000_ref011","doi-asserted-by":"crossref","first-page":"670","DOI":"10.1016\/j.mayocp.2018.09.004","article-title":"Peer review bias: a critical review","volume":"94","year":"2019","journal-title":"Mayo Clinic Proceedings"},{"issue":"11","key":"key2023082411561161000_ref012","doi-asserted-by":"crossref","first-page":"1997","DOI":"10.1377\/hlthaff.2017.0773","article-title":"Explicit bias toward high-income-country research: a randomized, blinded, crossover experiment of English clinicians","volume":"36","year":"2017","journal-title":"Health Affairs"},{"issue":"3","key":"key2023082411561161000_ref013","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1007\/s12144-010-9087-5","article-title":"Biased evaluation of abstracts depending on topic and conclusion: further evidence of a confirmation bias within scientific psychology","volume":"29","year":"2010","journal-title":"Current Psychology"},{"issue":"1","key":"key2023082411561161000_ref014","article-title":"Identification of highly-cited papers using topic-model-based and bibliometric features: the consideration of keyword popularity","volume":"14","year":"2020","journal-title":"Journal of Informetrics"},{"issue":"6","key":"key2023082411561161000_ref015","doi-asserted-by":"crossref","first-page":"897","DOI":"10.1177\/0306312712458478","article-title":"Communicating and compromising on disciplinary expertise in the peer review of research proposals","volume":"42","year":"2012","journal-title":"Social Studies of Science"},{"issue":"7","key":"key2023082411561161000_ref016","article-title":"The validity of peer review in a general medicine journal","volume":"6","year":"2011","journal-title":"PloS One"},{"issue":"3-4","key":"key2023082411561161000_ref017","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1016\/j.lisr.2018.09.002","article-title":"Does research using qualitative methods (grounded theory, ethnography and phenomenology) have more impact?","volume":"40","year":"2018","journal-title":"Library and Information Science Research"},{"issue":"2","key":"key2023082411561161000_ref018","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1007\/s11135-017-0652-y","article-title":"Is the gap widening among universities? On research output inequality and its measurement in the Korean higher education system","volume":"52","year":"2018","journal-title":"Quality and Quantity"},{"issue":"1","key":"key2023082411561161000_ref019","first-page":"1","article-title":"Hierarchies of evidence applied to lifestyle Medicine (HEALM): introduction of a strength-of-evidence approach based on a methodological systematic review","volume":"19","year":"2019","journal-title":"BMC Medical Research Methodology"},{"issue":"Supplement_1","key":"key2023082411561161000_ref020","first-page":"i166","article-title":"Trends in biomedical informatics: most cited topics from recent years","volume":"18","year":"2011","journal-title":"Journal of the American Medical Informatics Association"},{"issue":"4","key":"key2023082411561161000_ref021","article-title":"Editorial peer reviewers' recommendations at a general medical journal: are they reliable and do editors care?","volume":"5","year":"2010","journal-title":"PloS One"},{"issue":"1","key":"key2023082411561161000_ref022","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1007\/s11024-019-09385-2","article-title":"Co-existing notions of research quality: a framework to study context-specific understandings of good research","volume":"58","year":"2020","journal-title":"Minerva"},{"issue":"1","key":"key2023082411561161000_ref023","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1002\/asi.22784","article-title":"Bias in peer review","volume":"64","year":"2013","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"3","key":"key2023082411561161000_ref024","first-page":"3225","article-title":"The impact of qualitative methods on article citation: an international business research perspective","volume":"127","year":"2022","journal-title":"Scientometrics"},{"issue":"4","key":"key2023082411561161000_ref025","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1136\/ebmed-2016-110401","article-title":"New evidence pyramid","volume":"21","year":"2016","journal-title":"BMJ Evidence-Based Medicine"},{"issue":"6","key":"key2023082411561161000_ref026","doi-asserted-by":"crossref","first-page":"1033","DOI":"10.1080\/03075079.2015.1075197","article-title":"Gender consequences of a national performance-based funding model: new pieces in an old puzzle","volume":"42","year":"2017","journal-title":"Studies in Higher Education"},{"key":"key2023082411561161000_ref027","doi-asserted-by":"crossref","first-page":"586","DOI":"10.1016\/j.tourman.2015.08.011","article-title":"Tourism research quality: reviewing and assessing interdisciplinarity","volume":"52","year":"2016","journal-title":"Tourism Management"},{"issue":"4","key":"key2023082411561161000_ref028","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1111\/1467-8551.12122","article-title":"Business and management studies in the 2014 research excellence framework","volume":"26","year":"2015","journal-title":"British Journal of Management"},{"issue":"12","key":"key2023082411561161000_ref029","doi-asserted-by":"crossref","first-page":"2952","DOI":"10.1073\/pnas.1714379115","article-title":"Low\u00a0agreement among reviewers evaluating the same NIH grant applications","volume":"115","year":"2018","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"6","key":"key2023082411561161000_ref030","article-title":"Peer review evaluation process of Marie Curie actions under EU's seventh framework programme for research","volume":"10","year":"2015","journal-title":"PLoS One"},{"issue":"11","key":"key2023082411561161000_ref031","doi-asserted-by":"crossref","first-page":"2415","DOI":"10.1016\/j.biocon.2009.06.021","article-title":"Do gender, nationality, or academic age affect review decisions? An analysis of submissions to the journal Biological Conservation","volume":"142","year":"2009","journal-title":"Biological Conservation"},{"issue":"5","key":"key2023082411561161000_ref032","doi-asserted-by":"crossref","first-page":"763","DOI":"10.1016\/j.joca.2016.01.008","article-title":"Multiple p-values and Bonferroni correction","volume":"24","year":"2016","journal-title":"Osteoarthritis and Cartilage"},{"issue":"1","key":"key2023082411561161000_ref033","doi-asserted-by":"crossref","first-page":"929","DOI":"10.1007\/s11192-014-1402-8","article-title":"Mapping the intellectual structure of scientometrics: a co-word analysis of the journal Scientometrics (2005-2010)","volume":"102","year":"2015","journal-title":"Scientometrics"},{"key":"key2023082411561161000_ref034","unstructured":"REF2021 (2020), Guidance, available at: https:\/\/www.ref.ac.uk\/guidance-and-criteria-on-submissions\/guidance\/"},{"issue":"9","key":"key2023082411561161000_ref035","doi-asserted-by":"crossref","first-page":"1964","DOI":"10.1093\/brain\/123.9.1964","article-title":"Reproducibility of peer review in clinical neuroscience: is agreement between reviewers any greater than would be expected by chance alone?","volume":"123","year":"2000","journal-title":"Brain"},{"issue":"1","key":"key2023082411561161000_ref036","first-page":"1","article-title":"Peer review assessment of originality in tourism journals: critical perspective of key gatekeepers","volume":"77","year":"2019","journal-title":"Annals of Tourism Research"},{"issue":"1","key":"key2023082411561161000_ref037","doi-asserted-by":"crossref","first-page":"4","DOI":"10.3390\/urbansci4010004","article-title":"The most frequently cited topics in urban planning scholarship","volume":"4","year":"2020","journal-title":"Urban Science"},{"key":"key2023082411561161000_ref038","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.eist.2021.06.006","article-title":"Main topics in EIST during its first decade: a computational-linguistic analysis","volume":"41","year":"2021","journal-title":"Environmental Innovation and Societal Transitions"},{"issue":"5","key":"key2023082411561161000_ref039","doi-asserted-by":"crossref","first-page":"749","DOI":"10.1002\/asi.23985","article-title":"A critical evaluation of expert survey\u2010based journal rankings: the role of personal research interests","volume":"69","year":"2018","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"4","key":"key2023082411561161000_ref040","first-page":"2","article-title":"The Norwegian model in Norway","volume":"3","year":"2018","journal-title":"Journal of Data and Information Science"},{"issue":"1","key":"key2023082411561161000_ref041","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1080\/2159676X.2017.1393221","article-title":"Generalizability in qualitative research: misunderstandings, opportunities and recommendations for the sport and exercise sciences","volume":"10","year":"2018","journal-title":"Qualitative Research in Sport, Exercise and Health"},{"issue":"1","key":"key2023082411561161000_ref042","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1007\/s11192-016-2161-5","article-title":"The effect of keyword repetition in abstract and keyword frequency per journal in predicting citation counts","volume":"110","year":"2017","journal-title":"Scientometrics"},{"key":"key2023082411561161000_ref043","volume-title":"Word Association Thematic Analysis: A Social Media Text Exploration Strategy","year":"2021"},{"issue":"2","key":"key2023082411561161000_ref044","doi-asserted-by":"publisher","DOI":"10.1016\/j.lisr.2021.101094","article-title":"Is research with qualitative data more prevalent and impactful now? Interviews, case studies, focus groups and ethnographies","volume":"43","year":"2021","journal-title":"Library and Information Science Research"},{"issue":"3","key":"key2023082411561161000_ref045","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1002\/asi.24401","article-title":"Do new research issues attract more citations? A comparison between 25 Scopus subject categories","volume":"72","year":"2021","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"5","key":"key2023082411561161000_ref046","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1002\/asi.23474","article-title":"Does research with statistics have more impact? The citation rank advantage of structural equation modelling","volume":"67","year":"2016","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"6","key":"key2023082411561161000_ref047","doi-asserted-by":"crossref","first-page":"809","DOI":"10.1177\/0165551520938678","article-title":"Does the use of open, non-anonymous peer review in scholarly publishing introduce bias? Evidence from the F1000Research post-publication open peer review publishing model","volume":"47","year":"2021","journal-title":"Journal of Information Science"},{"issue":"48","key":"key2023082411561161000_ref048","doi-asserted-by":"crossref","first-page":"12708","DOI":"10.1073\/pnas.1707323114","article-title":"Reviewer bias in single-versus double-blind peer review","volume":"114","year":"2017","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"8","key":"key2023082411561161000_ref049","article-title":"Role of editorial and peer review processes in publication bias: analysis of drug trials submitted to eight medical journals","volume":"9","year":"2014","journal-title":"PloS One"},{"issue":"3","key":"key2023082411561161000_ref050","doi-asserted-by":"crossref","first-page":"624","DOI":"10.1111\/jep.13404","article-title":"Variation amongst hierarchies of evidence","volume":"27","year":"2021","journal-title":"Journal of Evaluation in Clinical Practice"},{"issue":"3","key":"key2023082411561161000_ref051","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1093\/reseval\/rvv009","article-title":"Defining the role of cognitive distance in the peer review process with an explorative study of a grant scheme in infection biology","volume":"24","year":"2015","journal-title":"Research Evaluation"},{"issue":"3","key":"key2023082411561161000_ref052","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1093\/reseval\/rvy018","article-title":"Follow the money? How Australian universities replicate national performance-based funding mechanisms","volume":"27","year":"2018","journal-title":"Research Evaluation"},{"issue":"7851","key":"key2023082411561161000_ref053","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1038\/d41586-021-00645-4","article-title":"A lot of room for bias': UK funder's data point to uneven playing field","volume":"591","year":"2021","journal-title":"Nature"},{"issue":"1","key":"key2023082411561161000_ref054","first-page":"180","article-title":"A corpus-driven cross-disciplinary study of inclusive and exclusive we in research article abstracts","volume":"15","year":"2022","journal-title":"LEARN Journal: Language Education and Acquisition Research Network"}],"container-title":["Journal of Documentation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/JD-12-2022-0261\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/JD-12-2022-0261\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T22:35:40Z","timestamp":1753396540000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/jd\/article\/79\/5\/1110-1123\/194257"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,7]]},"references-count":54,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2023,2,7]]},"published-print":{"date-parts":[[2023,8,25]]}},"alternative-id":["10.1108\/JD-12-2022-0261"],"URL":"https:\/\/doi.org\/10.1108\/jd-12-2022-0261","relation":{},"ISSN":["0022-0418"],"issn-type":[{"value":"0022-0418","type":"print"}],"subject":[],"published":{"date-parts":[[2023,2,7]]}}}