{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T19:18:51Z","timestamp":1775157531802,"version":"3.50.1"},"reference-count":72,"publisher":"Emerald","issue":"1","license":[{"start":{"date-parts":[[2023,10,2]],"date-time":"2023-10-02T00:00:00Z","timestamp":1696204800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["AJIM"],"published-print":{"date-parts":[[2025,1,2]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-subheading\">Purpose<\/jats:title><jats:p>Technology is sometimes used to support assessments of academic research in the form of automatically generated bibliometrics for reviewers to consult during their evaluations or by replacing some or all human judgements. With artificial intelligence (AI), there is increasing scope to use technology to assist research assessment processes in new ways. Since transparency and fairness are widely considered important for research assessment and AI introduces new issues, this review investigates their implications.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Design\/methodology\/approach<\/jats:title><jats:p>This article reviews and briefly summarises transparency and fairness concerns in general terms and through the issues that they raise for various types of Technology Assisted Research Assessment (TARA).<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Findings<\/jats:title><jats:p>Whilst TARA can have varying levels of problems with both transparency and bias, in most contexts it is unclear whether it worsens the transparency and bias problems that are inherent in peer review.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Originality\/value<\/jats:title><jats:p>This is the first analysis that focuses on algorithmic bias and transparency issues for technology assisted research assessment.<\/jats:p><\/jats:sec>","DOI":"10.1108\/ajim-04-2023-0119","type":"journal-article","created":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T09:41:15Z","timestamp":1695980475000},"page":"175-190","source":"Crossref","is-referenced-by-count":9,"title":["Technology assisted research assessment: algorithmic bias and\u00a0transparency issues"],"prefix":"10.1108","volume":"77","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6065-205X","authenticated-orcid":false,"given":"Mike","family":"Thelwall","sequence":"first","affiliation":[]},{"given":"Kayvan","family":"Kousha","sequence":"additional","affiliation":[]}],"member":"140","published-online":{"date-parts":[[2023,10,2]]},"reference":[{"issue":"3","key":"key2025010116220977300_ref001","doi-asserted-by":"crossref","first-page":"159","DOI":"10.3152\/147154403781776645","article-title":"Characteristics of highly cited papers","volume":"12","year":"2003","journal-title":"Research Evaluation"},{"issue":"3","key":"key2025010116220977300_ref002","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1093\/reseval\/rvv008","article-title":"Evaluating scientific research in Italy: the 2004-10 research evaluation exercise","volume":"24","year":"2015","journal-title":"Research Evaluation"},{"issue":"1","key":"key2025010116220977300_ref003","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.leaqua.2013.10.014","article-title":"What makes articles highly cited?","volume":"25","year":"2014","journal-title":"The Leadership Quarterly"},{"issue":"6","key":"key2025010116220977300_ref004","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1080\/20009666.2021.1965704","article-title":"Clinical peer review; A mandatory process with potential inherent bias in desperate need of reform","volume":"11","year":"2021","journal-title":"Journal of Community Hospital Internal Medicine Perspectives"},{"issue":"5","key":"key2025010116220977300_ref005","first-page":"394","article-title":"The academic conference as a chilly climate for women: effects of gender representation on experiences of sexism, coping responses, and career intentions","volume":"78","year":"2018","journal-title":"Sex Roles"},{"issue":"1","key":"key2025010116220977300_ref006","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1016\/j.joi.2012.10.001","article-title":"The use of percentiles and percentile rank classes in the analysis of bibliometric data: opportunities and limits","volume":"7","year":"2013","journal-title":"Journal of Informetrics"},{"issue":"10","key":"key2025010116220977300_ref007","doi-asserted-by":"crossref","first-page":"2765","DOI":"10.1287\/mnsc.2015.2285","article-title":"Looking across and looking beyond the knowledge Frontier: intellectual distance, novelty, and resource allocation in science","volume":"62","year":"2016","journal-title":"Management Science"},{"issue":"2","key":"key2025010116220977300_ref009","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1007\/s11192-008-2141-5","article-title":"Rejecting and resisting Nobel class discoveries: accounts by Nobel Laureates","volume":"81","year":"2009","journal-title":"Scientometrics"},{"issue":"7","key":"key2025010116220977300_ref010","doi-asserted-by":"crossref","first-page":"635","DOI":"10.1002\/(SICI)1097-4571(2000)51:7<635::AID-ASI6>3.0.CO;2-H","article-title":"How can we investigate citation behavior? A study of reasons for citing literature in communication","volume":"51","year":"2000","journal-title":"Journal of the American Society for Information Science"},{"issue":"8","key":"key2025010116220977300_ref011","doi-asserted-by":"crossref","first-page":"3157","DOI":"10.1073\/pnas.1014871108","article-title":"Understanding current causes of women's underrepresentation in science","volume":"108","year":"2011","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"1","key":"key2025010116220977300_ref012","first-page":"1","article-title":"AI-assisted peer review","volume":"8","year":"2021","journal-title":"Humanities and Social Sciences Communications"},{"key":"key2025010116220977300_ref013","first-page":"434","article-title":"Predicting citation counts of papers","year":"2015"},{"key":"key2025010116220977300_ref014","doi-asserted-by":"crossref","unstructured":"Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H. and Chen, K. (2015), \u201cXgboost: extreme gradient boosting. R package version 0.4-2\u201d, available at: https:\/\/xgboost.readthedocs.io\/en\/stable\/","DOI":"10.32614\/CRAN.package.xgboost"},{"issue":"5","key":"key2025010116220977300_ref015","doi-asserted-by":"crossref","first-page":"825","DOI":"10.1002\/sce.21671","article-title":"Can AI be racist? Color\u2010evasiveness in the application of machine learning to science assessments","volume":"105","year":"2021","journal-title":"Science Education"},{"issue":"8","key":"key2025010116220977300_ref016","doi-asserted-by":"crossref","first-page":"631","DOI":"10.1093\/biosci\/biz063","article-title":"Widespread biases in ecological and evolutionary studies","volume":"69","year":"2019","journal-title":"Bioscience"},{"issue":"7","key":"key2025010116220977300_ref017","doi-asserted-by":"crossref","first-page":"809","DOI":"10.1080\/21670811.2016.1208053","article-title":"Algorithmic transparency in the news media","volume":"5","year":"2017","journal-title":"Digital Journalism"},{"key":"key2025010116220977300_ref018","first-page":"241","volume-title":"Questionnaires Mentioned in Academic Research 1996-2019: Rapid Increase but Declining Citation Impact","year":"2022"},{"key":"key2025010116220977300_ref019","first-page":"580","article-title":"A SUPER* algorithm to optimize paper bidding in peer review","year":"2020"},{"issue":"6","key":"key2025010116220977300_ref020","first-page":"797","article-title":"Face recognition algorithms and the other\u2010race effect: computational mechanisms for a developmental contact hypothesis","volume":"26","year":"2002","journal-title":"Cognitive Science"},{"issue":"1","key":"key2025010116220977300_ref021","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1257\/jep.8.1.165","article-title":"How are the mighty fallen: rejected classic articles by leading economists","volume":"8","year":"1994","journal-title":"Journal of Economic Perspectives"},{"issue":"37","key":"key2025010116220977300_ref022","doi-asserted-by":"crossref","first-page":"eaay7120","DOI":"10.1126\/scirobotics.aay7120","article-title":"XAI\u2014explainable artificial intelligence","volume":"4","year":"2019","journal-title":"Science Robotics"},{"key":"key2025010116220977300_ref023","unstructured":"Harzing, A. (2017), \u201cWeb of Science: how to be robbed of 10 years of citations in one week!\u201d, available at: https:\/\/harzing.com\/blog\/2017\/02\/web-of-science-to-be-robbed-of-10-years-of-citations-in-one-week"},{"key":"key2025010116220977300_ref024","article-title":"The metric tide: correlation analysis of REF2014 scores and metrics (supplementary report II to the independent review of the role of metrics in research assessment and management)","author":"HEFCE","year":"2015","journal-title":"Higher Education Funding Council for England"},{"issue":"7548","key":"key2025010116220977300_ref025","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1038\/520429a","article-title":"Bibliometrics: the Leiden Manifesto for research metrics","volume":"520","year":"2015","journal-title":"Nature"},{"issue":"1","key":"key2025010116220977300_ref026","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1023\/A:1022670432373","article-title":"Impartial judgment by the \u2018gatekeepers\u2019 of science: fallibility and accountability in the peer review process","volume":"8","year":"2003","journal-title":"Advances in Health Sciences Education"},{"key":"key2025010116220977300_ref027","doi-asserted-by":"publisher","first-page":"23","DOI":"10.3389\/frma.2018.00023","article-title":"Dimensions: building context for search and evaluation","volume":"3","year":"2018","journal-title":"Frontiers in Research Metrics and Analytics"},{"issue":"9","key":"key2025010116220977300_ref028","doi-asserted-by":"crossref","first-page":"e1002541","DOI":"10.1371\/journal.pbio.1002541","article-title":"Relative citation ratio (RCR): a new metric that uses citation rates to measure influence at the article level","volume":"14","year":"2016","journal-title":"PLoS Biology"},{"issue":"3","key":"key2025010116220977300_ref029","doi-asserted-by":"crossref","first-page":"296","DOI":"10.1016\/j.jclinepi.2012.09.015","article-title":"Citation bias favoring statistically significant studies was present in medical research","volume":"66","year":"2013","journal-title":"Journal of Clinical Epidemiology"},{"issue":"1","key":"key2025010116220977300_ref030","first-page":"1","article-title":"Transparency of peer review: a semi-structured interview study with chief editors from social sciences and humanities","volume":"6","year":"2021","journal-title":"Research Integrity and Peer Review"},{"issue":"4","key":"key2025010116220977300_ref031","doi-asserted-by":"crossref","first-page":"984","DOI":"10.1002\/asi.23734","article-title":"Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge?","volume":"68","year":"2017","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"3","key":"key2025010116220977300_ref032","first-page":"388","article-title":"Algorithmic bias: review, synthesis, and future research directions","volume":"31","year":"2021","journal-title":"European Journal of Information Systems"},{"key":"key2025010116220977300_ref033","first-page":"1","article-title":"DeepASPeer: towards an aspect-level sentiment controllable framework for decision prediction from academic peer reviews","year":"2022"},{"issue":"7","key":"key2025010116220977300_ref034","doi-asserted-by":"crossref","first-page":"2966","DOI":"10.1287\/mnsc.2018.3093","article-title":"Algorithmic bias? An empirical study of apparent gender-based discrimination in the display of STEM career ads","volume":"65","year":"2019","journal-title":"Management Science"},{"issue":"3","key":"key2025010116220977300_ref035","doi-asserted-by":"crossref","first-page":"481","DOI":"10.1002\/asi.21682","article-title":"Citation flows in the zones of influence of scientific collaborations","volume":"63","year":"2012","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"3","key":"key2025010116220977300_ref036","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1108\/JICES-06-2018-0056","article-title":"Detecting racial bias in algorithms and machine learning","volume":"16","year":"2018","journal-title":"Journal of Information, Communication and Ethics in Society"},{"issue":"1","key":"key2025010116220977300_ref037","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1002\/asi.22784","article-title":"Bias in peer review","volume":"64","year":"2013","journal-title":"Journal of the American Society for Information Science and Technology"},{"key":"key2025010116220977300_ref038","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1007\/BF01173636","article-title":"Publication prejudices: an experimental study of confirmatory bias in the peer review system","volume":"1","year":"1977","journal-title":"Cognitive Therapy and Research"},{"issue":"4","key":"key2025010116220977300_ref039","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1093\/reseval\/rvu008","article-title":"A critical review of SCImago journal & country rank","volume":"24","year":"2015","journal-title":"Research Evaluation"},{"issue":"6","key":"key2025010116220977300_ref040","first-page":"1","article-title":"A survey on bias and fairness in machine learning","volume":"54","year":"2021","journal-title":"ACM Computing Surveys (CSUR)"},{"issue":"3810","key":"key2025010116220977300_ref041","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1126\/science.159.3810.56","article-title":"The Matthew Effect in Science: the reward and communication systems of science are considered","volume":"159","year":"1968","journal-title":"Science"},{"issue":"1","key":"key2025010116220977300_ref042","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1007\/s11192-015-1765-5","article-title":"The journal coverage of Web of Science and Scopus: a comparative analysis","volume":"106","year":"2016","journal-title":"Scientometrics"},{"issue":"1","key":"key2025010116220977300_ref043","first-page":"881","article-title":"Typographic layout and first impressions: testing how changes in text layout influence reader's judgments of documents","volume":"48","year":"2014","journal-title":"Visible Language"},{"key":"key2025010116220977300_ref044","first-page":"375","article-title":"Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review","volume":"2281","year":"2021","journal-title":"BMJ"},{"issue":"10","key":"key2025010116220977300_ref045","doi-asserted-by":"crossref","first-page":"6363","DOI":"10.1007\/s00521-019-04144-6","article-title":"Assessing gender bias in machine translation: a case study with google translate","volume":"32","year":"2020","journal-title":"Neural Computing and Applications"},{"issue":"3","key":"key2025010116220977300_ref073","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1016\/S0048-7333(00)00082-2","article-title":"Influence of interdisciplinarity on peer-review and bibliometric evaluations in physics research","volume":"30","year":"2001","journal-title":"Research Policy"},{"issue":"3","key":"key2025010116220977300_ref046","doi-asserted-by":"crossref","first-page":"947","DOI":"10.1007\/s10439-021-02739-6","article-title":"Citation diversity statement in BMES journals","volume":"49","year":"2021","journal-title":"Annals of Biomedical Engineering"},{"issue":"1","key":"key2025010116220977300_ref047","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1016\/j.joi.2014.11.010","article-title":"Field-normalized citation impact indicators using algorithmically constructed classification systems of science","volume":"9","year":"2015","journal-title":"Journal of Informetrics"},{"issue":"2","key":"key2025010116220977300_ref048","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1007\/s11192-008-0211-3","article-title":"Persistent nepotism in peer-review","volume":"74","year":"2008","journal-title":"Scientometrics"},{"issue":"2","key":"key2025010116220977300_ref049","doi-asserted-by":"publisher","first-page":"360","DOI":"10.1073\/pnas.1418218112","article-title":"Measuring the effectiveness of scientific gatekeeping","volume":"112","year":"2015","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"1","key":"key2025010116220977300_ref050","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1016\/j.joi.2016,12.002","article-title":"Three practical field normalised alternative indicator formulae for research evaluation","volume":"11","year":"2017","journal-title":"Journal of Informetrics"},{"issue":"3","key":"key2025010116220977300_ref051","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1108\/OIR-05-2017-0152","article-title":"Gender bias in machine learning for sentiment analysis","volume":"42","year":"2018","journal-title":"Online Information Review"},{"issue":"8","key":"key2025010116220977300_ref052","doi-asserted-by":"crossref","first-page":"979","DOI":"10.1002\/asi.24316","article-title":"Female citation impact superiority 1996-2018 in six out of seven English\u2010speaking nations","volume":"71","year":"2020","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"8","key":"key2025010116220977300_ref074","doi-asserted-by":"crossref","first-page":"941","DOI":"10.1002\/asi.24767","article-title":"In which fields are citations indicators of research quality?","volume":"74","year":"2023","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"6","key":"key2025010116220977300_ref053","doi-asserted-by":"crossref","first-page":"1124","DOI":"10.1002\/asi.23252","article-title":"Are scholarly articles disproportionately read in their own country? An analysis of Mendeley readers","volume":"66","year":"2015","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"key2025010116220977300_ref055","article-title":"Can REF output quality scores be assigned by AI? Experimental evidence","year":"2022","journal-title":"arXiv preprint arXiv:2212.08041"},{"issue":"2","key":"key2025010116220977300_ref056","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1162\/qss_a_00258","article-title":"Predicting article quality scores with machine learning: the UK Research Excellence Framework","volume":"4","year":"2023","journal-title":"Quantitative Science Studies"},{"issue":"3","key":"key2025010116220977300_ref054","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/j.lisr.2021.101094","article-title":"Is research with qualitative data more prevalent and impactful now? Interviews, case studies, focus groups and ethnographies","volume":"43","year":"2021","journal-title":"Library and Information Science Research"},{"issue":"1","key":"key2025010116220977300_ref057","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1007\/s40614-019-00191-5","article-title":"Replication research, publication bias, and applied behavior analysis","volume":"42","year":"2019","journal-title":"Perspectives on Behavior Science"},{"issue":"1","key":"key2025010116220977300_ref058","doi-asserted-by":"crossref","first-page":"article 29","DOI":"10.1057\/s41599-019-0233-x","article-title":"Systematic analysis of agreement between metrics and peer review in the UK REF","volume":"5","year":"2019","journal-title":"Palgrave Communications"},{"key":"key2025010116220977300_ref059","article-title":"An open review of openreview: a critical analysis of the machine learning conference review process","year":"2020","journal-title":"arXiv preprint arXiv:2010.05137"},{"key":"key2025010116220977300_ref060","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.jclinepi.2020.11.019","article-title":"Citation bias and other determinants of citation in biomedical research: findings from six citation networks","volume":"132","year":"2021","journal-title":"Journal of Clinical Epidemiology"},{"key":"key2025010116220977300_ref061","article-title":"Accuracy of citation data in Web of science and scopus","year":"2019","journal-title":"arXiv preprint arXiv:1906.07011"},{"issue":"1","key":"key2025010116220977300_ref062","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1023\/A:1010549719484","article-title":"Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance","volume":"51","year":"2001","journal-title":"Scientometrics"},{"issue":"3","key":"key2025010116220977300_ref063","doi-asserted-by":"crossref","first-page":"1803","DOI":"10.1007\/s11192-019-03264-z","article-title":"Web of science and scopus language coverage","volume":"121","year":"2019","journal-title":"Scientometrics"},{"key":"key2025010116220977300_ref064","article-title":"Explainable artificial intelligence: a systematic review","year":"2020","journal-title":"arXiv preprint arXiv:2006.00093"},{"issue":"3","key":"key2025010116220977300_ref065","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1007\/s11192-011-0354-5","article-title":"Towards a new crown indicator: an empirical analysis","volume":"87","year":"2011","journal-title":"Scientometrics"},{"issue":"3","key":"key2025010116220977300_ref066","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1093\/reseval\/rvv009","article-title":"Defining the role of cognitive distance in the peer review process with an explorative study of a grant scheme in infection biology","volume":"24","year":"2015","journal-title":"Research Evaluation"},{"issue":"8","key":"key2025010116220977300_ref067","doi-asserted-by":"crossref","first-page":"1416","DOI":"10.1016\/j.respol.2017.06.006","article-title":"Bias against novelty in science: a cautionary tale for users of bibliometric indicators","volume":"46","year":"2017","journal-title":"Research Policy"},{"key":"key2025010116220977300_ref068","doi-asserted-by":"crossref","unstructured":"Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S. and Johnson, B. (2015), \u201cThe metric tide\u201d, Report of the independent review of the role of metrics in research assessment and management\u201d, available at: https:\/\/www.ukri.org\/publications\/review-of-metrics-in-research-assessment-and-management\/","DOI":"10.4135\/9781473978782"},{"issue":"12","key":"key2025010116220977300_ref069","article-title":"Algorithmically outsourcing the detection of statistical errors and other problems","volume":"37","year":"2018","journal-title":"The EMBO Journal"},{"key":"key2025010116220977300_ref070","first-page":"563","article-title":"Explainable AI: a brief survey on history, research areas, approaches and challenges","year":"2019"},{"issue":"1","key":"key2025010116220977300_ref071","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1087\/20100103","article-title":"CrossCheck: an effective tool for detecting plagiarism","volume":"23","year":"2010","journal-title":"Learned Publishing"}],"container-title":["Aslib Journal of Information Management"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/AJIM-04-2023-0119\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/AJIM-04-2023-0119\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T23:00:40Z","timestamp":1753398040000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/ajim\/article\/77\/1\/175-190\/1243936"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,2]]},"references-count":72,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,10,2]]},"published-print":{"date-parts":[[2025,1,2]]}},"alternative-id":["10.1108\/AJIM-04-2023-0119"],"URL":"https:\/\/doi.org\/10.1108\/ajim-04-2023-0119","relation":{},"ISSN":["2050-3806"],"issn-type":[{"value":"2050-3806","type":"print"}],"subject":[],"published":{"date-parts":[[2023,10,2]]}}}