{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,23]],"date-time":"2026-01-23T22:11:13Z","timestamp":1769206273093,"version":"3.49.0"},"reference-count":43,"publisher":"Emerald","issue":"2\/3","license":[{"start":{"date-parts":[[2023,5,19]],"date-time":"2023-05-19T00:00:00Z","timestamp":1684454400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["EL"],"published-print":{"date-parts":[[2023,5,24]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-subheading\">Purpose<\/jats:title><jats:p>This paper aims to quantify the quality of peer reviews, evaluate them from different perspectives and develop a model to predict the review quality. In addition, this paper investigates effective features to distinguish the reviews' quality.\u00a0<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Design\/methodology\/approach<\/jats:title><jats:p>First, a fine-grained data set including peer review data, citations and review conformity scores was constructed. Second, metrics were proposed to evaluate the quality of peer reviews from three aspects. Third, five categories of features were proposed in terms of reviews, submissions and responses using natural language processing (NLP) techniques. Finally, different machine learning models were applied to predict the review quality, and feature analysis was performed to understand effective features.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Findings<\/jats:title><jats:p>The analysis results revealed that reviewers become more conservative and the review quality becomes worse over time in terms of these indicators. Among the three models, random forest model achieves the best performance on all three tasks. Sentiment polarity, review length, response length and readability are important factors that distinguish peer reviews\u2019 quality, which can help meta-reviewers value more worthy reviews when making final decisions.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-subheading\">Originality\/value<\/jats:title><jats:p>This study provides a new perspective for assessing review quality. Another originality of the research lies in the proposal of a novelty task that predict review quality. To address this task, a novel model was proposed which incorporated various of feature sets, thereby deepening the understanding of peer reviews.<\/jats:p><\/jats:sec>","DOI":"10.1108\/el-06-2022-0139","type":"journal-article","created":{"date-parts":[[2023,5,17]],"date-time":"2023-05-17T06:24:44Z","timestamp":1684304684000},"page":"186-203","source":"Crossref","is-referenced-by-count":8,"title":["Assessing and predicting the quality of peer reviews: a text mining approach"],"prefix":"10.1108","volume":"41","author":[{"given":"Jie","family":"Meng","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","published-online":{"date-parts":[[2023,5,19]]},"reference":[{"key":"key2023052306095988800_ref001","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1162\/tacl_a_00002","article-title":"Multiple instance learning networks for fine-grained sentiment analysis","volume":"6","year":"2018","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"key2023052306095988800_ref002","first-page":"518","article-title":"PaRe: a paper-reviewer matching approach using a common topic space","year":"2019"},{"key":"key2023052306095988800_ref003","first-page":"1916","article-title":"Peer grading the peer reviews: a dual-role approach for lightening the scholarly paper review process","year":"2021"},{"issue":"1","key":"key2023052306095988800_ref004","doi-asserted-by":"crossref","first-page":"347","DOI":"10.1007\/s11192-015-1704-5","article-title":"Quantifying the quality of peer reviewers through Zipf\u2019s law","volume":"106","year":"2016","journal-title":"Scientometrics"},{"issue":"3","key":"key2023052306095988800_ref005","doi-asserted-by":"crossref","first-page":"843","DOI":"10.1007\/s11192-011-0569-5","article-title":"Closed versus open reviewing of journal manuscripts: how far do comments differ in language use?","volume":"91","year":"2012","journal-title":"Scientometrics"},{"key":"key2023052306095988800_ref006","first-page":"207","article-title":"Aspect-based sentiment analysis of scientific reviews","year":"2020"},{"key":"key2023052306095988800_ref007","volume-title":"Readability Revisited: The New Dale-Chall Readability Formula","year":"1995"},{"key":"key2023052306095988800_ref008","article-title":"Inconsistency in conference peer review: revisiting the 2014 NeurIPS experiment","year":"2021"},{"issue":"1","key":"key2023052306095988800_ref009","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1002\/lob.10217","article-title":"Reviewing reviews: an evaluation of peer reviews of journal article submissions","volume":"27","year":"2018","journal-title":"Limnology and Oceanography Bulletin"},{"issue":"2","key":"key2023052306095988800_ref010","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1001\/jama.1994.03520020069019","article-title":"The effects of blinding on acceptance of research papers by peer review","volume":"272","year":"1994","journal-title":"JAMA: The Journal of the American Medical Association"},{"key":"key2023052306095988800_ref011","first-page":"1274","article-title":"Does my rebuttal matter? Insights from a major NLP","year":"2019"},{"key":"key2023052306095988800_ref012","first-page":"1120","article-title":"DeepSentiPeer: harnessing sentiment in review texts to recommend peer review decisions","year":"2019"},{"key":"key2023052306095988800_ref013","doi-asserted-by":"crossref","first-page":"e21718","DOI":"10.7554\/eLife.21718","article-title":"Gender bias in scholarly peer review","volume":"6","year":"2017","journal-title":"eLife"},{"issue":"1","key":"key2023052306095988800_ref014","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1016\/j.outlook.2008.05.006","article-title":"Quality of manuscript reviews in nursing research","volume":"57","year":"2009","journal-title":"Nursing Outlook"},{"key":"key2023052306095988800_ref015","first-page":"12533","article-title":"Mitigating manipulation in peer review via randomized reviewer assignments","volume":"33","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"key2023052306095988800_ref016","first-page":"1647","article-title":"A dataset of peer reviews (PeerRead): collection, insights and NLP applications","year":"2018"},{"issue":"4","key":"key2023052306095988800_ref017","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1145\/2732417","article-title":"The arbitrariness of reviews, and advice for school administrators","volume":"58","year":"2015","journal-title":"Communications of the ACM"},{"key":"key2023052306095988800_ref041","first-page":"1097","article-title":"Language technologies for enhancement of teaching and learning in writing","year":"2014"},{"issue":"2","key":"key2023052306095988800_ref018","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1016\/j.tele.2018.01.001","article-title":"Assessing the helpfulness of online hotel reviews: a classification-based approach","volume":"35","year":"2018","journal-title":"Telematics and Informatics"},{"key":"key2023052306095988800_ref019","first-page":"4914","article-title":"A neural citation count prediction model based on peer review text","year":"2019"},{"issue":"1","key":"key2023052306095988800_ref020","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1002\/bult.184","article-title":"Text mining","volume":"27","year":"2000","journal-title":"Bulletin of the American Society for Information Science and Technology"},{"issue":"4","key":"key2023052306095988800_ref02000","doi-asserted-by":"crossref","first-page":"1271","DOI":"10.1162\/qss_a_00156","article-title":"Analyzing sentiments in peer review reports: evidence from two science funding agencies","volume":"2","year":"2021","journal-title":"Quantitative Science Studies"},{"key":"key2023052306095988800_ref021","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1016\/j.dss.2014.01.011","article-title":"The influence of reviewer engagement characteristics on online review helpfulness: a text regression model","volume":"61","year":"2014","journal-title":"Decision Support Systems"},{"key":"key2023052306095988800_ref022","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1016\/j.patrec.2017.09.020","article-title":"A decision support tool using order weighted averaging for conference review assignment","volume":"105","year":"2018","journal-title":"Pattern Recognition Letters"},{"key":"key2023052306095988800_ref023","first-page":"56","article-title":"Context-aware helpfulness prediction for online product reviews","volume-title":"Asia Information Retrieval Symposium","year":"2020"},{"key":"key2023052306095988800_ref042","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1016\/j.procs.2017.10.035","article-title":"Development of review rating and reporting in open journal system","volume":"116","year":"2017","journal-title":"Procedia Computer Science"},{"key":"key2023052306095988800_ref024","first-page":"136","article-title":"Automated assessment of review quality using latent semantic analysis","year":"2011"},{"key":"key2023052306095988800_ref025","first-page":"1","article-title":"Open peer review in the era of open science: a pilot study of researchers' perceptions","year":"2017"},{"issue":"14","key":"key2023052306095988800_ref026","doi-asserted-by":"crossref","first-page":"1675","DOI":"10.1001\/jama.295.14.1675","article-title":"Effect of blinded peer review on abstract acceptance","volume":"295","year":"2006","journal-title":"JAMA"},{"key":"key2023052306095988800_ref027","first-page":"356","article-title":"JCDL 2021 tutorial on systemic challenges and computational solutions on bias and unfairness in peer review","year":"2021"},{"issue":"1","key":"key2023052306095988800_ref028","first-page":"1","article-title":"Content and communication: how can peer review provide helpful feedback about the writing?","volume":"8","year":"2008","journal-title":"BMC Medical Research Methodology"},{"key":"key2023052306095988800_ref029","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1016\/j.ijinfomgt.2018.07.002","article-title":"An overview of assessing the quality of peer review reports of scientific articles","volume":"46","year":"2019","journal-title":"International Journal of Information Management"},{"issue":"1","key":"key2023052306095988800_ref030","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1002\/leap.1009","article-title":"The JBJS peer\u2010review scoring scale: a valid, reliable instrument for measuring the quality of peer review reports","volume":"29","year":"2016","journal-title":"Learned Publishing"},{"issue":"48","key":"key2023052306095988800_ref031","doi-asserted-by":"crossref","first-page":"12708","DOI":"10.1073\/pnas.1707323114","article-title":"Reviewer bias in single-versus double-blind peer review","volume":"114","year":"2017","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"key2023052306095988800_ref032","article-title":"An open review of OpenReview: a critical analysis of the machine learning conference review process","year":"2020"},{"key":"key2023052306095988800_ref033","doi-asserted-by":"crossref","first-page":"72870","DOI":"10.1109\/ACCESS.2018.2878478","article-title":"Text mining analysis of teaching evaluation questionnaires for the selection of outstanding teaching faculty members","volume":"6","year":"2018","journal-title":"IEEE Access"},{"issue":"7","key":"key2023052306095988800_ref034","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1016\/S0895-4356(99)00047-5","article-title":"Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts","volume":"52","year":"1999","journal-title":"Journal of Clinical Epidemiology"},{"issue":"1","key":"key2023052306095988800_ref035","article-title":"Optimal spot-checking for improving evaluation accuracy of peer grading systems","volume":"32","year":"2018","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"key2023052306095988800_ref036","first-page":"175","article-title":"Sentiment analysis of peer review texts for scholarly papers","year":"2018"},{"key":"key2023052306095988800_ref037","first-page":"502","article-title":"Automatically predicting peer-review helpfulness","year":"2011"},{"key":"key2023052306095988800_ref038","first-page":"211","article-title":"Assessing reviewers' performance based on mining problem localization in peer-review data","year":"2010"},{"key":"key2023052306095988800_ref039","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1613\/jair.1.12862","article-title":"Can we automate scientific reviewing?","volume":"75","year":"2022","journal-title":"Journal of Artificial Intelligence Research"},{"key":"key2023052306095988800_ref040","first-page":"2620","article-title":"Guided helpful answer identification in E-commerce","year":"2020"}],"container-title":["The Electronic Library"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/EL-06-2022-0139\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/EL-06-2022-0139\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T01:07:32Z","timestamp":1753405652000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/el\/article\/41\/2-3\/186-203\/90980"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,19]]},"references-count":43,"journal-issue":{"issue":"2\/3","published-online":{"date-parts":[[2023,5,19]]},"published-print":{"date-parts":[[2023,5,24]]}},"alternative-id":["10.1108\/EL-06-2022-0139"],"URL":"https:\/\/doi.org\/10.1108\/el-06-2022-0139","relation":{},"ISSN":["0264-0473","0264-0473"],"issn-type":[{"value":"0264-0473","type":"print"},{"value":"0264-0473","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,19]]}}}