{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T04:30:06Z","timestamp":1774326606123,"version":"3.50.1"},"reference-count":36,"publisher":"Emerald","issue":"5","license":[{"start":{"date-parts":[[2020,6,15]],"date-time":"2020-06-15T00:00:00Z","timestamp":1592179200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["OIR"],"published-print":{"date-parts":[[2020,6,15]]},"abstract":"<jats:sec>\n                    <jats:title content-type=\"abstract-subheading\">Purpose<\/jats:title>\n                    <jats:p>Peer reviewer evaluations of academic papers are known to be variable in content and overall judgements but are important academic publishing safeguards. This article introduces a sentiment analysis program, PeerJudge, to detect praise and criticism in peer evaluations. It is designed to support editorial management decisions and reviewers in the scholarly publishing process and for grant funding decision workflows. The initial version of PeerJudge is tailored for reviews from F1000Research's open peer review publishing platform.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title content-type=\"abstract-subheading\">Design\/methodology\/approach<\/jats:title>\n                    <jats:p>PeerJudge uses a lexical sentiment analysis approach with a human-coded initial sentiment lexicon and machine learning adjustments and additions. It was built with an F1000Research development corpus and evaluated on a different F1000Research test corpus using reviewer ratings.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title content-type=\"abstract-subheading\">Findings<\/jats:title>\n                    <jats:p>PeerJudge can predict F1000Research judgements from negative evaluations in reviewers' comments more accurately than baseline approaches, although not from positive reviewer comments, which seem to be largely unrelated to reviewer decisions. Within the F1000Research mode of post-publication peer review, the absence of any detected negative comments is a reliable indicator that an article will be \u2018approved\u2019, but the presence of moderately negative comments could lead to either an approved or approved with reservations decision.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title content-type=\"abstract-subheading\">Originality\/value<\/jats:title>\n                    <jats:p>PeerJudge is the first transparent AI approach to peer review sentiment detection. It may be used to identify anomalous reviews with text potentially not matching judgements for individual checks or systematic bias assessments.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1108\/oir-11-2019-0347","type":"journal-article","created":{"date-parts":[[2020,7,1]],"date-time":"2020-07-01T09:33:11Z","timestamp":1593595991000},"page":"1057-1076","source":"Crossref","is-referenced-by-count":25,"title":["Automatically detecting open academic review praise and criticism"],"prefix":"10.1108","volume":"44","author":[{"given":"Mike","family":"Thelwall","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Eleanor-Rose","family":"Papas","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zena","family":"Nyakoojo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liz","family":"Allen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Verena","family":"Weigert","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"issue":"1","key":"key2020082008434622900_ref001","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1038\/s41467-018-08250-2","article-title":"The effect of publishing peer review reports on referee behavior in five scholarly journals","volume":"10","year":"2019","journal-title":"Nature Communications"},{"key":"key2020082008434622900_ref002","article-title":"PeerTax: investigating the taxonomy of peer reviews","year":"2019","journal-title":"eLife"},{"key":"key2020082008434622900_ref003","unstructured":"Clarivate (2018), \u201c2018 global state of peer review\u201d, available at: https:\/\/publons.com\/static\/Publons-Global-State-Of-Peer-Review-2018.pdf (accessed 5 November 2019)."},{"issue":"21","key":"key2020082008434622900_ref004","first-page":"1934","article-title":"Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial","volume":"170","year":"2010","journal-title":"Archives of Internal Medicine"},{"key":"key2020082008434622900_ref005","unstructured":"F1000Research (2019a), \u201cFAQs\u201d, available at: https:\/\/f1000research.com\/faqs (accessed 5 November 2019)."},{"key":"key2020082008434622900_ref006","unstructured":"F1000Research (2019b), \u201cReviewer guidelines\u201d, available at: https:\/\/f1000research.com\/for-referees\/guidelines (accessed 5 November 2019)."},{"issue":"4","key":"key2020082008434622900_ref007","doi-asserted-by":"crossref","first-page":"311","DOI":"10.3138\/jsp.44-4-001","article-title":"Defining and characterizing open peer review: a review of the literature","volume":"44","year":"2013","journal-title":"Journal of Scholarly Publishing"},{"key":"key2020082008434622900_ref008","unstructured":"Frontiersin (2018), \u201cAI-enhanced peer review: frontiers launches next generation of efficient, high-quality peer review, available at: https:\/\/blog.frontiersin.org\/2018\/12\/14\/artificial-intelligence-peer-review-assistant-aira\/ (accessed 5 November 2019)."},{"key":"key2020082008434622900_ref009","first-page":"1120","article-title":"DeepSentiPeer: harnessing sentiment in review texts to recommend peer review decisions","year":"2019"},{"key":"key2020082008434622900_ref010","volume-title":"AI Peer Reviewers Unleashed to Ease Publishing Grind","year":"2018"},{"issue":"21","key":"key2020082008434622900_ref011","doi-asserted-by":"crossref","first-page":"2784","DOI":"10.1001\/jama.287.21.2784","article-title":"Effects of editorial peer review: a systematic review","volume":"287","year":"2002","journal-title":"Jama"},{"issue":"21","key":"key2020082008434622900_ref012","doi-asserted-by":"crossref","first-page":"2786","DOI":"10.1001\/jama.287.21.2786","article-title":"Measuring the quality of editorial peer review","volume":"287","year":"2002","journal-title":"Jama"},{"key":"key2020082008434622900_ref013","first-page":"1647","article-title":"A dataset of peer reviews (PeerRead): collection, insights and NLP applications","year":"2018"},{"issue":"9","key":"key2020082008434622900_ref014","article-title":"Retrospective analysis of the quality of reports by author-suggested and non-author-suggested reviewers in journals operating on open or single-blind peer review models","volume":"5","year":"2015","journal-title":"BMJ Open"},{"issue":"4","key":"key2020082008434622900_ref015","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1145\/2732417","article-title":"The arbitrariness of reviews, and advice for school administrators","volume":"58","year":"2015","journal-title":"Communications of the ACM"},{"key":"key2020082008434622900_ref016","first-page":"167","article-title":"Applying machine learning techniques to investigate the influence of peer feedback on the writing process","volume-title":"Methods in Writing Process Research","year":"2014"},{"issue":"1","key":"key2020082008434622900_ref017","doi-asserted-by":"crossref","first-page":"1","DOI":"10.2200\/S00416ED1V01Y201204HLT016","article-title":"Sentiment analysis and opinion mining","volume":"5","year":"2012","journal-title":"Synthesis Lectures on Human Language Technologies"},{"issue":"1","key":"key2020082008434622900_ref018","doi-asserted-by":"publisher","first-page":"150547","DOI":"10.1098\/rsos.150547","article-title":"The peer reviewers' openness initiative: incentivizing open research practices through peer review","volume":"3","year":"2016","journal-title":"Royal Society Open Science"},{"key":"key2020082008434622900_ref019","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1186\/2050-6511-15-55","article-title":"Open, single-blind, double-blind: which peer review process do you prefer?","volume":"15","year":"2014","journal-title":"BMC Pharmacology and Toxicology"},{"issue":"9","key":"key2020082008434622900_ref020","article-title":"Artificial intelligence in peer review: how can evolutionary computation support journal editors?","volume":"12","year":"2017","journal-title":"PLoS One"},{"issue":"1","key":"key2020082008434622900_ref021","doi-asserted-by":"crossref","first-page":"132","DOI":"10.1002\/asi.22798","article-title":"Peer review in a changing world: an international study measuring the attitudes of researchers","volume":"64","year":"2013","journal-title":"Journal of the American Society for Information Science and Technology"},{"key":"key2020082008434622900_ref022","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1016\/j.procs.2017.10.035","article-title":"Development of review rating and reporting in open journal system","volume":"116","year":"2017","journal-title":"Procedia Computer Science"},{"issue":"3","key":"key2020082008434622900_ref023","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1145\/2979672","article-title":"Computational support for academic peer review: a perspective from artificial intelligence","volume":"60","year":"2017","journal-title":"Communications of the ACM"},{"key":"key2020082008434622900_ref024","doi-asserted-by":"publisher","first-page":"588","DOI":"10.12688\/f1000research.11369.2","article-title":"What is open peer review? A systematic review","volume":"6","year":"2017","journal-title":"F1000Research"},{"issue":"12","key":"key2020082008434622900_ref025","article-title":"Survey on open peer review: attitudes and experience amongst editors, authors and reviewers","volume":"12","year":"2017","journal-title":"PLoS ONE"},{"issue":"1","key":"key2020082008434622900_ref026","first-page":"286","article-title":"An overview of assessing the quality of peer review reports of scientific articles","volume":"46","year":"2019","journal-title":"International Journal of Information Management"},{"key":"key2020082008434622900_ref027","article-title":"Detecting arguing and sentiment in meetings","year":"2007"},{"key":"key2020082008434622900_ref028","volume-title":"Support Vector Machines","year":"2008"},{"issue":"2","key":"key2020082008434622900_ref029","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1162\/COLI_a_00049","article-title":"Lexicon-based methods for sentiment analysis","volume":"37","year":"2011","journal-title":"Computational Linguistics"},{"issue":"1","key":"key2020082008434622900_ref030","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1002\/asi.21662","article-title":"Sentiment strength detection for the social web","volume":"63","year":"2012","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"7175","key":"key2020082008434622900_ref031","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1136\/bmj.318.7175.23","article-title":"Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial","volume":"318","year":"1999","journal-title":"BMJ"},{"key":"key2020082008434622900_ref032","doi-asserted-by":"crossref","first-page":"c5729","DOI":"10.1136\/bmj.c5729","article-title":"Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial","volume":"341","year":"2010","journal-title":"BMJ"},{"issue":"1","key":"key2020082008434622900_ref033","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1192\/bjp.176.1.47","article-title":"Open peer review: a randomised controlled trial","volume":"176","year":"2000","journal-title":"The British Journal of Psychiatry"},{"key":"key2020082008434622900_ref034","first-page":"175","article-title":"Sentiment analysis of peer review texts for scholarly papers\u201d","year":"2018"},{"key":"key2020082008434622900_ref035","first-page":"502","article-title":"Automatically predicting peer-review helpfulness","year":"2011"},{"key":"key2020082008434622900_ref036","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1007\/978-981-287-868-7_41","article-title":"Metrics for automated review classification: what review data show","volume-title":"State-of-the-Art and Future Directions of Smart Learning","year":"2016"}],"container-title":["Online Information Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/OIR-11-2019-0347\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/OIR-11-2019-0347\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T22:44:16Z","timestamp":1753397056000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/oir\/article\/44\/5\/1057-1076\/511766"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,15]]},"references-count":36,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2020,6,15]]}},"alternative-id":["10.1108\/OIR-11-2019-0347"],"URL":"https:\/\/doi.org\/10.1108\/oir-11-2019-0347","relation":{"has-review":[{"id-type":"doi","id":"10.1108\/OIR-11-2019-0347\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-11-2019-0347\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-11-2019-0347\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-11-2019-0347\/v3\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-11-2019-0347\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-11-2019-0347\/v3\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1108\/OIR-11-2019-0347\/v1\/review1","asserted-by":"object"}]},"ISSN":["1468-4527"],"issn-type":[{"value":"1468-4527","type":"print"}],"subject":[],"published":{"date-parts":[[2020,6,15]]}}}