{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T04:01:20Z","timestamp":1772164880890,"version":"3.50.1"},"reference-count":48,"publisher":"MIT Press","issue":"4","license":[{"start":{"date-parts":[[2021,9,30]],"date-time":"2021-09-30T00:00:00Z","timestamp":1632960000000},"content-version":"vor","delay-in-days":272,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001602","name":"Science Foundation Ireland","doi-asserted-by":"publisher","award":["17\/SPR\/5319"],"award-info":[{"award-number":["17\/SPR\/5319"]}],"id":[{"id":"10.13039\/501100001602","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,12,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Using a novel combination of methods and data sets from two national funding agency contexts, this study explores whether review sentiment can be used as a reliable proxy for understanding peer reviewer opinions. We measure reviewer opinions via their review sentiments on both specific review subjects and proposals\u2019 overall funding worthiness with three different methods: manual content analysis and two dictionary-based sentiment analysis algorithms (TextBlob and VADER). The reliability of review sentiment to detect reviewer opinions is addressed by its correlation with review scores and proposals\u2019 rankings and funding decisions. We find in our samples that review sentiments correlate with review scores or rankings positively, and the correlation is stronger for manually coded than for algorithmic results; manual and algorithmic results are overall correlated across different funding programs, review sections, languages, and agencies, but the correlations are not strong; and manually coded review sentiments can quite accurately predict whether proposals are funded, whereas the two algorithms predict funding success with moderate accuracy. The results suggest that manual analysis of review sentiments can provide a reliable proxy of grant reviewer opinions, whereas the two SA algorithms can be useful only in some specific situations.<\/jats:p>","DOI":"10.1162\/qss_a_00156","type":"journal-article","created":{"date-parts":[[2021,9,30]],"date-time":"2021-09-30T11:59:02Z","timestamp":1633003142000},"page":"1271-1295","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":25,"title":["Analyzing sentiments in peer review reports: Evidence from two science funding agencies"],"prefix":"10.1162","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3347-3982","authenticated-orcid":true,"given":"Junwen","family":"Luo","sequence":"first","affiliation":[{"name":"School of Information and Communication Studies, University College Dublin, Dublin, Ireland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4977-0877","authenticated-orcid":true,"given":"Thomas","family":"Feliciani","sequence":"additional","affiliation":[{"name":"School of Sociology and Geary Institute of Public Policy, University College Dublin, Dublin, Ireland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5507-8177","authenticated-orcid":true,"given":"Martin","family":"Reinhart","sequence":"additional","affiliation":[{"name":"Robert K. Merton Center for Science Studies, Humboldt-Universit\u00e4t zu Berlin, Berlin, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1710-2647","authenticated-orcid":true,"given":"Judith","family":"Hartstein","sequence":"additional","affiliation":[{"name":"German Centre for Higher Education Research and Science Studies (DZHW), Berlin, Germany"},{"name":"Faculty of Humanities and Social Sciences, Humboldt-Universit\u00e4t zu Berlin, Berlin, Germany"}]},{"given":"Vineeth","family":"Das","sequence":"additional","affiliation":[{"name":"School of Sociology and Geary Institute of Public Policy, University College Dublin, Dublin, Ireland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4394-6560","authenticated-orcid":true,"given":"Olalere","family":"Alabi","sequence":"additional","affiliation":[{"name":"School of Sociology and Geary Institute of Public Policy, University College Dublin, Dublin, Ireland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8788-466X","authenticated-orcid":true,"given":"Kalpana","family":"Shankar","sequence":"additional","affiliation":[{"name":"School of Information and Communication Studies, University College Dublin, Dublin, Ireland"}]}],"member":"281","published-online":{"date-parts":[[2021,12,1]]},"reference":[{"issue":"9","key":"2022040819223555600_bib1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0046054","article-title":"Peer review of grant applications: criteria used and qualitative study of reviewer practices","volume":"7","author":"Abdoul","year":"2012","journal-title":"PLOS ONE"},{"key":"2022040819223555600_bib2","first-page":"16","article-title":"Bias and truth in science evaluation: A simulation model of grant review panel discussions","volume-title":"Proceedings of the Workshop Reducing Online Misinformation through Credible Information Retrieval (ROMCIR 2021) Co-Located with the 43rd European Conference on Information Retrieval (ECIR 2021)","author":"Bethencourt","year":"2021"},{"key":"2022040819223555600_bib3","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1007\/s11192-020-03348-1","article-title":"Arbitrariness in the peer review process","volume":"123","author":"Brezis","year":"2020","journal-title":"Scientometrics"},{"key":"2022040819223555600_bib4","doi-asserted-by":"publisher","first-page":"e53249","DOI":"10.7554\/eLife.53249","article-title":"Meta-research: Large-scale language analysis of peer review reports","volume":"9","author":"Buljan","year":"2020","journal-title":"eLife"},{"issue":"25","key":"2022040819223555600_bib5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1057\/s41599-020-00703-8","article-title":"AI-assisted peer review","volume":"8","author":"Checco","year":"2021","journal-title":"Humanities and Social Science Communications"},{"issue":"1","key":"2022040819223555600_bib6","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1017\/S0140525X00065675","article-title":"The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation","volume":"14","author":"Cicchetti","year":"1991","journal-title":"Behavioral and Brain Sciences"},{"issue":"4523","key":"2022040819223555600_bib7","doi-asserted-by":"publisher","first-page":"881","DOI":"10.1126\/science.7302566","article-title":"Chance and consensus in peer review","volume":"214","author":"Cole","year":"1981","journal-title":"Science"},{"key":"2022040819223555600_bib8","doi-asserted-by":"publisher","first-page":"316","DOI":"10.1038\/d41586-019-01517-8","article-title":"Artificial intelligence is selecting grant reviewers in China","volume":"569","author":"Cyranoski","year":"2019","journal-title":"Nature"},{"key":"2022040819223555600_bib9","doi-asserted-by":"publisher","first-page":"4171","DOI":"10.18653\/v1\/N19-1423","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Devlin","year":"2019"},{"key":"2022040819223555600_bib10","volume-title":"H2020 Evaluation Guidelines","author":"European Commission","year":"2015"},{"issue":"1","key":"2022040819223555600_bib11","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1007\/s11192-019-03205-w","article-title":"A scoping review of simulation models of peer review","volume":"121","author":"Feliciani","year":"2019","journal-title":"Scientometrics"},{"issue":"3","key":"2022040819223555600_bib12","doi-asserted-by":"publisher","first-page":"8","DOI":"10.18564\/jasss.4284","article-title":"Grade language heterogeneity in simulation models of peer review","volume":"23","author":"Feliciani","year":"2020","journal-title":"Journal of Artificial Societies and Social Simulation"},{"issue":"1","key":"2022040819223555600_bib13","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1016\/j.jclinepi.2011.05.001","article-title":"Panel discussion does not improve reliability of peer review for medical research grant proposals","volume":"65","author":"Fogelholm","year":"2012","journal-title":"Journal of Clinical Epidemiology"},{"key":"2022040819223555600_bib15","first-page":"21","article-title":"Refereeing and peer review: How the peer review of research grant proposals works and what scientists say about it","volume":"10","author":"Garfield","year":"1987","journal-title":"Essays of an Information Scientists"},{"key":"2022040819223555600_bib16","doi-asserted-by":"publisher","first-page":"105383","DOI":"10.1016\/j.knosys.2019.105383","article-title":"Predicting literature\u2019s early impact with sentiment analysis in Twitter","volume":"192","author":"Hassan","year":"2020","journal-title":"Knowledge-Based Systems"},{"key":"2022040819223555600_bib17","doi-asserted-by":"publisher","first-page":"419","DOI":"10.1007\/BF02020704","article-title":"Peer review at the Deutsche Forschungsgemeinschaft","volume":"19","author":"Hartmann","year":"1990","journal-title":"Scientometrics"},{"issue":"1","key":"2022040819223555600_bib18","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1177\/0306312709335405","article-title":"Editorial judgments: A praxeology of \u201cVoting\u201d in peer review","volume":"40","author":"Hirschauer","year":"2010","journal-title":"Social Studies of Science"},{"issue":"1","key":"2022040819223555600_bib19","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1609\/icwsm.v8i1.14550","article-title":"VADER: a parsimonious rule-based model for sentiment analysis of social media text","volume":"8","author":"Hutto","year":"2014","journal-title":"Proceedings of the International AAAI Conference on Web and Social Media"},{"key":"2022040819223555600_bib20","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1023\/A:1017181826899","article-title":"Glossary of terms. Machine learning\u2014Special issue on applications of machine learning and the knowledge discovery process","volume":"30","author":"Kohavi","year":"1998","journal-title":"Machine Learning"},{"key":"2022040819223555600_bib21","first-page":"135","article-title":"Textvergleich als Grundlage zur Beschreibung einer wissenschaftlichen Textsorte: Das Peer Review","volume-title":"Kontrastive Fachsprachenforschung","author":"Kretzenbacher","year":"1992"},{"key":"2022040819223555600_bib22","first-page":"175","article-title":"\u2018\u2026 sicherlich von Interesse, wenngleich\u2026\u2019 Das Peer Review als bewertende Textsorte der Wissenschaftssprache","volume-title":"Linguistik der Wissenschaftssprache","author":"Kretzenbacher","year":"1995"},{"issue":"1\u20132","key":"2022040819223555600_bib23","doi-asserted-by":"publisher","first-page":"59","DOI":"10.24989\/fs.v34i1-2.1261","article-title":"\u201cThe wording is on occasion somewhat emotional\u201d: A qualitative study of English and German peer reviews for a chemical journal","volume":"39","author":"Kretzenbacher","year":"2017","journal-title":"Fachsprache"},{"key":"2022040819223555600_bib24","doi-asserted-by":"publisher","DOI":"10.4159\/9780674054158","volume-title":"How professors think: Inside the curious world of academic judgment","author":"Lamont","year":"2010"},{"key":"2022040819223555600_bib25","volume-title":"Efficiency and flexibility in research funding. A comparative study of funding instruments and review criteria","author":"Langfeldt","year":"2016"},{"issue":"1","key":"2022040819223555600_bib26","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1002\/asi.22784","article-title":"Bias in peer review","volume":"64","author":"Lee","year":"2013","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"6348","key":"2022040819223555600_bib27","doi-asserted-by":"publisher","first-page":"256","DOI":"10.1126\/science.aan4141","article-title":"Promote scientific integrity via journal peer review data","volume":"357","author":"Lee","year":"2017","journal-title":"Science"},{"key":"2022040819223555600_bib28","article-title":"Sentiment analysis and subjectivity","volume-title":"Handbook of Natural Language Processing","author":"Liu","year":"2010","edition":"2nd ed."},{"key":"2022040819223555600_bib29","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1007\/s11192-017-2279-0","article-title":"What we can learn from tweets linking to research papers","volume":"111","author":"Liu","year":"2017","journal-title":"Scientometrics"},{"key":"2022040819223555600_bib30","article-title":"Textblob: Simplified text processing. Secondary TextBlob: Simplified Text Processing","author":"Loria","year":"2014"},{"key":"2022040819223555600_bib31","doi-asserted-by":"crossref","DOI":"10.3115\/1118108.1118117","article-title":"NLTK: The natural language toolkit","author":"Loper","year":"2002"},{"key":"2022040819223555600_bib32","article-title":"Peer reviews\u2019 prediction in proposals\u2019 funding success: A sentiment analysis of grant reviews at Science Foundation Ireland","author":"Luo","year":"2020"},{"key":"2022040819223555600_bib33","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/reseval\/rvaa022","article-title":"How to evaluate ex ante impact of funding proposals? An analysis of reviewers\u2019 comments on impact statements","volume":"rvaa022","author":"Ma","year":"2020","journal-title":"Research Evaluation"},{"key":"2022040819223555600_bib34","doi-asserted-by":"publisher","first-page":"573","DOI":"10.1177\/0162243908329381","article-title":"Fairness as appropriateness: Negotiating epistemological differences in peer review","volume":"34","author":"Mallard","year":"2009","journal-title":"Science Technology & Human Values"},{"issue":"20","key":"2022040819223555600_bib35","doi-asserted-by":"publisher","first-page":"7176","DOI":"10.1073\/pnas.1319946111","article-title":"Use (and abuse) of expert elicitation in support of decision making for public policy","volume":"111","author":"Morgan","year":"2014","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"2","key":"2022040819223555600_bib36","doi-asserted-by":"publisher","first-page":"79","DOI":"10.3152\/095820207X223785","article-title":"Examining the value added by committee discussion in the review of applications for research awards","volume":"16","author":"Obrecht","year":"2007","journal-title":"Research Evaluation"},{"issue":"12","key":"2022040819223555600_bib37","doi-asserted-by":"publisher","first-page":"2952","DOI":"10.1073\/pnas.1714379115","article-title":"Low agreement among reviewers evaluating the same NIH grant applications","volume":"115","author":"Pier","year":"2018","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"1","key":"2022040819223555600_bib38","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/reseval\/rvw025","article-title":"\u2018Your comments are meaner than your score\u2019: Score calibration talk influences intra- and inter-panel variability during scientific grant peer review","volume":"26","author":"Pier","year":"2017","journal-title":"Research Evaluation"},{"key":"2022040819223555600_bib39","doi-asserted-by":"publisher","first-page":"e59338","DOI":"10.7554\/eLife.59338","article-title":"A retrospective analysis of the peer review of more than 75,000 Marie Curie proposals between 2007 and 2018","volume":"10","author":"Pina","year":"2021","journal-title":"eLife"},{"issue":"5","key":"2022040819223555600_bib40","doi-asserted-by":"publisher","first-page":"317","DOI":"10.3152\/095820210X12809191250843","article-title":"Peer review practices: A content analysis of external reviews in science funding","volume":"19","author":"Reinhart","year":"2010","journal-title":"Research Evaluation"},{"issue":"4","key":"2022040819223555600_bib41","doi-asserted-by":"publisher","first-page":"e18680","DOI":"10.1371\/journal.pone.0018680","article-title":"Peer review, program officers and science funding","volume":"6","author":"Roebber","year":"2011","journal-title":"PLOS ONE"},{"issue":"1","key":"2022040819223555600_bib42","doi-asserted-by":"publisher","DOI":"10.3224\/dms.v13i1.10","article-title":"Die Regierung der Wissenschaft im Peer Review\/Governing Science Through Peer Review, dms\u2013der moderne staat\u2013Zeitschrift f\u00fcr Public Policy","volume":"13","author":"Schendzielorz","year":"2020","journal-title":"Recht und Management"},{"key":"2022040819223555600_bib43","article-title":"Transparency and accountability in research funding bodies: An open data lacuna in science policy","author":"Shankar","year":"2020"},{"issue":"1","key":"2022040819223555600_bib44","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1007\/s11192-017-2518-4","article-title":"Scientometrics of peer review","volume":"113","author":"Squazzoni","year":"2017","journal-title":"Scientometrics"},{"issue":"7796","key":"2022040819223555600_bib45","doi-asserted-by":"publisher","first-page":"512","DOI":"10.1038\/d41586-020-00500-y","article-title":"Unlock ways to share data on peer review","volume":"578","author":"Squazzoni","year":"2020","journal-title":"Nature"},{"key":"2022040819223555600_bib46","doi-asserted-by":"publisher","first-page":"417","DOI":"10.3115\/1073083.1073153","article-title":"Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews","volume-title":"ACL \u201902: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics","author":"Turney","year":"2002"},{"key":"2022040819223555600_bib47","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1007\/s11192-018-2848-x","article-title":"Studying grant decision-making: A linguistic analysis of review reports","volume":"117","author":"Van den Besselaar","year":"2018","journal-title":"Scientometrics"},{"issue":"2","key":"2022040819223555600_bib48","doi-asserted-by":"publisher","first-page":"664","DOI":"10.1162\/qss_a_00040","article-title":"The relationship between journal citation impact and citation sentiment: A study of 32 million citances in PubMed Central","volume":"1","author":"Yan","year":"2020","journal-title":"Quantitative Science Studies"},{"key":"2022040819223555600_bib49","doi-asserted-by":"publisher","first-page":"1490","DOI":"10.1002\/asi.22850","article-title":"Citation content analysis (CCA): A framework for syntactic and semantic analysis of citation content","volume":"64","author":"Zhang","year":"2013","journal-title":"Journal of the American Society for Information Science and Technology"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/2\/4\/1271\/2007963\/qss_a_00156.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/2\/4\/1271\/2007963\/qss_a_00156.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,10]],"date-time":"2023-01-10T16:03:51Z","timestamp":1673366631000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/2\/4\/1271\/107503\/Analyzing-sentiments-in-peer-review-reports"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":48,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,12,1]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00156","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00156\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00156\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00156\/v2\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00156\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00156\/v3\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00156\/v3\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00156\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00156\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00156\/v1\/review2","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}