{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T02:31:12Z","timestamp":1773109872964,"version":"3.50.1"},"reference-count":69,"publisher":"MIT Press","issue":"3","license":[{"start":{"date-parts":[[2022,8,25]],"date-time":"2022-08-25T00:00:00Z","timestamp":1661385600000},"content-version":"vor","delay-in-days":236,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001602","name":"Science Foundation Ireland","doi-asserted-by":"publisher","award":["17\/SPR\/5319"],"award-info":[{"award-number":["17\/SPR\/5319"]}],"id":[{"id":"10.13039\/501100001602","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>One of the main critiques of academic peer review is that interrater reliability (IRR) among reviewers is low. We examine an underinvestigated factor possibly contributing to low IRR: reviewers\u2019 diversity in their topic-criteria mapping (\u201cTC-mapping\u201d). It refers to differences among reviewers pertaining to which topics they choose to emphasize in their evaluations, and how they map those topics onto various evaluation criteria. In this paper we look at the review process of grant proposals in one funding agency to ask: How much do reviewers differ in TC-mapping, and do their differences contribute to low IRR? Through a content analysis of review forms submitted to a national funding agency (Science Foundation Ireland) and a survey of its reviewers, we find evidence of interreviewer differences in their TC-mapping. Using a simulation experiment we show that, under a wide range of conditions, even strong differences in TC-mapping have only a negligible impact on IRR. Although further empirical work is needed to corroborate simulation results, these tentatively suggest that reviewers\u2019 heterogeneous TC-mappings might not be of concern for designers of peer review panels to safeguard IRR.<\/jats:p>","DOI":"10.1162\/qss_a_00207","type":"journal-article","created":{"date-parts":[[2022,8,25]],"date-time":"2022-08-25T16:55:00Z","timestamp":1661446500000},"page":"832-856","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":6,"title":["Peer reviewer topic choice and its impact on interrater reliability: A mixed-method study"],"prefix":"10.1162","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4977-0877","authenticated-orcid":true,"given":"Thomas","family":"Feliciani","sequence":"first","affiliation":[{"name":"School of Sociology and Geary Institute for Public Policy, University College Dublin, Dublin, Ireland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3347-3982","authenticated-orcid":true,"given":"Junwen","family":"Luo","sequence":"additional","affiliation":[{"name":"School of Information and Communication Studies and Geary Institute for Public Policy, University College Dublin, Dublin, Ireland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8788-466X","authenticated-orcid":true,"given":"Kalpana","family":"Shankar","sequence":"additional","affiliation":[{"name":"School of Information and Communication Studies and Geary Institute for Public Policy, University College Dublin, Dublin, Ireland"}]}],"member":"281","published-online":{"date-parts":[[2022,11,1]]},"reference":[{"issue":"9","key":"2022111422091128400_bib1","doi-asserted-by":"publisher","first-page":"e46054","DOI":"10.1371\/journal.pone.0046054","article-title":"Peer review of grant applications: criteria used and qualitative study of reviewer practices","volume":"7","author":"Abdoul","year":"2012","journal-title":"PLOS ONE"},{"issue":"1","key":"2022111422091128400_bib2","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1017\/S0140525X00065705","article-title":"Reliability, fairness, objectivity and other inappropriate goals in peer review","volume":"14","author":"Bailar","year":"1991","journal-title":"Behavioral and Brain Sciences"},{"issue":"1","key":"2022111422091128400_bib3","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1002\/aris.2011.1440450112","article-title":"Scientific peer review","volume":"45","author":"Bornmann","year":"2011","journal-title":"Annual Review of Information Science and Technology"},{"issue":"2","key":"2022111422091128400_bib4","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1002\/asi.22803","article-title":"What is societal impact of research and how can it be assessed? A literature survey","volume":"64","author":"Bornmann","year":"2013","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"12","key":"2022111422091128400_bib5","doi-asserted-by":"publisher","first-page":"e14331","DOI":"10.1371\/journal.pone.0014331","article-title":"A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants","volume":"5","author":"Bornmann","year":"2010","journal-title":"PLOS ONE"},{"issue":"3\u20134","key":"2022111422091128400_bib6","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1080\/02691720903364019","article-title":"Broad impacts and narrow perspectives: Passing the buck on science and social impacts","volume":"23","author":"Bozeman","year":"2009","journal-title":"Social Epistemology"},{"issue":"1","key":"2022111422091128400_bib7","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1007\/s11192-020-03348-1","article-title":"Arbitrariness in the peer review process","volume":"123","author":"Brezis","year":"2020","journal-title":"Scientometrics"},{"issue":"4","key":"2022111422091128400_bib8","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1007\/s10588-005-5586-6","article-title":"Some simple algorithms for structural comparison","volume":"11","author":"Butts","year":"2005","journal-title":"Computational and Mathematical Organization Theory"},{"issue":"1","key":"2022111422091128400_bib9","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1017\/S0140525X00065675","article-title":"The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation","volume":"14","author":"Cicchetti","year":"1991","journal-title":"Behavioral and Brain Sciences"},{"issue":"1","key":"2022111422091128400_bib10","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1093\/reseval\/rvy036","article-title":"Who benefits from ex ante societal impact evaluation in the European funding arena? A cross-country comparison of societal impact capacity in the social sciences and humanities","volume":"29","author":"de Jong","year":"2020","journal-title":"Research Evaluation"},{"issue":"11","key":"2022111422091128400_bib11","doi-asserted-by":"publisher","first-page":"2147","DOI":"10.1016\/j.pec.2017.06.007","article-title":"Peer-review for selection of oral presentations for conferences: Are we reliable?","volume":"100","author":"Deveugele","year":"2017","journal-title":"Patient Education and Counseling"},{"issue":"3","key":"2022111422091128400_bib12","doi-asserted-by":"publisher","first-page":"904","DOI":"10.1111\/rssa.12681","article-title":"When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review","volume":"184","author":"Erosheva","year":"2021","journal-title":"Journal of the Royal Statistical Society: Series A (Statistics in Society)"},{"issue":"1","key":"2022111422091128400_bib13","doi-asserted-by":"publisher","first-page":"555","DOI":"10.1007\/s11192-019-03205-w","article-title":"A scoping review of simulation models of peer review","volume":"121","author":"Feliciani","year":"2019","journal-title":"Scientometrics"},{"issue":"3","key":"2022111422091128400_bib14","doi-asserted-by":"publisher","first-page":"8","DOI":"10.18564\/jasss.4284","article-title":"Grade language heterogeneity in simulation models of peer review","volume":"23","author":"Feliciani","year":"2020","journal-title":"Journal of Artificial Societies and Social Simulation"},{"issue":"4","key":"2022111422091128400_bib15","doi-asserted-by":"publisher","first-page":"104467","DOI":"10.1016\/j.respol.2021.104467","article-title":"Designing grant-review panels for better funding decisions: Lessons from an empirically calibrated simulation model","volume":"51","author":"Feliciani","year":"2022","journal-title":"Research Policy"},{"key":"2022111422091128400_bib16","doi-asserted-by":"publisher","first-page":"723","DOI":"10.1037\/10109-048","article-title":"But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments","volume-title":"Methodological issues & strategies in clinical research","author":"Fiske","year":"1992"},{"key":"2022111422091128400_bib17","doi-asserted-by":"publisher","first-page":"103894","DOI":"10.1016\/j.compedu.2020.103894","article-title":"Reviewing and analyzing peer review Inter-Rater Reliability in a MOOC platform","volume":"154","author":"Garcia-Loro","year":"2020","journal-title":"Computers & Education"},{"issue":"1","key":"2022111422091128400_bib18","doi-asserted-by":"publisher","DOI":"10.13130\/2282-5398\/3834","article-title":"Selecting applications for funding: Why random choice is better than peer review","volume":"2","author":"Gillies","year":"2014","journal-title":"RT. A Journal on Research Policy and Evaluation"},{"issue":"1","key":"2022111422091128400_bib19","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1177\/016327879401700101","article-title":"The reliability of peer assessments: A meta-analysis","volume":"17","author":"Goldman","year":"1994","journal-title":"Evaluation & the Health Professions"},{"issue":"9103","key":"2022111422091128400_bib20","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1016\/S0140-6736(05)78485-3","article-title":"Chance and grants","volume":"351","author":"Greenberg","year":"1998","journal-title":"The Lancet"},{"key":"2022111422091128400_bib21","doi-asserted-by":"publisher","first-page":"1335","DOI":"10.12688\/f1000research.11917.2","article-title":"What do we know about grant peer review in the health sciences?","volume":"6","author":"Guthrie","year":"2018","journal-title":"F1000Research"},{"issue":"7","key":"2022111422091128400_bib22","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1002\/j.2326-1951.1979.tb01767.x","article-title":"Creative disagreement","volume":"19","author":"Harnad","year":"1979","journal-title":"The Sciences"},{"issue":"1","key":"2022111422091128400_bib23","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1057\/s41599-020-0412-9","article-title":"Criteria for assessing grant applications: A systematic review","volume":"6","author":"Hug","year":"2020","journal-title":"Palgrave Communications"},{"issue":"1","key":"2022111422091128400_bib24","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1093\/reseval\/rvab034","article-title":"Do peers share the same criteria for assessing grant applications?","volume":"31","author":"Hug","year":"2022","journal-title":"Research Evaluation"},{"issue":"3","key":"2022111422091128400_bib25","doi-asserted-by":"publisher","first-page":"591","DOI":"10.1007\/s11192-006-0171-4","article-title":"A new reader trial approach to peer review in funding research grants: An Australian experiment","volume":"69","author":"Jayasinghe","year":"2006","journal-title":"Scientometrics"},{"key":"2022111422091128400_bib26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/03623319.2020.1728506","article-title":"Are peer-reviews of grant proposals reliable? An analysis of Economic and Social Research Council (ESRC) funding applications","author":"Jerrim","year":"2020","journal-title":"The Social Science Journal"},{"issue":"2","key":"2022111422091128400_bib27","doi-asserted-by":"publisher","first-page":"1059","DOI":"10.1007\/s11192-017-2516-6","article-title":"Inter-rater reliability and validity of peer reviews in an interdisciplinary field","volume":"113","author":"Jirschitzka","year":"2017","journal-title":"Scientometrics"},{"issue":"2","key":"2022111422091128400_bib28","doi-asserted-by":"publisher","first-page":"155","DOI":"10.1016\/j.jcm.2016.02.012","article-title":"A guideline of selecting and reporting intraclass correlation coefficients for reliability research","volume":"15","author":"Koo","year":"2016","journal-title":"Journal of Chiropractic Medicine"},{"key":"2022111422091128400_bib29","doi-asserted-by":"publisher","DOI":"10.4159\/9780674054158","volume-title":"How professors think: Inside the curious world of academic judgment","author":"Lamont","year":"2010"},{"issue":"6","key":"2022111422091128400_bib30","doi-asserted-by":"publisher","first-page":"820","DOI":"10.1177\/030631201031006002","article-title":"The decision-making constraints and processes of grant peer review, and their effects on the review outcome","volume":"31","author":"Langfeldt","year":"2001","journal-title":"Social Studies of Science"},{"key":"2022111422091128400_bib69","volume-title":"Efficiency and flexibility in research funding. A comparative study of funding instruments and review criteria","author":"Langfeldt","year":"2016"},{"issue":"4","key":"2022111422091128400_bib31","doi-asserted-by":"publisher","first-page":"815","DOI":"10.1177\/1094428106296642","article-title":"Answers to 20 questions about interrater reliability and interrater agreement","volume":"11","author":"LeBreton","year":"2008","journal-title":"Organizational Research Methods"},{"issue":"5","key":"2022111422091128400_bib32","doi-asserted-by":"publisher","first-page":"1272","DOI":"10.1086\/683652","article-title":"Commensuration bias in peer review","volume":"82","author":"Lee","year":"2015","journal-title":"Philosophy of Science"},{"issue":"1","key":"2022111422091128400_bib33","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1002\/asi.22784","article-title":"Bias in peer review","volume":"64","author":"Lee","year":"2013","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"4","key":"2022111422091128400_bib34","doi-asserted-by":"publisher","first-page":"431","DOI":"10.1093\/reseval\/rvaa022","article-title":"How to evaluate ex ante impact of funding proposals? An analysis of reviewers\u2019 comments on impact statements","volume":"29","author":"Ma","year":"2020","journal-title":"Research Evaluation"},{"issue":"5","key":"2022111422091128400_bib35","doi-asserted-by":"publisher","first-page":"573","DOI":"10.1177\/0162243908329381","article-title":"Fairness as appropriateness: Negotiating epistemological differences in peer review","volume":"34","author":"Mallard","year":"2009","journal-title":"Science, Technology, & Human Values"},{"key":"2022111422091128400_bib36","doi-asserted-by":"publisher","first-page":"11","DOI":"10.4135\/9781446294413.n2","article-title":"Social network analysis: An introduction","volume-title":"The SAGE handbook of social network analysis","author":"Marin","year":"2014"},{"issue":"1","key":"2022111422091128400_bib37","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1080\/00050060600823275","article-title":"Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid","volume":"42","author":"Marsh","year":"2007","journal-title":"Australian Psychologist"},{"issue":"3","key":"2022111422091128400_bib38","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1037\/0003-066X.63.3.160","article-title":"Improving the peer-review process for grant applications: Reliability, validity, bias, and generalizability","volume":"63","author":"Marsh","year":"2008","journal-title":"American Psychologist"},{"key":"2022111422091128400_bib39","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2206.03255","volume-title":"Do interests affect grant application success? The role of organizational proximity","author":"Mom","year":"2022"},{"issue":"1","key":"2022111422091128400_bib40","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1186\/1472-6963-2-8","article-title":"Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference","volume":"2","author":"Montgomery","year":"2002","journal-title":"BMC Health Services Research"},{"issue":"20","key":"2022111422091128400_bib41","doi-asserted-by":"publisher","first-page":"7176","DOI":"10.1073\/pnas.1319946111","article-title":"Use (and abuse) of expert elicitation in support of decision making for public policy","volume":"111","author":"Morgan","year":"2014","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"23\u201324","key":"2022111422091128400_bib42","doi-asserted-by":"publisher","first-page":"2465","DOI":"10.1002\/sim.4780132310","article-title":"A critical discussion of intraclass correlation coefficients","volume":"13","author":"M\u00fcller","year":"1994","journal-title":"Statistics in Medicine"},{"issue":"10","key":"2022111422091128400_bib43","doi-asserted-by":"publisher","first-page":"e48509","DOI":"10.1371\/journal.pone.0048509","article-title":"Heterogeneity of inter-rater reliabilities of grant peer reviews and its determinants: A general estimating equations approach","volume":"7","author":"Mutz","year":"2012","journal-title":"PLOS ONE"},{"key":"2022111422091128400_bib44","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1007\/978-3-319-09785-5_7","article-title":"Interrater reliability of the peer review process in management journals","volume-title":"Incentives and Performance","author":"Nicolai","year":"2015"},{"issue":"8","key":"2022111422091128400_bib45","doi-asserted-by":"publisher","first-page":"543","DOI":"10.3152\/030234207X254396","article-title":"Peer review and the relevance gap: Ten suggestions for policy-makers","volume":"34","author":"Nightingale","year":"2007","journal-title":"Science and Public Policy"},{"issue":"2","key":"2022111422091128400_bib46","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1017\/S0140525X00011183","article-title":"Peer-review practices of psychological journals: The fate of published articles, submitted again","volume":"5","author":"Peters","year":"1982","journal-title":"Behavioral and Brain Sciences"},{"issue":"12","key":"2022111422091128400_bib47","doi-asserted-by":"publisher","first-page":"2952","DOI":"10.1073\/pnas.1714379115","article-title":"Low agreement among reviewers evaluating the same NIH grant applications","volume":"115","author":"Pier","year":"2018","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2022111422091128400_bib48","volume-title":"R: A language and environment for statistical computing","author":"R Core Team","year":"2021"},{"issue":"3","key":"2022111422091128400_bib49","doi-asserted-by":"publisher","first-page":"789","DOI":"10.1007\/s11192-008-2220-7","article-title":"Peer review of grant applications in biology and medicine. Reliability, fairness, and validity","volume":"81","author":"Reinhart","year":"2009","journal-title":"Scientometrics"},{"issue":"5","key":"2022111422091128400_bib50","doi-asserted-by":"publisher","first-page":"317","DOI":"10.3152\/095820210X12809191250843","article-title":"Peer review practices: A content analysis of external reviews in science funding","volume":"19","author":"Reinhart","year":"2010","journal-title":"Research Evaluation"},{"issue":"1","key":"2022111422091128400_bib51","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1017\/S0140525X00065936","article-title":"Is unreliability in peer review harmful?","volume":"14","author":"Roediger","year":"1991","journal-title":"Behavioral and Brain Sciences"},{"issue":"5","key":"2022111422091128400_bib52","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1007\/BF02600092","article-title":"How reliable is peer review of scientific abstracts?: Looking back at the 1991 Annual Meeting of the Society of General Internal Medicine","volume":"8","author":"Rubin","year":"1993","journal-title":"Journal of General Internal Medicine"},{"key":"2022111422091128400_bib53","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1155\/2014\/135641","article-title":"Creation of reliable relevance judgments in information retrieval systems evaluation experimentation through crowdsourcing: A review","volume":"2014","author":"Samimi","year":"2014","journal-title":"The Scientific World Journal"},{"issue":"13","key":"2022111422091128400_bib54","doi-asserted-by":"publisher","first-page":"2126","DOI":"10.1002\/asi.20681","article-title":"Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance","volume":"58","author":"Saracevic","year":"2007","journal-title":"Journal of the American Society for Information Science and Technology"},{"issue":"6","key":"2022111422091128400_bib55","doi-asserted-by":"publisher","first-page":"e0130450","DOI":"10.1371\/journal.pone.0130450","article-title":"Grant peer review: Improving inter-rater reliability with training","volume":"10","author":"Sattler","year":"2015","journal-title":"PLOS ONE"},{"key":"2022111422091128400_bib56","volume-title":"SFI Investigators Programme","author":"Science Foundation Ireland","year":"2017"},{"key":"2022111422091128400_bib57","volume-title":"SFI Industry RD&I Fellowship Programme","author":"Science Foundation Ireland","year":"2019"},{"issue":"3","key":"2022111422091128400_bib58","doi-asserted-by":"publisher","first-page":"349","DOI":"10.1093\/reseval\/rvab011","article-title":"Does reviewing experience reduce disagreement in proposals evaluation? Insights from Marie Sk\u0142odowska-Curie and COST Actions","volume":"30","author":"Seeber","year":"2021","journal-title":"Research Evaluation"},{"key":"2022111422091128400_bib59","doi-asserted-by":"publisher","DOI":"10.6084\/M9.FIGSHARE.13651058.V1","volume-title":"SPRING 2020 survey: Peer review of grant proposals","author":"Shankar","year":"2021"},{"issue":"3","key":"2022111422091128400_bib60","doi-asserted-by":"publisher","first-page":"637","DOI":"10.1148\/radiology.178.3.1994394","article-title":"Assassins and zealots: Variations in peer review. Special report","volume":"178","author":"Siegelman","year":"1991","journal-title":"Radiology"},{"issue":"2","key":"2022111422091128400_bib61","doi-asserted-by":"publisher","first-page":"3","DOI":"10.18564\/jasss.2128","article-title":"Opening the black-box of peer review: An agent-based model of scientist behaviour","volume":"16","author":"Squazzoni","year":"2013","journal-title":"Journal of Artificial Societies and Social Simulation"},{"issue":"4","key":"2022111422091128400_bib62","doi-asserted-by":"publisher","first-page":"707","DOI":"10.1140\/epjb\/e2011-20545-7","article-title":"Peer-review in a world with rational scientists: Toward selection of the average","volume":"84","author":"Thurner","year":"2011","journal-title":"European Physical Journal B"},{"key":"2022111422091128400_bib63","volume-title":"Towards Outstanding Research Reviews (TORR)","author":"TORR","year":"2022"},{"issue":"6157","key":"2022111422091128400_bib64","doi-asserted-by":"publisher","first-page":"468","DOI":"10.1126\/science.1240474","article-title":"Atypical combinations and scientific impact","volume":"342","author":"Uzzi","year":"2013","journal-title":"Science"},{"key":"2022111422091128400_bib65","article-title":"Applying social judgment theory to better understand what peer-reviewers pay attention to when evaluating proposals","volume-title":"27th International (Virtual) Meeting of the Brunswik Society","author":"Vall\u00e9e-Tourangeau","year":"2021"},{"issue":"1","key":"2022111422091128400_bib66","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1057\/s41599-022-01050-6","article-title":"Peer reviewers\u2019 dilemmas: A qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences","volume":"9","author":"Vall\u00e9e-Tourangeau","year":"2022","journal-title":"Humanities and Social Sciences Communications"},{"issue":"1","key":"2022111422091128400_bib67","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1007\/s11192-018-2848-x","article-title":"Studying grant decision-making: A linguistic analysis of review reports","volume":"117","author":"van den Besselaar","year":"2018","journal-title":"Scientometrics"},{"issue":"9124","key":"2022111422091128400_bib68","doi-asserted-by":"publisher","first-page":"301","DOI":"10.1016\/S0140-6736(97)11129-1","article-title":"Peer review of grant applications: What do we know?","volume":"352","author":"Wessely","year":"1998","journal-title":"The Lancet"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/3\/3\/832\/2057744\/qss_a_00207.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/qss\/article-pdf\/3\/3\/832\/2057744\/qss_a_00207.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,14]],"date-time":"2022-11-14T17:09:54Z","timestamp":1668445794000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/3\/3\/832\/112792\/Peer-reviewer-topic-choice-and-its-impact-on"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"references-count":69,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,11,1]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00207","relation":{"has-review":[{"id-type":"doi","id":"10.1162\/QSS_A_00207\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00207\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00207\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00207\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00207\/v2\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1162\/QSS_A_00207\/v1\/decision1","asserted-by":"object"}]},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022]]},"published":{"date-parts":[[2022]]}}}