{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T07:34:51Z","timestamp":1768548891593,"version":"3.49.0"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2022,4,9]],"date-time":"2022-04-09T00:00:00Z","timestamp":1649462400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,9]],"date-time":"2022-04-09T00:00:00Z","timestamp":1649462400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Deutsches Zentrum f\u00fcr Hochschul- und Wissenschaftsforschung\u00a0GmbH (DZHW)"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Scientometrics"],"published-print":{"date-parts":[[2022,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The primary aims of peer review are to detect flaws and deficiencies in the design and interpretation of studies, and ensure the clarity and quality of their presentation. However, it has been questioned whether peer review fulfils this function. Studies have highlighted a stronger focus of reviewers on critiquing methodological aspects of studies and the quality of writing in biomedical sciences, with less focus on theoretical grounding. In contrast, reviewers in the social sciences appear more concerned with theoretical underpinnings. These studies also found the effect of peer review on manuscripts\u2019 content to be variable, but generally modest and positive. I qualitatively analysed 1430 peer reviewers\u2019 comments for a sample of 40 social science preprint-publication pairs to identify the key foci of reviewers\u2019 comments. I then quantified the effect of peer review on manuscripts by examining differences between the preprint and published versions using the normalised Levenshtein distance, cosine similarity, and word count ratios for titles, abstracts, document sections and full-texts. I also examined changes in references used between versions and linked changes to reviewers\u2019 comments. Reviewers\u2019 comments were nearly equally split between issues of methodology (30.7%), theory (30.0%), and writing quality (29.2%). Titles, abstracts, and the semantic content of documents remained similar, although publications were typically longer than preprints. Two-thirds of citations were unchanged, 20.9% were added during review and 13.1% were removed. These findings indicate reviewers equally attended to the theoretical and methodological details and communication style of manuscripts, although the effect on quantitative measures of the manuscripts was limited.<\/jats:p>","DOI":"10.1007\/s11192-022-04357-y","type":"journal-article","created":{"date-parts":[[2022,4,9]],"date-time":"2022-04-09T03:28:56Z","timestamp":1649474936000},"page":"3413-3435","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Peer reviewers equally critique theory, method, and writing, with limited effect on the final content of accepted manuscripts"],"prefix":"10.1007","volume":"127","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7787-6081","authenticated-orcid":false,"given":"Dimity","family":"Stephen","sequence":"first","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,4,9]]},"reference":[{"key":"4357_CR1","doi-asserted-by":"publisher","unstructured":"Akbaritabar, A., Stephen, D., & Squazonni, F. (2022). A study of referencing changes in preprint-publication pairs across multiple fields. Journal of Informetrics, 16(2), 101258. https:\/\/doi.org\/10.1016\/j.joi.2022.101258.","DOI":"10.1016\/j.joi.2022.101258"},{"issue":"5","key":"4357_CR2","doi-asserted-by":"publisher","first-page":"631","DOI":"10.2307\/2095599","volume":"52","author":"V Bakanic","year":"1987","unstructured":"Bakanic, V., McPhail, C., & Simon, R. (1987). The manuscript review and decision-making process. American Sociological Review, 52(5), 631\u2013642.","journal-title":"American Sociological Review"},{"issue":"9","key":"4357_CR3","doi-asserted-by":"publisher","first-page":"889","DOI":"10.1097\/00001888-200109000-00010","volume":"76","author":"G Bordage","year":"2001","unstructured":"Bordage, G. (2001). Reasons reviewers reject and accept manuscripts. Academic Medicine, 76(9), 889\u2013896. https:\/\/doi.org\/10.1097\/00001888-200109000-00010","journal-title":"Academic Medicine"},{"issue":"1","key":"4357_CR4","doi-asserted-by":"publisher","first-page":"e40","DOI":"10.1371\/journal.pmed.0040040","volume":"4","author":"ML Callaham","year":"2007","unstructured":"Callaham, M. L., & Tercier, J. (2007). The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLOS Medicine, 4(1), e40. https:\/\/doi.org\/10.1371\/journal.pmed.0040040","journal-title":"PLOS Medicine"},{"key":"4357_CR5","doi-asserted-by":"publisher","DOI":"10.1186\/s41073-020-00101-3","author":"CFD Carneiro","year":"2020","unstructured":"Carneiro, C. F. D., Queiroz, V. G. S., Moulin, T. C., Carvalho, C. A. M., Haas, C. B., Ray\u00eae, D., Henshall, D. E., De-Souza, E. A., Amorim, F. E., Boos, F. Z., & Guercio, G. D. (2020). Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature. Research Integrity and Peer Review. https:\/\/doi.org\/10.1186\/s41073-020-00101-3","journal-title":"Research Integrity and Peer Review"},{"issue":"6","key":"4357_CR6","doi-asserted-by":"publisher","first-page":"270","DOI":"10.1577\/1548-8446-34.6.270","volume":"34","author":"DR De Vries","year":"2009","unstructured":"De Vries, D. R., Marschall, E. A., & Stein, R. A. (2009). Exploring the peer review process: What is it, does it work, and can it be improved? Fisheries, 34(6), 270\u2013279.","journal-title":"Fisheries"},{"issue":"2","key":"4357_CR7","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1177\/1558689813495111","volume":"8","author":"A Fakis","year":"2014","unstructured":"Fakis, A., Hilliam, R., Stoneley, H., & Townend, M. (2014). Quantitative analysis of qualitative information from interviews: A systematic literature review. Journal of Mixed Methods Research, 8(2), 139\u2013161. https:\/\/doi.org\/10.1177\/1558689813495111","journal-title":"Journal of Mixed Methods Research"},{"issue":"4","key":"4357_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pbio.3000959","volume":"19","author":"N Fraser","year":"2021","unstructured":"Fraser, N., Brierley, L., Dey, G., Polka, J. K., P\u00e1lfy, M., Nanni, F., & Coates, J. A. (2021). The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLOS Biology, 19(4), 1\u201328. https:\/\/doi.org\/10.1371\/journal.pbio.3000959","journal-title":"PLOS Biology"},{"issue":"9","key":"4357_CR9","doi-asserted-by":"publisher","first-page":"743","DOI":"10.1093\/embo-reports\/kve188","volume":"2","author":"F Gannon","year":"2001","unstructured":"Gannon, F. (2001). The essential role of peer review. EMBO Reports, 2(9), 743. https:\/\/doi.org\/10.1093\/embo-reports\/kve188","journal-title":"EMBO Reports"},{"key":"4357_CR10","doi-asserted-by":"publisher","DOI":"10.1186\/s41073-020-00096-x","author":"GT Gerwing","year":"2020","unstructured":"Gerwing, G. T., & Rash, J. (2020). Quantifying professionalism in peer review. Research Integrity and Peer Review. https:\/\/doi.org\/10.1186\/s41073-020-00096-x","journal-title":"Research Integrity and Peer Review"},{"issue":"1","key":"4357_CR11","doi-asserted-by":"publisher","first-page":"11","DOI":"10.7326\/0003-4819-121-1-199407010-00003","volume":"121","author":"SN Goodman","year":"1994","unstructured":"Goodman, S. N., Berlin, J., Fletcher, S. W., & Fletcher, R. H. (1994). Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Annals of Internal Medicine, 121(1), 11\u201321. https:\/\/doi.org\/10.7326\/0003-4819-121-1-199407010-00003","journal-title":"Annals of Internal Medicine"},{"issue":"1","key":"4357_CR12","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1016\/j.outlook.2008.05.006","volume":"57","author":"SJ Henly","year":"2009","unstructured":"Henly, S. J., & Dougherty, M. C. (2009). Quality of manuscript reviews in nursing research. Nursing Outlook, 57(1), 18\u201326. https:\/\/doi.org\/10.1016\/j.outlook.2008.05.006","journal-title":"Nursing Outlook"},{"key":"4357_CR13","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-020-01005-y","author":"OR Herber","year":"2020","unstructured":"Herber, O. R., Bradbury-Jones, C., B\u00f6ling, S., Combes, S., Hirt, J., Koop, Y., Nyhagen, R., Veldhuizen, J. D., & Taylor, J. (2020). What feedback do reviewers give when reviewing qualitative manuscripts? A focused mapping review and synthesis. BMC Medical Research Methodology. https:\/\/doi.org\/10.1186\/s12874-020-01005-y","journal-title":"BMC Medical Research Methodology"},{"key":"4357_CR14","doi-asserted-by":"publisher","DOI":"10.1093\/reseval\/rvaa037","author":"S Horbach","year":"2021","unstructured":"Horbach, S. (2021). No time for that now! Qualitative changes in manuscript peer review during the COVID-19 pandemic. Research Evaluation. https:\/\/doi.org\/10.1093\/reseval\/rvaa037","journal-title":"Research Evaluation"},{"key":"4357_CR15","doi-asserted-by":"publisher","DOI":"10.1186\/s41073-018-0051-5","author":"S Horbach","year":"2018","unstructured":"Horbach, S., & Halffman, W. (2018). The changing forms and expectations of peer review. Research Integrity and Peer Review. https:\/\/doi.org\/10.1186\/s41073-018-0051-5","journal-title":"Research Integrity and Peer Review"},{"key":"4357_CR16","doi-asserted-by":"publisher","DOI":"10.1001\/jama.1994.03520020022005","author":"JP Kassirer","year":"1994","unstructured":"Kassirer, J. P., & Campion, E. W. (1994). Peer review: Crude and understudied, but indispensible. JAMA. https:\/\/doi.org\/10.1001\/jama.1994.03520020022005","journal-title":"JAMA"},{"issue":"4","key":"4357_CR17","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1007\/s00799-018-0234-1","volume":"20","author":"M Klein","year":"2019","unstructured":"Klein, M., Broadwell, P., Farb, S. E., & Grappone, T. (2019). Comparing published scientific journal articles to their pre-print versions. International Journal on Digital Libraries, 20(4), 335\u2013350. https:\/\/doi.org\/10.1007\/s00799-018-0234-1","journal-title":"International Journal on Digital Libraries"},{"issue":"6","key":"4357_CR18","doi-asserted-by":"publisher","first-page":"1731","DOI":"10.2214\/ajr.184.6.01841731","volume":"184","author":"MA Kliewer","year":"2005","unstructured":"Kliewer, M. A., Freed, K. S., DeLong, D. M., Pickhardt, P. J., & Provenzale, J. M. (2005). Reviewing the reviewers: Comparison of review quality and reviewer characteristics at the American Journal of Roentgenology. American Journal of Roentgenology, 184(6), 1731\u20131735. https:\/\/doi.org\/10.2214\/ajr.184.6.01841731","journal-title":"American Journal of Roentgenology"},{"issue":"8","key":"4357_CR19","first-page":"707","volume":"10","author":"VI Levenshtein","year":"1966","unstructured":"Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 10(8), 707\u2013710.","journal-title":"Soviet Physics Doklady"},{"issue":"9040","key":"4357_CR20","doi-asserted-by":"publisher","first-page":"1480","DOI":"10.1016\/S0140-6736(96)05016-7","volume":"348","author":"J-P Pierie","year":"1996","unstructured":"Pierie, J.-P., Walvoort, H., & Overbeke, A. J. (1996). Readers\u2019 evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift Voor Geneeskunde. Lancet, 348(9040), 1480\u20131483. https:\/\/doi.org\/10.1016\/S0140-6736(96)05016-7","journal-title":"Lancet"},{"key":"4357_CR21","unstructured":"R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Retrieved July 21, 2021, from https:\/\/www.R-project.org\/."},{"issue":"2","key":"4357_CR22","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1001\/jama.1994.03520020045012","volume":"272","author":"JC Roberts","year":"1994","unstructured":"Roberts, J. C., Fletcher, R. H., & Fletcher, S. W. (1994). Effects of peer review and editing on the readability of articles published in annals of internal medicine. JAMA, 272(2), 119\u2013121. https:\/\/doi.org\/10.1001\/jama.1994.03520020045012","journal-title":"JAMA"},{"issue":"2","key":"4357_CR23","doi-asserted-by":"publisher","first-page":"180","DOI":"10.3138\/jsp.47.2.180","volume":"47","author":"O Sabaj Meruane","year":"2016","unstructured":"Sabaj Meruane, O., Gonz\u00e1lez Vergara, C., & Pi\u00f1a-Stranger, \u00c1. (2016). What we still don\u2019t know about peer review. Journal of Scholarly Publishing, 47(2), 180\u2013212. https:\/\/doi.org\/10.3138\/jsp.47.2.180","journal-title":"Journal of Scholarly Publishing"},{"key":"4357_CR24","unstructured":"Signorell, A. (2021). DescTools: Tools for descriptive statistics. Retrieved July 21, 2021, from https:\/\/cran.r-project.org\/package=DescTools."},{"key":"4357_CR25","doi-asserted-by":"crossref","unstructured":"Solomon, D. (2002). Talking past each other: Making sense of the debate over electronic publication. First Monday, 7(8). Retrieved June 25, 2021, from http:\/\/firstmonday.org\/issues\/issue7_8\/solomon\/index.html.","DOI":"10.5210\/fm.v7i8.978"},{"issue":"1","key":"4357_CR26","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1177\/0735275115572152","volume":"33","author":"D Strang","year":"2015","unstructured":"Strang, D., & Siler, K. (2015). Revising as reframing: Original submissions versus published papers in administrative science quarterly, 2005 to 2009. Sociological Theory, 33(1), 71\u201396. https:\/\/doi.org\/10.1177\/0735275115572152","journal-title":"Sociological Theory"},{"issue":"16","key":"4357_CR27","doi-asserted-by":"publisher","first-page":"E489","DOI":"10.1503\/cmaj.170901","volume":"190","author":"R Tamblyn","year":"2018","unstructured":"Tamblyn, R., Girard, N., Qian, C. J., & Hanley, J. (2018). Assessment of potential bias in research grant peer review in Canada. CMAJ, 190(16), E489\u2013E499. https:\/\/doi.org\/10.1503\/cmaj.170901","journal-title":"CMAJ"},{"issue":"2\u20133","key":"4357_CR28","doi-asserted-by":"publisher","first-page":"264","DOI":"10.1007\/s12108-015-9288-3","volume":"47","author":"M Teplitskiy","year":"2016","unstructured":"Teplitskiy, M. (2016). Frame search and re-search: How quantitative sociological articles change during peer review. The American Sociologist, 47(2\u20133), 264\u2013288. https:\/\/doi.org\/10.1007\/s12108-015-9288-3","journal-title":"The American Sociologist"},{"key":"4357_CR29","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1007\/BF03086243","volume":"17","author":"EE van der Wall","year":"2009","unstructured":"van der Wall, E. E. (2009). Peer review under review: Room for improvement? Netherlands Heart Journal, 17, 187. https:\/\/doi.org\/10.1007\/BF03086243","journal-title":"Netherlands Heart Journal"},{"key":"4357_CR30","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2015-007961","author":"M van Lent","year":"2015","unstructured":"van Lent, M., IntHout, J., & Jan Out, H. (2015). Peer review comments on drug trials submitted to medical journals differ depending on sponsorship, results and acceptance: A retrospective cohort study. BMJ Open. https:\/\/doi.org\/10.1136\/bmjopen-2015-007961","journal-title":"BMJ Open"},{"key":"4357_CR31","doi-asserted-by":"crossref","unstructured":"Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. Springer. Retrieved July 21, 2021, from https:\/\/ggplot2.tidyverse.org.","DOI":"10.1007\/978-3-319-24277-4"},{"issue":"43","key":"4357_CR32","doi-asserted-by":"publisher","first-page":"1686","DOI":"10.21105\/joss.01686","volume":"4","author":"H Wickham","year":"2019","unstructured":"Wickham, H., Averick, M., Bryan, J., Chang, W., D\u2019Agostino McGowa, nL., Francois, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Bache, S. M., Muller, K., Ooms, J., Robinson, D., Seidel, D. P., Spinu, V., \u2026 Yutani, H. (2019). Welcome to the tidyverse. Journal of Open Source Software, 4(43), 1686. https:\/\/doi.org\/10.21105\/joss.01686","journal-title":"Journal of Open Source Software"},{"key":"4357_CR33","unstructured":"Wild, F. (2020). lsa: Latent semantic analysis. https:\/\/CRAN.R-project.org\/package=lsa"}],"container-title":["Scientometrics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-022-04357-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11192-022-04357-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11192-022-04357-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,15]],"date-time":"2022-06-15T08:38:12Z","timestamp":1655282292000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11192-022-04357-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,9]]},"references-count":33,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,6]]}},"alternative-id":["4357"],"URL":"https:\/\/doi.org\/10.1007\/s11192-022-04357-y","relation":{},"ISSN":["0138-9130","1588-2861"],"issn-type":[{"value":"0138-9130","type":"print"},{"value":"1588-2861","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,9]]},"assertion":[{"value":"15 October 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 March 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 April 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The author has no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}