{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T01:54:01Z","timestamp":1790301241692,"version":"4.1.0"},"reference-count":57,"publisher":"MIT Press","issue":"2","license":[{"start":{"date-parts":[[2021,3,8]],"date-time":"2021-03-08T00:00:00Z","timestamp":1615161600000},"content-version":"vor","delay-in-days":66,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000155","name":"Social Sciences and Humanities Research Council of Canada","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000155","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,7,15]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We analyzed a data set of scientific manuscripts that were submitted to various conferences in artificial intelligence. We performed a combination of semantic, lexical, and psycholinguistic analyses of the full text of the manuscripts and compared them with the outcome of the peer review process. We found that accepted manuscripts scored lower than rejected manuscripts on two indicators of readability, and that they also used more scientific and artificial intelligence jargon. We also found that accepted manuscripts were written with words that are less frequent, that are acquired at an older age, and that are more abstract than rejected manuscripts. The analysis of references included in the manuscripts revealed that the subset of accepted submissions were more likely to cite the same publications. This finding was echoed by pairwise comparisons of the word content of the manuscripts (i.e., an indicator of semantic similarity), which were more similar in the subset of accepted manuscripts. Finally, we predicted the peer review outcome of manuscripts with their word content, with words related to machine learning and neural networks positively related to acceptance, whereas words related to logic, symbolic processing, and knowledge-based systems negatively related to acceptance.<\/jats:p>","DOI":"10.1162\/qss_a_00125","type":"journal-article","created":{"date-parts":[[2021,2,23]],"date-time":"2021-02-23T14:40:15Z","timestamp":1614091215000},"page":"662-677","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":20,"title":["Textual analysis of artificial intelligence manuscripts reveals features associated with peer review outcome"],"prefix":"10.1162","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1358-8937","authenticated-orcid":true,"given":"Philippe","family":"Vincent-Lamarre","sequence":"first","affiliation":[{"name":"\u00c9cole de biblioth\u00e9conomie et des sciences de l\u2019information, Universit\u00e9 de Montr\u00e9al, Montr\u00e9al, Canada"},{"name":"\u00c9cole de psychologie, Universit\u00e9 d\u2019Ottawa, Ottawa, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2733-0689","authenticated-orcid":true,"given":"Vincent","family":"Larivi\u00e8re","sequence":"additional","affiliation":[{"name":"\u00c9cole de biblioth\u00e9conomie et des sciences de l\u2019information, Universit\u00e9 de Montr\u00e9al, Montr\u00e9al, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,7,15]]},"reference":[{"issue":"1","key":"2021072016160700800_bib1","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1017\/S0140525X00065705","article-title":"Reliability, fairness, objectivity and other inappropriate goals in peer review","volume":"14","author":"Bailar","year":"1991","journal-title":"Behavioral and Brain Sciences"},{"issue":"12","key":"2021072016160700800_bib2","doi-asserted-by":"crossref","first-page":"e14331","DOI":"10.1371\/journal.pone.0014331","article-title":"A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants","volume":"5","author":"Bornmann","year":"2010","journal-title":"PLOS ONE"},{"issue":"4","key":"2021072016160700800_bib3","doi-asserted-by":"crossref","first-page":"977","DOI":"10.3758\/BRM.41.4.977","article-title":"Moving beyond Kuc\u02c7era and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English","volume":"41","author":"Brysbaert","year":"2009","journal-title":"Behavior Research Methods"},{"issue":"3","key":"2021072016160700800_bib4","doi-asserted-by":"crossref","first-page":"904","DOI":"10.3758\/s13428-013-0403-5","article-title":"Concreteness ratings for 40 thousand generally known English word lemmas","volume":"46","author":"Brysbaert","year":"2014","journal-title":"Behavior Research Methods"},{"issue":"6110","key":"2021072016160700800_bib5","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1126\/science.1227833","article-title":"Flows of research manuscripts among scientific journals reveal hidden submission patterns","volume":"338","author":"Calcagno","year":"2012","journal-title":"Science"},{"key":"2021072016160700800_bib6","volume-title":"Readability revisited: The new Dale-Chall readability formula","author":"Chall","year":"1995"},{"issue":"1","key":"2021072016160700800_bib7","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1017\/S0140525X00065675","article-title":"The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation","volume":"14","author":"Cicchetti","year":"1991","journal-title":"Behavioral and Brain Sciences"},{"issue":"2","key":"2021072016160700800_bib8","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/j.jslw.2009.02.002","article-title":"Computational assessment of lexical differences in L1 and L2 writing","volume":"18","author":"Crossley","year":"2009","journal-title":"Journal of Second Language Writing"},{"issue":"4","key":"2021072016160700800_bib9","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1177\/0265532210378031","article-title":"Predicting lexical proficiency in language learner texts using computational indices","volume":"28","author":"Crossley","year":"2011","journal-title":"Language Testing"},{"key":"2021072016160700800_bib10","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1007\/978-3-319-50627-2_6","article-title":"Preserving the quality of scientific research: Peer review of research articles","volume-title":"Scientific scholarly communication: The changing landscape","author":"De Silva","year":"2017"},{"issue":"3","key":"2021072016160700800_bib11","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1037\/h0057532","article-title":"A new readability yardstick","volume":"32","author":"Flesch","year":"1948","journal-title":"Journal of Applied Psychology"},{"key":"2021072016160700800_bib12","doi-asserted-by":"crossref","DOI":"10.31234\/osf.io\/483zj","article-title":"How many reviewers are required to obtain reliable evaluations of NIH R01 grant proposals?","author":"Forscher","year":"2019"},{"key":"2021072016160700800_bib13","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.cobeha.2018.12.010","article-title":"Reconciling deep learning with symbolic artificial intelligence: Representing objects and relations","volume":"29","author":"Garnelo","year":"2019","journal-title":"Current Opinion in Behavioral Sciences"},{"key":"2021072016160700800_bib14","article-title":"Model-free, model-based, and general intelligence","author":"Geffner","year":"2018","journal-title":"arXiv:1806.02308 [cs]"},{"key":"2021072016160700800_bib15","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1109\/JCDL.2019.00096","article-title":"A sentiment augmented deep architecture to predict peer review outcomes","volume-title":"2019 ACM\/IEEE Joint Conference on Digital Libraries (JCDL)","author":"Ghosal","year":"2019"},{"issue":"1","key":"2021072016160700800_bib16","doi-asserted-by":"crossref","first-page":"11","DOI":"10.7326\/0003-4819-121-1-199407010-00003","article-title":"Manuscript quality before and after peer review and editing at annals of internal medicine","volume":"121","author":"Goodman","year":"1994","journal-title":"Annals of Internal Medicine"},{"key":"2021072016160700800_bib17","article-title":"We analyzed 16,625 papers to figure out where AI is headed next","author":"Hao","year":"2019","journal-title":"MIT Technology Review"},{"issue":"6719","key":"2021072016160700800_bib18","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1038\/17194","article-title":"Language bias discredits the peer-review system","volume":"397","author":"Herrera","year":"1999","journal-title":"Nature"},{"issue":"7","key":"2021072016160700800_bib19","doi-asserted-by":"crossref","first-page":"e22475","DOI":"10.1371\/journal.pone.0022475","article-title":"The validity of peer review in a general medicine journal","volume":"6","author":"Jackson","year":"2011","journal-title":"PLOS ONE"},{"key":"2021072016160700800_bib20","first-page":"7","article-title":"Predicting conference paper acceptance","author":"Jen","year":"2018"},{"issue":"1","key":"2021072016160700800_bib21","article-title":"Artificial intelligence\u2014The revolution hasn\u2019t happened yet","volume":"1","author":"Jordan","year":"2018","journal-title":"Harvard Data Science Review"},{"key":"2021072016160700800_bib22","doi-asserted-by":"crossref","DOI":"10.29007\/x163","article-title":"A dataset of peer reviews (PeerRead): Collection, insights and NLP applications","author":"Kang","year":"2018","journal-title":"arXiv preprint"},{"issue":"4","key":"2021072016160700800_bib23","doi-asserted-by":"crossref","first-page":"27","DOI":"10.15858\/engtea.69.4.201412.27","article-title":"Predicting L2 writing proficiency using linguistic complexity measures: A corpus-based study","volume":"69","author":"Kim","year":"2014","journal-title":"English Teaching"},{"key":"2021072016160700800_bib24","doi-asserted-by":"crossref","DOI":"10.21236\/ADA006655","volume-title":"Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for navy enlisted personnel","author":"Kincaid","year":"1975"},{"key":"2021072016160700800_bib25","article-title":"One of the fathers of AI is worried about its future","author":"Knight","year":"2018","journal-title":"MIT Technology Review"},{"key":"2021072016160700800_bib26","first-page":"1097","article-title":"Imagenet classification with deep convolutional neural networks","volume-title":"Advances in Neural Information Processing Systems","author":"Krizhevsky","year":"2012"},{"issue":"4","key":"2021072016160700800_bib27","doi-asserted-by":"crossref","first-page":"978","DOI":"10.3758\/s13428-012-0210-4","article-title":"Age-of-acquisition ratings for 30,000 English words","volume":"44","author":"Kuperman","year":"2012","journal-title":"Behavior Research Methods"},{"key":"2021072016160700800_bib28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.asw.2017.11.002","article-title":"Analysis of syntactic complexity in secondary education EFL writers at different proficiency levels","volume":"35","author":"Lahuerta Mart\u00ednez","year":"2018","journal-title":"Assessing Writing"},{"issue":"1","key":"2021072016160700800_bib29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.29173\/cjs29548","article-title":"Introduction: The dissemination of national knowledge in an internationalized scientific community","volume":"44","author":"Larivi\u00e8re","year":"2019","journal-title":"Canadian Journal of Sociology"},{"issue":"5","key":"2021072016160700800_bib30","doi-asserted-by":"crossref","first-page":"859","DOI":"10.1086\/667841","article-title":"A Kuhnian critique of psychometric research on peer review","volume":"79","author":"Lee","year":"2012","journal-title":"Philosophy of Science"},{"issue":"1","key":"2021072016160700800_bib31","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1002\/asi.22784","article-title":"Bias in peer review","volume":"64","author":"Lee","year":"2013","journal-title":"Journal of the American Society for Information Science and Technology"},{"key":"2021072016160700800_bib32","article-title":"Deep learning: A critical appraisal","author":"Marcus","year":"2018","journal-title":"arXiv preprint"},{"issue":"2","key":"2021072016160700800_bib33","first-page":"117","article-title":"Peer review: Publication\u2019s gold standard","volume":"3","author":"Mayden","year":"2012","journal-title":"Journal of the Advanced Practitioner in Oncology"},{"key":"2021072016160700800_bib34","doi-asserted-by":"crossref","first-page":"e47338","DOI":"10.7554\/eLife.47338","article-title":"Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations","volume":"8","author":"McKiernan","year":"2019","journal-title":"eLife"},{"issue":"3","key":"2021072016160700800_bib35","doi-asserted-by":"crossref","first-page":"e2004089","DOI":"10.1371\/journal.pbio.2004089","article-title":"Assessing scientists for hiring, promotion, and tenure","volume":"16","author":"Moher","year":"2018","journal-title":"PLOS Biology"},{"issue":"2","key":"2021072016160700800_bib36","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1016\/j.oraloncology.2004.11.001","article-title":"Is peer review in crisis?","volume":"41","author":"Mulligan","year":"2005","journal-title":"Oral Oncology"},{"key":"2021072016160700800_bib37","article-title":"Gender and international diversity improves equity in peer review","author":"Murray","year":"2018","journal-title":"bioRxiv"},{"key":"2021072016160700800_bib57","unstructured":"Nur Najah Radhiah, Z. A. (2018). A longitudinal corpus study of syntactic complexity development in L2 writing (Doctoral dissertation, University of Malaya)."},{"issue":"19","key":"2021072016160700800_bib38","doi-asserted-by":"crossref","first-page":"9566","DOI":"10.1002\/ece3.4467","article-title":"The effectiveness of journals as arbiters of scientific impact","volume":"8","author":"Paine","year":"2018","journal-title":"Ecology and Evolution"},{"issue":"9040","key":"2021072016160700800_bib39","doi-asserted-by":"crossref","first-page":"1480","DOI":"10.1016\/S0140-6736(96)05016-7","article-title":"Readers\u2019 evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde","volume":"348","author":"Pierie","year":"1996","journal-title":"The Lancet"},{"key":"2021072016160700800_bib40","doi-asserted-by":"crossref","first-page":"e4375","DOI":"10.7717\/peerj.4375","article-title":"The state of OA: A large-scale analysis of the prevalence and impact of Open Access articles","volume":"6","author":"Piwowar","year":"2018","journal-title":"PeerJ"},{"key":"2021072016160700800_bib41","doi-asserted-by":"crossref","first-page":"e27725","DOI":"10.7554\/eLife.27725","article-title":"The readability of scientific texts is decreasing over time","volume":"6","author":"Plav\u00e9n-Sigray","year":"2017","journal-title":"eLife"},{"issue":"3","key":"2021072016160700800_bib42","doi-asserted-by":"crossref","first-page":"130","DOI":"10.1108\/eb046814","article-title":"An algorithm for suffix stripping","volume":"14","author":"Porter","year":"1980","journal-title":"Program"},{"key":"2021072016160700800_bib43","article-title":"The NIPS experiment","author":"Price","year":"2014","journal-title":"Moody Rd blog post"},{"key":"2021072016160700800_bib44","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1007\/978-3-030-02934-0_7","article-title":"Modularized and attention-based recurrent convolutional neural network for automatic academic paper aspect scoring","volume-title":"Web Information Systems and Applications","author":"Qiao","year":"2018"},{"key":"2021072016160700800_bib45","article-title":"A longitudinal corpus study of syntactic complexity development in L2 writing","author":"Radhiah","year":"2018"},{"key":"2021072016160700800_bib46","article-title":"A step toward quantifying independently reproducible machine learning research","author":"Raff","year":"2019","journal-title":"arXiv:1909.06674 [cs, stat]"},{"issue":"14","key":"2021072016160700800_bib47","doi-asserted-by":"crossref","first-page":"1675","DOI":"10.1001\/jama.295.14.1675","article-title":"Effect of blinded peer review on abstract acceptance","volume":"295","author":"Ross","year":"2006","journal-title":"JAMA"},{"issue":"2","key":"2021072016160700800_bib48","doi-asserted-by":"crossref","first-page":"180","DOI":"10.3138\/jsp.47.2.180","article-title":"What we still don\u2019t know about peer review","volume":"47","author":"Sabaj Meruane","year":"2016","journal-title":"Journal of Scholarly Publishing"},{"key":"2021072016160700800_bib49","first-page":"649","article-title":"Do you cite what I mean? Assessing the semantic scope of bibliographic coupling in economics","volume-title":"23rd International Conference on Science and Technology Indicators","author":"Sainte-Marie","year":"2018"},{"issue":"4","key":"2021072016160700800_bib50","doi-asserted-by":"crossref","first-page":"S13","DOI":"10.1186\/bcr2742","article-title":"Classical peer review: An empty gun","volume":"12","author":"Smith","year":"2010","journal-title":"Breast Cancer Research"},{"key":"2021072016160700800_bib51","article-title":"Popularity of arXiv.org within computer science","author":"Sutton","year":"2017","journal-title":"arXiv:1710.05225 [cs]"},{"key":"2021072016160700800_bib52","doi-asserted-by":"crossref","first-page":"390","DOI":"10.3389\/fnhum.2017.00390","article-title":"When null hypothesis significance testing is unsuitable for research: A reassessment","volume":"11","author":"Szucs","year":"2017","journal-title":"Frontiers in Human Neuroscience"},{"key":"2021072016160700800_bib53","doi-asserted-by":"crossref","first-page":"e5707","DOI":"10.7717\/peerj.5707","article-title":"Predicting academic career outcomes by predoctoral publication record","volume":"6","author":"Tregellas","year":"2018","journal-title":"PeerJ"},{"issue":"8","key":"2021072016160700800_bib54","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1016\/S0169-5347(02)02545-4","article-title":"Gender bias in the refereeing process?","volume":"17","author":"Tregenza","year":"2002","journal-title":"Trends in Ecology & Evolution"},{"issue":"1","key":"2021072016160700800_bib55","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1177\/0165551518819965","article-title":"\u2018No comment\u2019? A study of commenting on PLOS articles","volume":"46","author":"Wakeling","year":"2019","journal-title":"Journal of Information Science"},{"key":"2021072016160700800_bib56","volume-title":"Real science: What it is and what it means","author":"Ziman","year":"2002"}],"container-title":["Quantitative Science Studies"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/qss\/article-pdf\/2\/2\/662\/1930783\/qss_a_00125.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/qss\/article-pdf\/2\/2\/662\/1930783\/qss_a_00125.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,18]],"date-time":"2022-12-18T18:06:44Z","timestamp":1671386804000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/qss\/article\/2\/2\/662\/97556\/Textual-analysis-of-artificial-intelligence"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":57,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,7,15]]}},"URL":"https:\/\/doi.org\/10.1162\/qss_a_00125","relation":{},"ISSN":["2641-3337"],"issn-type":[{"value":"2641-3337","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}