{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T06:53:14Z","timestamp":1775544794058,"version":"3.50.1"},"reference-count":130,"publisher":"Association for Computing Machinery (ACM)","issue":"7","funder":[{"DOI":"10.13039\/501100001711","name":"Swiss National Science Foundation","doi-asserted-by":"crossref","award":["TMSGI2_211379"],"award-info":[{"award-number":["TMSGI2_211379"]}],"id":[{"id":"10.13039\/501100001711","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,10,18]]},"abstract":"<jats:p>Journals and conferences worry that peer reviews assisted by artificial intelligence (AI), in particular, large language models (LLMs), may negatively influence the validity and fairness of the peer-review system, a cornerstone of modern science. In this work, we address this concern with a study of the prevalence and impact of AI-assisted peer reviews in the context of the 2024 International Conference on Learning Representations (ICLR), a large and prestigious machine-learning conference. Our contributions are threefold. Firstly, we obtain a lower bound for the prevalence of AI-assisted reviews at ICLR 2024 using the closed- and open-source LLM detectors, estimating that at least 15.8% of reviews were written with AI assistance. Secondly, we estimate the impact of AI-assisted reviews on submission scores. Considering pairs of reviews with different scores assigned to the same paper, we find that in 53.4% of pairs, the AI-assisted review scores higher than the human review (p = 0.002; relative difference in probability of scoring higher: +14.4% in favor of AI-assisted reviews). Thirdly, we assess the impact of receiving an AI-assisted peer review on submission acceptance. In a matched study, submissions near the acceptance threshold that received an AI-assisted peer review were 4.9 percentage points (p = 0.024) more likely to be accepted than submissions that did not. Overall, we show that AI-assisted reviews are consequential to the peer-review process and offer a discussion on future implications of current trends.<\/jats:p>","DOI":"10.1145\/3757667","type":"journal-article","created":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:06:01Z","timestamp":1760634361000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7583-6879","authenticated-orcid":false,"given":"Giuseppe","family":"Russo","sequence":"first","affiliation":[{"name":"EPFL, Lausanne, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6159-9657","authenticated-orcid":false,"given":"Manoel","family":"Horta Ribeiro","sequence":"additional","affiliation":[{"name":"Princeton University, Princeton, New Jersey, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-4366-6833","authenticated-orcid":false,"given":"Tim Ruben","family":"Davidson","sequence":"additional","affiliation":[{"name":"EPFL, Lausanne, Switzerland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9814-4373","authenticated-orcid":false,"given":"Veniamin","family":"Veselovsky","sequence":"additional","affiliation":[{"name":"Princeton University, Princeton, New Jersey, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3984-1232","authenticated-orcid":false,"given":"Robert","family":"West","sequence":"additional","affiliation":[{"name":"EPFL, Lausanne, Switzerland"}]}],"member":"320","published-online":{"date-parts":[[2025,10,16]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP40001.2021.00083"},{"key":"e_1_2_1_2_1","volume-title":"Advanced information networking and applications: Proceedings of the 34th international conference on advanced information networking and applications (AINA-2020)","author":"Adelani David Ifeoluwa","unstructured":"David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, Huy H Nguyen, Junichi Yamagishi, and Isao Echizen. 2020. Generating sentiment-preserving fake online reviews using neural language models and their human-and machine-based detection. In Advanced information networking and applications: Proceedings of the 34th international conference on advanced information networking and applications (AINA-2020). Springer, 1341-1354."},{"key":"e_1_2_1_3_1","volume-title":"An empirical study of ai generated text detection tools. arXiv preprint arXiv:2310.01423","author":"Akram Arslan","year":"2023","unstructured":"Arslan Akram. 2023. An empirical study of ai generated text detection tools. arXiv preprint arXiv:2310.01423 (2023)."},{"key":"e_1_2_1_4_1","doi-asserted-by":"crossref","unstructured":"Bruce Alberts Brooks Hanson and Katrina L Kelner. 2008. Reviewing peer review. 15-15 pages.","DOI":"10.1126\/science.1162115"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2301839120"},{"key":"e_1_2_1_6_1","volume-title":"Peer review and the publication process. Nursing open","author":"Ali Parveen Azam","year":"2016","unstructured":"Parveen Azam Ali and Roger Watson. 2016. Peer review and the publication process. Nursing open, Vol. 3, 4 (2016), 193-202."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3002184"},{"key":"e_1_2_1_8_1","volume-title":"Machine bias: There's software used across the country to predict future criminals. And it's biased against blacks. ProPublica","author":"Angwin Julia","year":"2016","unstructured":"Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias: There's software used across the country to predict future criminals. And it's biased against blacks. ProPublica (2016)."},{"key":"e_1_2_1_9_1","first-page":"467","volume-title":"Nature","volume":"515","author":"Arns Martijn","year":"2014","unstructured":"Martijn Arns. 2014. Open access is tiring out peer reviewers. Nature, Vol. 515, 7528 (2014), 467-467."},{"key":"e_1_2_1_10_1","unstructured":"Association for Computational Linguistics. 2024. Reviewer Guidelines for ACL Rolling Review. https:\/\/aclrollingreview.org\/reviewerguidelines Accessed: 2024-10-27."},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","unstructured":"Ahmed Salem BaHammam. 2025. Peer Review in the Artificial Intelligence Era: A Call for Developing Responsible Integration Guidelines. 159-164 pages.","DOI":"10.2147\/NSS.S513872"},{"key":"e_1_2_1_12_1","volume-title":"NeurIPS","volume":"36","author":"Bai Yushi","year":"2024","unstructured":"Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al., 2024. Benchmarking foundation models with language-model-as-an-examiner. NeurIPS, Vol. 36 (2024)."},{"key":"e_1_2_1_13_1","volume-title":"Working 9 to 5, not the way to make an academic living: observational analysis of manuscript and peer review submissions over time. bmj","author":"Barnett Adrian","year":"2019","unstructured":"Adrian Barnett, Inger Mewburn, and Sara Schroter. 2019. Working 9 to 5, not the way to make an academic living: observational analysis of manuscript and peer review submissions over time. bmj, Vol. 367 (2019)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1002\/wcc.547"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1108\/INTR-01-2022-0042"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5863\/1551-6776-29.4.441"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23329"},{"key":"e_1_2_1_18_1","volume-title":"ACL 2023 policy on AI writing assistance. https:\/\/2023","author":"Boyd-Graber Jordan","year":"2023","unstructured":"Jordan Boyd-Graber, Naoaki Okazaki, and Anna Rogers. 2023. ACL 2023 policy on AI writing assistance. https:\/\/2023.aclweb.org\/blog\/ACL-2023-policy."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1049096515000827"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1177\/2515245918823199"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3579612"},{"key":"e_1_2_1_22_1","volume-title":"AI-assisted peer review. Humanities and social sciences communications","author":"Checco Alessandro","year":"2021","unstructured":"Alessandro Checco, Lorenzo Bracciale, Pierpaolo Loreti, Stephen Pinfield, and Giuseppe Bianchi. 2021. AI-assisted peer review. Humanities and social sciences communications, Vol. 8, 1 (2021), 1-11."},{"key":"e_1_2_1_23_1","volume-title":"AI fairness in data management and analytics: A review on challenges, methodologies and applications. Applied sciences","author":"Chen Pu","year":"2023","unstructured":"Pu Chen, Linna Wu, and Lei Wang. 2023. AI fairness in data management and analytics: A review on challenges, methodologies and applications. Applied sciences, Vol. 13, 18 (2023), 10258."},{"key":"e_1_2_1_24_1","volume-title":"Peerless science: Peer review and US science policy","author":"Chubin Daryl E","unstructured":"Daryl E Chubin and Edward J Hackett. 1990. Peerless science: Peer review and US science policy. State University of New York Press."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501819"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X00065675"},{"key":"e_1_2_1_27_1","unstructured":"Clarivate. 2018. Global State of peer review report. https:\/\/clarivate.com\/lp\/global-state-of-peer-review-report\/."},{"key":"e_1_2_1_28_1","volume-title":"Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092","author":"Cui Jiaxi","year":"2023","unstructured":"Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092 (2023)."},{"key":"e_1_2_1_29_1","volume-title":"Self-recognition in language models. arXiv preprint arXiv:2407.06946","author":"Davidson Tim R","year":"2024","unstructured":"Tim R Davidson, Viacheslav Surkov, Veniamin Veselovsky, Giuseppe Russo, Robert West, and Caglar Gulcehre. 2024. Self-recognition in language models. arXiv preprint arXiv:2407.06946 (2024)."},{"key":"e_1_2_1_30_1","volume-title":"Artificial intelligence in medicine and cardiac imaging: harnessing big data and advanced computing to provide personalized medical diagnosis and treatment. Current cardiology reports","author":"Dilsizian Steven E","year":"2014","unstructured":"Steven E Dilsizian and Eliot L Siegel. 2014. Artificial intelligence in medicine and cardiac imaging: harnessing big data and advanced computing to provide personalized medical diagnosis and treatment. Current cardiology reports, Vol. 16 (2014), 1-8."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1473-3099(23)00290-6"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544549.3582745"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1049096517001081"},{"key":"e_1_2_1_34_1","first-page":"205","article-title":"Peer review, rejection, and success in academic publishing","volume":"50","author":"Esarey Justin","year":"2017","unstructured":"Justin Esarey. 2017b. Peer review, rejection, and success in academic publishing. PS: Political Science & Politics, Vol. 50, 1 (2017), 205-211.","journal-title":"PS: Political Science & Politics"},{"key":"e_1_2_1_35_1","volume-title":"Hierarchical neural story generation. arXiv preprint arXiv:1805.04833","author":"Fan Angela","year":"2018","unstructured":"Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833 (2018)."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3581641.3584034"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1186\/s41073-017-0027-x"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10439-023-03299-7"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-3019"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-022-04524-1"},{"key":"e_1_2_1_41_1","volume-title":"The use of artificial intelligence to improve the scientific writing of non-native English speakers. Revista da Associa\u00e7 ao M\u00e9dica Brasileira","author":"Giglio Auro Del","year":"2023","unstructured":"Auro Del Giglio and Mateus Uerlei Pereira da Costa. 2023. The use of artificial intelligence to improve the scientific writing of non-native English speakers. Revista da Associa\u00e7 ao M\u00e9dica Brasileira, Vol. 69, 9 (2023), e20230560."},{"key":"e_1_2_1_42_1","volume-title":"The ethical need for watermarks in machine-generated language. arXiv preprint arXiv:2209.03118","author":"Grinbaum Alexei","year":"2022","unstructured":"Alexei Grinbaum and Laurynas Adomaitis. 2022. The ethical need for watermarks in machine-generated language. arXiv preprint arXiv:2209.03118 (2022)."},{"key":"e_1_2_1_43_1","volume-title":"How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597","author":"Guo Biyang","year":"2023","unstructured":"Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597 (2023)."},{"key":"e_1_2_1_44_1","unstructured":"Abhimanyu Hans Avi Schwarzschild Valeriia Cherepanova Hamid Kazemi Aniruddha Saha Micah Goldblum Jonas Geiping and Tom Goldstein. 2024. Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text. arXiv:2401.12070 [cs.CL]"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1049096521000469"},{"key":"e_1_2_1_46_1","volume-title":"Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review. Research integrity and peer review","author":"Hosseini Mohammad","year":"2023","unstructured":"Mohammad Hosseini and Serge PJM Horbach. 2023. Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review. Research integrity and peer review, Vol. 8, 1 (2023), 4."},{"key":"e_1_2_1_47_1","volume-title":"ICML 2023","author":"ICML.","year":"2024","unstructured":"ICML. 2024. ICML 2023. https:\/\/icml.cc\/Conferences\/2023\/llm-policy."},{"key":"e_1_2_1_48_1","volume-title":"Assisting ICLR 2025 reviewers with feedback. https:\/\/blog.iclr.cc\/2024\/10\/09\/iclr2025-assisting-reviewers\/","author":"ICRL.","year":"2024","unstructured":"ICRL. 2024. Assisting ICLR 2025 reviewers with feedback. https:\/\/blog.iclr.cc\/2024\/10\/09\/iclr2025-assisting-reviewers\/"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2208839120"},{"key":"e_1_2_1_50_1","first-page":"376","article-title":"Evaluating the peer review process: a case study from the Annals of Internal Medicine","volume":"136","author":"Jefferson Tom","year":"2002","unstructured":"Tom Jefferson, Michael Rudin, Stanley Brodney Folse, and Frank Davidoff. 2002. Evaluating the peer review process: a case study from the Annals of Internal Medicine. Annals of Internal Medicine, Vol. 136, 5 (2002), 376-383.","journal-title":"Annals of Internal Medicine"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3676492"},{"key":"e_1_2_1_52_1","volume-title":"Patterns","volume":"4","author":"Kapoor Sayash","year":"2023","unstructured":"Sayash Kapoor and Arvind Narayanan. 2023. Leakage and the reproducibility crisis in machine-learning-based science. Patterns, Vol. 4, 9 (2023)."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3581641.3584078"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1177\/1023263X20978649"},{"key":"e_1_2_1_55_1","volume-title":"The impact of geographical bias when judging scientific studies. Scientometrics","author":"Kowal Marta","year":"2022","unstructured":"Marta Kowal, Piotr Sorokowski, Emanuel Kulczycki, and Agnieszka. Zela'zniewicz. 2022. The impact of geographical bias when judging scientific studies. Scientometrics (2022), 1-9."},{"key":"e_1_2_1_56_1","volume-title":"Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, et al.","author":"Kuznetsov Ilia","year":"2024","unstructured":"Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, et al., 2024. What Can Natural Language Processing Do for Peer Review? arXiv preprint arXiv:2405.06563 (2024)."},{"key":"e_1_2_1_57_1","volume-title":"A New Publishing Model in Computer Science. Commun. ACM","author":"LeCun Yann","year":"2024","unstructured":"Yann LeCun. 2024. A New Publishing Model in Computer Science. Commun. ACM (2024)."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.22784"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.22784"},{"key":"e_1_2_1_60_1","volume-title":"Plagbench: Exploring the duality of large language models in plagiarism generation and detection. arXiv preprint arXiv:2406.16288","author":"Lee Jooyoung","year":"2024","unstructured":"Jooyoung Lee, Toshini Agrawal, Adaku Uchendu, Thai Le, Jinghui Chen, and Dongwon Lee. 2024. Plagbench: Exploring the duality of large language models in plagiarism generation and detection. arXiv preprint arXiv:2406.16288 (2024)."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3610218"},{"key":"e_1_2_1_62_1","volume-title":"Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews. arXiv preprint arXiv:2403.07183","author":"Liang Weixin","year":"2024","unstructured":"Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al., 2024a. Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews. arXiv preprint arXiv:2403.07183 (2024)."},{"key":"e_1_2_1_63_1","unstructured":"Weixin Liang Yaohui Zhang Zhengxuan Wu Haley Lepp Wenlong Ji Xuandong Zhao Hancheng Cao Sheng Liu Siyu He Zhi Huang et al. 2024b. Mapping the Increasing Use of LLMs in Scientific Papers. arXiv preprint arXiv:2404.01268 (2024)."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2018.08.009"},{"key":"e_1_2_1_65_1","volume-title":"Techniques for supercharging academic writing with generative AI. Nature Biomedical Engineering","author":"Lin Zhicheng","year":"2024","unstructured":"Zhicheng Lin. 2024. Techniques for supercharging academic writing with generative AI. Nature Biomedical Engineering (2024), 1-6."},{"key":"e_1_2_1_66_1","volume-title":"Scholarly Peer Review is an Age-Old Practice","author":"MacKinney Taryn","year":"2023","unstructured":"Taryn MacKinney. 2023. Scholarly Peer Review is an Age-Old Practice, But Publishing is Changing. http:\/\/www.aps.org\/publications\/apsnews\/202310\/peer-review.cfm."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41398-023-02592-2"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1980.tb01109.x"},{"key":"e_1_2_1_69_1","unstructured":"Meta. 2024. Llama 3. https:\/\/ai.meta.com\/blog\/meta-llama-3\/"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jeconom.2017.06.012"},{"key":"e_1_2_1_71_1","volume-title":"The cost of training AI could soon become too much to bear. Fortune (April","author":"Meyer David","year":"2024","unstructured":"David Meyer. 2024. The cost of training AI could soon become too much to bear. Fortune (April 2024). https:\/\/fortune.com\/2024\/04\/04\/ai-training-costs-how-much-is-too-much-openai-gpt-anthropic-microsoft"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.system.2004.11.003"},{"key":"e_1_2_1_73_1","first-page":"1","article-title":"Reimagining the future of peer review beyond operations","volume":"8","author":"Misra Adya","year":"2023","unstructured":"Adya Misra. 2023. Reimagining the future of peer review beyond operations. Research Integrity and Peer Review, Vol. 8 (2023), 1-3.","journal-title":"Research Integrity and Peer Review"},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing.","author":"Mitchell Emily","year":"2023","unstructured":"Emily Mitchell, Eric Yang, et al., 2023. DetectGPT: Log Probability-Based Detection of AI-Generated Text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing."},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0155876"},{"key":"e_1_2_1_76_1","unstructured":"Neurips. 2021. The NeurIPS 2021 Consistency Experiment. https:\/\/blog.neurips.cc\/2021\/12\/08\/the-neurips-2021-consistency-experiment\/."},{"key":"e_1_2_1_77_1","unstructured":"NIH. 2024. Peer Review. https:\/\/grants.nih.gov\/grants\/peer-review.htm."},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1515\/ci-2024-0104"},{"key":"e_1_2_1_79_1","unstructured":"NSF. 2024. Merit Review textbar NSF - National Science Foundation. https:\/\/www.nsf.gov\/bfa\/dias\/policy\/merit_review\/."},{"key":"e_1_2_1_80_1","unstructured":"OpenAI. 2022. Introducing ChatGPT. https:\/\/openai.com\/blog\/chatgpt"},{"key":"e_1_2_1_81_1","volume-title":"Introducing the GPT Store. https:\/\/openai.com\/index\/introducing-the-gpt-store [Online","author":"AI.","year":"2024","unstructured":"OpenAI. 2024. Introducing the GPT Store. https:\/\/openai.com\/index\/introducing-the-gpt-store [Online; accessed 16. Jun. 2024]."},{"key":"e_1_2_1_82_1","unstructured":"Riccardo Pellecchia. 2022. Leveraging AI via speech-to-text and LLM integration for improved healthcare decision-making in primary care. (2022)."},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1714379115"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1093\/reseval\/rvw025"},{"key":"e_1_2_1_85_1","volume-title":"An Empirical Analysis of LLMs for Countering Misinformation. arXiv preprint arXiv:2503.01902","author":"Proma Adiba Mahbub","year":"2025","unstructured":"Adiba Mahbub Proma, Neeley Pate, James Druckman, Gourab Ghoshal, Hangfeng He, and Ehsan Hoque. 2025. An Empirical Analysis of LLMs for Countering Misinformation. arXiv preprint arXiv:2503.01902 (2025)."},{"key":"e_1_2_1_86_1","volume-title":"Sentence-bert: Sentence embeddings using siamese bert-networks. EMNLP","author":"Reimers Nils","year":"2019","unstructured":"Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. EMNLP (2019)."},{"key":"e_1_2_1_87_1","first-page":"1","article-title":"The Impact of Artificial Intelligence on Legal Decision-Making","volume":"70","author":"Remus Dana","year":"2016","unstructured":"Dana Remus and Frank Levy. 2016. The Impact of Artificial Intelligence on Legal Decision-Making. Law Review, Vol. 70 (2016), 1-34.","journal-title":"Law Review"},{"key":"e_1_2_1_88_1","first-page":"31","volume-title":"Nature","volume":"535","author":"Rennie Drummond","year":"2016","unstructured":"Drummond Rennie. 2016. Let's make peer review scientific. Nature, Vol. 535, 7610 (2016), 31-33."},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1198\/jasa.2009.tm08470"},{"key":"e_1_2_1_90_1","volume-title":"Meat-Free Day Reduces Greenhouse Gas Emissions but Poses Challenges for Customer Retention and Adherence to Dietary Guidelines. arXiv preprint arXiv:2504.02899","author":"Russo Giuseppe","year":"2025","unstructured":"Giuseppe Russo, Kristina Gligori\u0107, Vincent Moreau, and Robert West. 2025a. Meat-Free Day Reduces Greenhouse Gas Emissions but Poses Challenges for Customer Retention and Adherence to Dietary Guidelines. arXiv preprint arXiv:2504.02899 (2025)."},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/3578503.3583608"},{"key":"e_1_2_1_92_1","volume-title":"The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models. arXiv preprint arXiv:2507.17216","author":"Russo Giuseppe","year":"2025","unstructured":"Giuseppe Russo, Debora Nozza, Paul R\u00f6ttger, and Dirk Hovy. 2025b. The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models. arXiv preprint arXiv:2507.17216 (2025)."},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v18i1.31393"},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v19i1.35897"},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v17i1.22184"},{"key":"e_1_2_1_96_1","first-page":"706","volume-title":"Companion Proceedings of the ACM Web Conference","author":"Latona Giuseppe Russo","year":"2024","unstructured":"Giuseppe Russo Latona, Christoph Gote, Christian Zingg, Giona Casiraghi, Luca Verginer, and Frank Schweitzer. 2024. Shock! Quantifying the Impact of Core Developers' Dropout on the Productivity of OSS Projects. In Companion Proceedings of the ACM Web Conference 2024. 706-709."},{"key":"e_1_2_1_97_1","volume-title":"Can AI-generated text be reliably detected? arXiv preprint arXiv:2303.11156","author":"Sadasivan Vinu Sankar","year":"2023","unstructured":"Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can AI-generated text be reliably detected? arXiv preprint arXiv:2303.11156 (2023)."},{"key":"e_1_2_1_98_1","volume-title":"On the value of preprints: An early career researcher perspective. PLoS biology","author":"Sarabipour Sarvenaz","year":"2019","unstructured":"Sarvenaz Sarabipour, Humberto J Debat, Edward Emmott, Steven J Burgess, Benjamin Schwessinger, and Zach Hensel. 2019. On the value of preprints: An early career researcher perspective. PLoS biology, Vol. 17, 2 (2019), e3000151."},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.38023.700775.AE"},{"key":"e_1_2_1_100_1","unstructured":"Science. 2024. Peer Review at Science Journals. https:\/\/www.science.org\/content\/page\/peer-review-science-publications."},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642159"},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.281.20.1900"},{"key":"e_1_2_1_103_1","volume-title":"David Wadden, Khendra G Lucas, Adam S Miner, Theresa Nguyen, and Tim Althoff.","author":"Sharma Ashish","year":"2023","unstructured":"Ashish Sharma, Kevin Rushton, Inna Wanyin Lin, David Wadden, Khendra G Lucas, Adam S Miner, Theresa Nguyen, and Tim Althoff. 2023. Cognitive reframing of negative thoughts through human-language model interaction. arXiv preprint arXiv:2305.02466 (2023)."},{"key":"e_1_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2383041109"},{"key":"e_1_2_1_105_1","doi-asserted-by":"publisher","DOI":"10.1177\/014107680609900414"},{"key":"e_1_2_1_106_1","volume-title":"Sarah Kreps, et al.","author":"Solaiman Irene","year":"2019","unstructured":"Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al., 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203 (2019)."},{"key":"e_1_2_1_107_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijinfomgt.2023.102700"},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1145\/3640543.3645159"},{"key":"e_1_2_1_109_1","doi-asserted-by":"publisher","DOI":"10.1145\/3637371"},{"key":"e_1_2_1_110_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSEN.2020.3027564"},{"key":"e_1_2_1_111_1","volume-title":"The state of the art in peer review. FEMS Microbiology letters","author":"Tennant Jonathan P","year":"2018","unstructured":"Jonathan P Tennant. 2018. The state of the art in peer review. FEMS Microbiology letters, Vol. 365, 19 (2018), fny204."},{"key":"e_1_2_1_112_1","unstructured":"Jonathan P Tennant and et al. [n.d.]. The future of peer review in scholarly communication. F1000Research ([n.d.])."},{"key":"e_1_2_1_113_1","doi-asserted-by":"publisher","DOI":"10.2478\/jdis-2024-0013"},{"key":"e_1_2_1_114_1","unstructured":"Edward Tian and et al. 2023. GPTZero: Detecting AI-generated content through linguistic markers. Journal of Computational Linguistics (2023)."},{"key":"e_1_2_1_115_1","article-title":"The role of peer review in assessing research","volume":"309","author":"Travis GD","year":"1994","unstructured":"GD Travis and HM Collins. 1994. The role of peer review in assessing research. British Medical Journal, Vol. 309, 6956 (1994), 513-515.","journal-title":"British Medical Journal"},{"key":"e_1_2_1_116_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v11i1.27557"},{"key":"e_1_2_1_117_1","volume-title":"Nature","volume":"10","author":"Vesper Inga","year":"2018","unstructured":"Inga Vesper. 2018. Peer reviewers unmasked: largest global survey reveals trends. Nature, Vol. 10 (2018)."},{"key":"e_1_2_1_118_1","first-page":"789","article-title":"Randomness and uncertainty in peer review processes: Exploring the implications for scientific advancement","volume":"47","author":"Vesper Seanna","year":"2018","unstructured":"Seanna Vesper and et al., 2018. Randomness and uncertainty in peer review processes: Exploring the implications for scientific advancement. Research Policy, Vol. 47, 4 (2018), 789-797.","journal-title":"Research Policy"},{"key":"e_1_2_1_119_1","volume-title":"Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation. arXiv preprint arXiv:2401.06391","author":"Wang Chong","year":"2024","unstructured":"Chong Wang, Jian Zhang, Yebo Feng, Tianlin Li, Weisong Sun, Yang Liu, and Xin Peng. 2024. Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation. arXiv preprint arXiv:2401.06391 (2024)."},{"key":"e_1_2_1_120_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359313"},{"key":"e_1_2_1_121_1","unstructured":"Mark Ware and Michael Mabe. 2015. The STM report: An overview of scientific and scholarly journal publishing."},{"key":"e_1_2_1_122_1","volume-title":"Peer review of grant applications: what do we know? The lancet","author":"Wessely Simon","year":"1998","unstructured":"Simon Wessely. 1998. Peer review of grant applications: what do we know? The lancet, Vol. 352, 9124 (1998), 301-305."},{"key":"e_1_2_1_123_1","volume-title":"Epistemic trust in science. The British Journal for the Philosophy of Science","author":"Wilholt Torsten","year":"2013","unstructured":"Torsten Wilholt. 2013. Epistemic trust in science. The British Journal for the Philosophy of Science (2013)."},{"key":"e_1_2_1_124_1","volume-title":"Gender and racial\/ethnic disparities in rates of publishing and inclusion in scientific-review processes. Translational issues in psychological science","author":"Willis Malachi","year":"2021","unstructured":"Malachi Willis, Ana J Bridges, and Kristen N Jozkowski. 2021. Gender and racial\/ethnic disparities in rates of publishing and inclusion in scientific-review processes. Translational issues in psychological science, Vol. 7, 4 (2021), 451."},{"key":"e_1_2_1_125_1","volume-title":"Social skill training with large language models. arXiv preprint arXiv:2404.04204","author":"Yang Diyi","year":"2024","unstructured":"Diyi Yang, Caleb Ziems, William Held, Omar Shaikh, Michael S Bernstein, and John Mitchell. 2024. Social skill training with large language models. arXiv preprint arXiv:2404.04204 (2024)."},{"key":"e_1_2_1_126_1","doi-asserted-by":"publisher","DOI":"10.1002\/hcs2.61"},{"key":"e_1_2_1_127_1","volume-title":"Bertscore: Evaluating text generation with bert. ICLR","author":"Zhang Tianyi","year":"2020","unstructured":"Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. ICLR (2020)."},{"key":"e_1_2_1_128_1","volume-title":"NeurIPS","volume":"36","author":"Zheng Lianmin","year":"2024","unstructured":"Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al., 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS, Vol. 36 (2024)."},{"key":"e_1_2_1_129_1","doi-asserted-by":"crossref","unstructured":"Shuang Zhou Zidu Xu Mian Zhang Chunpu Xu Yawen Guo Zaifu Zhan Sirui Ding Jiashuo Wang Kaishuai Xu Yi Fang et al. 2024. Large Language Models for Disease Diagnosis: A Scoping Review. arXiv preprint arXiv:2409.00097 (2024).","DOI":"10.1038\/s44387-025-00011-z"},{"key":"e_1_2_1_130_1","volume-title":"Sanjay A Pai, Leslie Citrome, Murad Alam, Michael Voight, et al.","author":"Zielinski Chris","year":"2023","unstructured":"Chris Zielinski, Margaret A Winker, Rakesh Aggarwal, Lorraine E Ferris, Markus Heinemann, Jose Florencio Lape na Jr, Sanjay A Pai, Leslie Citrome, Murad Alam, Michael Voight, et al., 2023. Chatbots, generative AI, and scholarly manuscripts: WAME recommendations on chatbots and generative artificial intelligence in relation to scholarly publications. Colombia M\u00e9dica, Vol. 54, 3 (2023)."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3757667","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T01:55:47Z","timestamp":1760666147000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3757667"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,16]]},"references-count":130,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,10,18]]}},"alternative-id":["10.1145\/3757667"],"URL":"https:\/\/doi.org\/10.1145\/3757667","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,16]]},"assertion":[{"value":"2025-10-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}