{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T19:20:54Z","timestamp":1792783254323,"version":"4.5.0"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","license":[{"start":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T00:00:00Z","timestamp":1790899200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T00:00:00Z","timestamp":1790899200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"University of Geneva","id":[{"id":"https:\/\/ror.org\/01swzsf04","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J GEN INTERN MED"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>The use of artificial intelligence (AI) models as reviewers of scientific content raises concerns about potential biases related to author identity and about the reproducibility of their evaluations. We assessed whether AI-based reviewers exhibit gender or geographic bias and evaluated the reproducibility of their scoring of scientific abstracts.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>\n                      We randomly selected 10 general internal medicine journals indexed in the Journal Citation Reports (impact factor\u2009\u2265\u20091.5). For each journal, five original research abstracts were retrieved in April 2026 from the Web of Science. Each abstract (\n                      <jats:italic>n<\/jats:italic>\n                      \u2009=\u200950) was assigned to four fictional author identities (African female, African male, American female, American male). Abstracts were independently evaluated twice by two AI models (ChatGPT and Claude), scoring quality\/novelty\/acceptance on a 0\u201310 scale, yielding 800 evaluations. Bias was assessed using multivariable ordinal logistic regression, and reproducibility using percent agreement and Fleiss\u2019 kappa.\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      For ChatGPT, quality and novelty scores were identical across identities (median\u00a0[IQR]\u2009= 7\u00a0[1] and 5\u00a0[2], respectively), with minimal variation in acceptance (median\u00a0[IQR]\u2009=\u20096\u20137\u00a0[1\u20132]). For Claude, all scores were identical across identities (quality = 7\u00a0[1], novelty =\u00a05\u00a0[2], acceptance =\u00a06\u00a0[2]). Multivariable analyses showed no overall association between author identity and scores, with the exception of acceptance for Claude at the global level, although no individual comparisons were statistically significant. Reproducibility was high for both models, with percent agreement\u2009&gt;\u20090.98 and substantial\/excellent agreement (\n                      <jats:italic>\u03ba<\/jats:italic>\n                      \u2009=\u20090.74\u20130.89).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>In this controlled setting, we detected no consistent evidence of gender or geographic bias, and reproducibility was high, although the restricted score range may have contributed to the observed agreement. Further studies are needed to assess the validity of LLM-based evaluation and compare it with human peer review.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1007\/s11606-026-10806-8","type":"journal-article","created":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T15:25:51Z","timestamp":1790954751000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Bias and Reliability of AI-Based Peer Review: A Comparative Study of ChatGPT and Claude Evaluating Scientific Abstracts"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7616-0017","authenticated-orcid":false,"given":"Paul","family":"Sebo","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ting","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,10,2]]},"reference":[{"key":"10806_CR1","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","volume":"25","author":"EJ Topol","year":"2019","unstructured":"Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25:44\u201356. https:\/\/doi.org\/10.1038\/s41591-018-0300-7.","journal-title":"Nat Med"},{"key":"10806_CR2","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1136\/svn-2017-000101","volume":"2","author":"F Jiang","year":"2017","unstructured":"Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. 2017;2:230\u201343. https:\/\/doi.org\/10.1136\/svn-2017-000101.","journal-title":"Stroke Vasc Neurol"},{"key":"10806_CR3","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1038\/s41591-018-0316-z","volume":"25","author":"A Esteva","year":"2019","unstructured":"Esteva A, Robicquet A, Ramsundar B, Kuleshov V, DePristo M, Chou K, et al. A guide to deep learning in healthcare. Nat Med. 2019;25:24\u20139. https:\/\/doi.org\/10.1038\/s41591-018-0316-z.","journal-title":"Nat Med"},{"key":"10806_CR4","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","volume":"29","author":"AJ Thirunavukarasu","year":"2023","unstructured":"Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med.2023;29:1930\u201340. https:\/\/doi.org\/10.1038\/s41591-023-02448-8.","journal-title":"Nat Med"},{"key":"10806_CR5","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1038\/s43856-023-00370-1","volume":"3","author":"J Clusmann","year":"2023","unstructured":"Clusmann J, Kolbinger FR, Muti HS, Carrero ZI, Eckardt J-N, Laleh NG, et al. The future landscape of large language models in medicine. Commun Med. 2023;3:141. https:\/\/doi.org\/10.1038\/s43856-023-00370-1.","journal-title":"Commun Med"},{"key":"10806_CR6","doi-asserted-by":"publisher","DOI":"10.2196\/48659","volume":"25","author":"A Rao","year":"2023","unstructured":"Rao A, Pang M, Kim J, Kamineni M, Lie W, Prasad AK, et al. Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study. J Med Internet Res. 2023;25:e48659. https:\/\/doi.org\/10.2196\/48659.","journal-title":"J Med Internet Res"},{"key":"10806_CR7","doi-asserted-by":"publisher","first-page":"2817","DOI":"10.1007\/s00330-023-10213-1","volume":"34","author":"K Jeblick","year":"2024","unstructured":"Jeblick K, Schachtner B, Dexl J, Mittermeier A, St\u00fcber AT, Topalis J, et al. ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports. Eur Radiol. 2024;34:2817\u201325. https:\/\/doi.org\/10.1007\/s00330-023-10213-1.","journal-title":"Eur Radiol"},{"key":"10806_CR8","doi-asserted-by":"publisher","first-page":"7094","DOI":"10.1097\/MS9.0000000000002716","volume":"86","author":"S Abhari","year":"2024","unstructured":"Abhari S, Afshari Y, Fatehi F, Salmani H, Garavand A, Chumachenko D, et al. Exploring ChatGPT in clinical inquiry: a scoping review of characteristics, applications, challenges, and evaluation. Ann Med Surg. 2024;86:7094\u2013104. https:\/\/doi.org\/10.1097\/MS9.0000000000002716.","journal-title":"Ann Med Surg"},{"key":"10806_CR9","doi-asserted-by":"publisher","DOI":"10.1007\/s11739-026-04291-4","author":"P Sebo","year":"2026","unstructured":"Sebo P, Wang T. Evaluating the performance of ChatGPT-5, Claude, and DeepSeek for statistical analysis in medicine. Intern Emerg Med. 2026. https:\/\/doi.org\/10.1007\/s11739-026-04291-4.","journal-title":"Intern Emerg Med"},{"key":"10806_CR10","doi-asserted-by":"publisher","unstructured":"Sebo P, Wang T. ChatGPT\u2019s performance in sample size estimation: a preliminary study on the capabilities of artificial intelligence. Fam Pract. 2025;42:cmaf069. https:\/\/doi.org\/10.1093\/fampra\/cmaf069.","DOI":"10.1093\/fampra\/cmaf069"},{"key":"10806_CR11","doi-asserted-by":"publisher","unstructured":"Sebo P, Nie B, Wang T. Can ChatGPT write better scientific titles? A comparative evaluation of human-written and AI-generated titles. F1000Research. 2025;14:1470. https:\/\/doi.org\/10.12688\/f1000research.173647.2.","DOI":"10.12688\/f1000research.173647.2"},{"key":"10806_CR12","doi-asserted-by":"publisher","unstructured":"Ko\u00e7ak D. Examination of ChatGPT\u2019s Performance as a Data Analysis Tool. Educ Psychol Meas. 2025:00131644241302721. https:\/\/doi.org\/10.1177\/00131644241302721.","DOI":"10.1177\/00131644241302721"},{"key":"10806_CR13","doi-asserted-by":"publisher","DOI":"10.2196\/84322","volume":"5","author":"SM Shen","year":"2026","unstructured":"Shen SM, Wang Z, Paul K, Li M-H, Huang X, Koizumi N. Evaluation of Large Language Models for Peer Review in Transplantation Research: Algorithm Validation Study. JMIR AI. 2026;5:e84322. https:\/\/doi.org\/10.2196\/84322.","journal-title":"JMIR AI"},{"key":"10806_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijmedinf.2026.106418","volume":"214","author":"A Nabavi","year":"2026","unstructured":"Nabavi A, Safari F, Shmoury AH, Tabet S, Perdomo-Luna C, Celi LA. Artificial intelligence in scholarly peer review: a scoping review of applications, risks, and governance challenges. Int J Med Inf. 2026;214:106418. https:\/\/doi.org\/10.1016\/j.ijmedinf.2026.106418.","journal-title":"Int J Med Inf"},{"issue":"26","key":"10806_CR15","doi-asserted-by":"publisher","first-page":"00763","DOI":"10.1016\/j.jpainsymman.2026.04.614","volume":"S0885\u20133924","author":"D Hui","year":"2026","unstructured":"Hui D, Mehta AK, Shalev D, Rosa WE, Chwistek M, Wallace C, et al. Generative Artificial Intelligence Use in the Peer Review Process. J Pain Symptom Manage. 2026:S0885-3924(26)00763-3. https:\/\/doi.org\/10.1016\/j.jpainsymman.2026.04.614.","journal-title":"J Pain Symptom Manage"},{"key":"10806_CR16","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1186\/s41073-023-00133-5","volume":"8","author":"M Hosseini","year":"2023","unstructured":"Hosseini M, Horbach SPJM. Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review. Res Integr Peer Rev. 2023;8:4. https:\/\/doi.org\/10.1186\/s41073-023-00133-5.","journal-title":"Res Integr Peer Rev"},{"key":"10806_CR17","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1186\/s41073-025-00161-3","volume":"10","author":"Vecchio N Lo","year":"2025","unstructured":"Lo Vecchio N. Personal experience with AI-generated peer reviews: a case study. Res Integr Peer Rev. 2025;10:4. https:\/\/doi.org\/10.1186\/s41073-025-00161-3.","journal-title":"Res Integr Peer Rev"},{"key":"10806_CR18","doi-asserted-by":"publisher","unstructured":"Ng JY, Krishnamurthy M, Deol G, Al-Khafaji WA-Z, Balaji V, Abebe M, et al. Attitudes and perceptions of biomedical journal editors in chief towards the use of artificial intelligence chatbots in the scholarly publishing process: a cross-sectional survey. Res Integr Peer Rev. 2025;10:19. https:\/\/doi.org\/10.1186\/s41073-025-00178-8.","DOI":"10.1186\/s41073-025-00178-8"},{"key":"10806_CR19","doi-asserted-by":"crossref","unstructured":"Jones AH, McLellan F. Ethical Issues in Biomedical Publication. JHU Press; 2000.","DOI":"10.56021\/9780801863141"},{"key":"10806_CR20","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pbio.3003574","volume":"24","author":"D Alvarez-Ponce","year":"2026","unstructured":"Alvarez-Ponce D, Batz G, Ramirez Torres L. Biomedical and life science articles by female researchers spend longer under review. PLoS Biol. 2026;24:e3003574. https:\/\/doi.org\/10.1371\/journal.pbio.3003574.","journal-title":"PLoS Biol"},{"key":"10806_CR21","doi-asserted-by":"publisher","first-page":"1771","DOI":"10.1007\/s11606-022-07946-y","volume":"38","author":"P Sebo","year":"2023","unstructured":"Sebo P. Time to Publication in High-Impact General Medical Journals Differs Between Female and Male Researchers. J Gen Intern Med. 2023;38:1771\u20135. https:\/\/doi.org\/10.1007\/s11606-022-07946-y.","journal-title":"J Gen Intern Med"},{"key":"10806_CR22","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1186\/s41073-019-0088-0","volume":"5","author":"M Skopec","year":"2020","unstructured":"Skopec M, Issa H, Reed J, Harris M. The role of geographic bias in knowledge diffusion: a systematic review and narrative synthesis. Res Integr Peer Rev. 2020;5:2. https:\/\/doi.org\/10.1186\/s41073-019-0088-0.","journal-title":"Res Integr Peer Rev"},{"key":"10806_CR23","doi-asserted-by":"publisher","DOI":"10.2196\/57667","volume":"26","author":"P Sebo","year":"2024","unstructured":"Sebo P. Use of ChatGPT to Explore Gender and Geographic Disparities in Scientific Peer Review. J Med Internet Res. 2024;26:e57667. https:\/\/doi.org\/10.2196\/57667.","journal-title":"J Med Internet Res"},{"key":"10806_CR24","doi-asserted-by":"publisher","unstructured":"Verharen JPH. ChatGPT identifies gender disparities in scientific peer review. eLife. 2023;12:RP90230. https:\/\/doi.org\/10.7554\/eLife.90230.","DOI":"10.7554\/eLife.90230"},{"key":"10806_CR25","doi-asserted-by":"publisher","DOI":"10.1016\/j.chbah.2025.100145","volume":"4","author":"JQH Ho","year":"2025","unstructured":"Ho JQH, Hartanto A, Koh A, Majeed NM. Gender biases within Artificial Intelligence and ChatGPT: Evidence, Sources of Biases and Solutions. Comput Hum Behav Artif Hum. 2025;4:100145. https:\/\/doi.org\/10.1016\/j.chbah.2025.100145.","journal-title":"Comput Hum Behav Artif Hum"},{"key":"10806_CR26","doi-asserted-by":"publisher","unstructured":"An J, Huang D, Lin C, Tai M. Measuring gender and racial biases in large language models: Intersectional evidence from automated resume evaluation. PNAS Nexus. 2025;4:pgaf089. https:\/\/doi.org\/10.1093\/pnasnexus\/pgaf089.","DOI":"10.1093\/pnasnexus\/pgaf089"},{"key":"10806_CR27","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1007\/s10115-026-02731-3","volume":"68","author":"T Sachdeva","year":"2026","unstructured":"Sachdeva T, Singhal M, Khetarpaul S. Promoting fairness in LLMs: detection and mitigation of gender bias. Knowl Inf Syst. 2026;68:107. https:\/\/doi.org\/10.1007\/s10115-026-02731-3.","journal-title":"Knowl Inf Syst"},{"key":"10806_CR28","doi-asserted-by":"publisher","first-page":"2045","DOI":"10.1007\/s00146-023-01675-4","volume":"39","author":"S O\u2019Connor","year":"2024","unstructured":"O\u2019Connor S, Liu H. Gender bias\u00a0perpetuation and mitigation in AI technologies: challenges and opportunities. AI Soc. 2024;39:2045\u201357. https:\/\/doi.org\/10.1007\/s00146-023-01675-4.","journal-title":"AI Soc"},{"key":"10806_CR29","doi-asserted-by":"publisher","DOI":"10.2196\/54242","volume":"26","author":"M Achtari","year":"2024","unstructured":"Achtari M, Salihu A, Muller O, Abb\u00e9 E, Clair C, Schwarz J, et al. Gender Bias in AI\u2019s Perception of Cardiovascular Risk. J Med Internet Res. 2024;26:e54242. https:\/\/doi.org\/10.2196\/54242.","journal-title":"J Med Internet Res"},{"key":"10806_CR30","unstructured":"Bias is a Math Problem, AI Bias is a Technical Problem: 10-year Literature Review of AI\/LLM Bias Research Reveals Narrow [Gender-Centric] Conceptions of \\sayBias, and Academia-Industry Gap. n.d. https:\/\/arxiv.org\/html\/2508.11067v1?utm_source=chatgpt.com. Accessed 3 May 2026."},{"key":"10806_CR31","doi-asserted-by":"publisher","unstructured":"Pataranutaporn P, Powdthavee N, Achiwaranguprok C, Maes P. Can AI Solve the Peer Review Crisis? A Large Scale Cross Model Experiment of LLMs\u2019 Performance and Biases in Evaluating over 1000 Economics Papers; 2025. https:\/\/doi.org\/10.48550\/arXiv.2502.00070.","DOI":"10.48550\/arXiv.2502.00070"},{"key":"10806_CR32","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1001\/jama.2023.24641","volume":"331","author":"D von Wedel","year":"2024","unstructured":"von Wedel D, Schmitt RA, Thiele M, Leuner R, Shay D, Redaelli S, et al. Affiliation Bias in Peer Review of Abstracts by a Large Language Model. JAMA. 2024;331:252\u20133. https:\/\/doi.org\/10.1001\/jama.2023.24641.","journal-title":"JAMA"},{"key":"10806_CR33","doi-asserted-by":"publisher","unstructured":"Thakkar N, Yuksekgonul M, Silberg J, Garg A, Peng N, Sha F, et al. Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025; 2025. https:\/\/doi.org\/10.48550\/arXiv.2504.09737.","DOI":"10.48550\/arXiv.2504.09737"},{"key":"10806_CR34","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","volume":"33","author":"JR Landis","year":"1977","unstructured":"Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159\u201374.","journal-title":"Biometrics"},{"key":"10806_CR35","doi-asserted-by":"publisher","first-page":"871","DOI":"10.1177\/1536867X1801800408","volume":"18","author":"D Klein","year":"2018","unstructured":"Klein D. Implementing a General Framework for Assessing Interrater Agreement in Stata. Stata J. 2018;18:871\u2013901. https:\/\/doi.org\/10.1177\/1536867X1801800408.","journal-title":"Stata J"},{"key":"10806_CR36","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbusres.2025.115804","volume":"203","author":"LDW Thomas","year":"2026","unstructured":"Thomas LDW, Romasanta AKG, Pujol Priego L. Jagged competencies: Measuring the reliability of generative AI in academic research. J Bus Res. 2026;203:115804. https:\/\/doi.org\/10.1016\/j.jbusres.2025.115804.","journal-title":"J Bus Res"},{"key":"10806_CR37","doi-asserted-by":"publisher","first-page":"1262","DOI":"10.3390\/electronics15061262","volume":"15","author":"L Huynh","year":"2026","unstructured":"Huynh L, McNamara DS. Evaluation of Linguistic Consistency of LLM-Generated Text Personalization Using Natural Language Processing. Electronics. 2026;15:1262. https:\/\/doi.org\/10.3390\/electronics15061262.","journal-title":"Electronics"},{"key":"10806_CR38","doi-asserted-by":"publisher","unstructured":"Wang JJ, Wang VX. Assessing Consistency and Reproducibility in the Outputs of Large Language Models: Evidence Across Diverse Finance and Accounting Tasks; 2025. https:\/\/doi.org\/10.48550\/arXiv.2503.16974.","DOI":"10.48550\/arXiv.2503.16974"}],"container-title":["Journal of General Internal Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11606-026-10806-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11606-026-10806-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11606-026-10806-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T15:25:53Z","timestamp":1790954753000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11606-026-10806-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,2]]},"references-count":38,"alternative-id":["10806"],"URL":"https:\/\/doi.org\/10.1007\/s11606-026-10806-8","relation":{},"ISSN":["0884-8734","1525-1497"],"issn-type":[{"value":"0884-8734","type":"print"},{"value":"1525-1497","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,2]]},"assertion":[{"value":"16 May 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 August 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 October 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"During the preparation of this work the authors used ChatGPT (OpenAI) in order to improve language, grammar, and readability, and to assist with the drafting and revision of the manuscript. After using this tool\/service, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article.","order":1,"name":"Ethics","label":"Declaration of Generative AI and AI-Assisted Technologies in the Manuscript Preparation Process","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This study did not require ethical review, according to current Swiss law.","order":2,"name":"Ethics","label":"Ethical Approval","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they do not have a conflict of interest.","order":3,"name":"Ethics","label":"Conflict of interest","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":4,"name":"Ethics","label":"Clinical Trial Number","group":{"name":"EthicsHeading","label":"Declarations"}}]}}