{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T19:53:56Z","timestamp":1792785236625,"version":"4.5.0"},"reference-count":34,"publisher":"National Academy of Sciences","issue":"41","license":[{"start":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T00:00:00Z","timestamp":1791158400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000006","name":"DOW | USN | Office of Naval Research","doi-asserted-by":"publisher","award":["N000142512346"],"award-info":[{"award-number":["N000142512346"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2200410"],"award-info":[{"award-number":["2200410"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1942124"],"award-info":[{"award-number":["1942124"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"AI2050 program at Schmidt Sciences","award":["24-66924"],"award-info":[{"award-number":["24-66924"]}],"id":[{"id":"https:\/\/ror.org\/044fk6795","id-type":"ROR","asserted-by":"crossref"}]}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2026,10,13]]},"abstract":"<jats:p>The rapid rise of autonomous AI scientists marks a paradigm shift in scientific discovery by automating the research lifecycle. Yet their rushed development has outpaced critical oversight, leaving key workflow decisions dangerously unscrutinized. We present a much-needed systematic analysis of open-source AI scientist systems, investigating four primary pitfalls: inappropriate benchmark selection, data leakage, metric misuse, and post hoc selection bias. Through controlled experiments that isolate each pitfall, we find systematic vulnerabilities across two representative open-source systems. Crucially, we find that these flaws are largely invisible at the level of the final manuscript, suggesting that current manuscript-centric peer review paradigms are fundamentally insufficient for ensuring the integrity of automated research. We further propose mitigation strategies and demonstrate that access to full workflow artifacts (log traces and code) enables more effective auditing. Our findings suggest that journals, conferences, and researchers should move beyond manuscript-only evaluation toward process auditing the end-to-end workflow artifacts of AI scientist systems.<\/jats:p>","DOI":"10.1073\/pnas.2610214123","type":"journal-article","created":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T19:03:46Z","timestamp":1791227026000},"update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":1,"title":["The more you automate, the less you see: Hidden pitfalls of autonomous AI scientists"],"prefix":"10.1073","volume":"123","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-0310-677X","authenticated-orcid":false,"given":"Ziming","family":"Luo","sequence":"first","affiliation":[{"id":[{"id":"https:\/\/ror.org\/05x2bcf33","id-type":"ROR","asserted-by":"publisher"}],"name":"Machine Learning Department, School of Computer Science, Carnegie Mellon University","place":["Pittsburgh"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Atoosa","family":"Kasirzadeh","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/05x2bcf33","id-type":"ROR","asserted-by":"publisher"}],"name":"Department of Philosophy, Dietrich College of Humanities and Social Sciences, Carnegie Mellon University","place":["Pittsburgh"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5158-9677","authenticated-orcid":false,"given":"Nihar B.","family":"Shah","sequence":"additional","affiliation":[{"id":[{"id":"https:\/\/ror.org\/05x2bcf33","id-type":"ROR","asserted-by":"publisher"}],"name":"Machine Learning Department, School of Computer Science, Carnegie Mellon University","place":["Pittsburgh"]},{"id":[{"id":"https:\/\/ror.org\/05x2bcf33","id-type":"ROR","asserted-by":"publisher"}],"name":"Computer Science Department, School of Computer Science, Carnegie Mellon University","place":["Pittsburgh"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"341","published-online":{"date-parts":[[2026,10,5]]},"reference":[{"key":"e_1_3_4_1_2","unstructured":"Z. Luo Z. Yang Z. Xu W. Yang X. Du LLM4SR: A survey on large language models for scientific research. arXiv [Preprint] (2025). https:\/\/arxiv.org\/abs\/2501.04306 (Accessed 15 June 2025)."},{"key":"e_1_3_4_2_2","unstructured":"Y. Yamada The AI Scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv [Preprint] (2025). https:\/\/doi.org\/10.48550\/arXiv.2504.08066 (Accessed 15 June 2025)."},{"key":"e_1_3_4_3_2","unstructured":"Intology AI Zochi technical report (2025). https:\/\/www.intology.ai\/blog\/zochi-tech-report. Accessed: 15 June 2025."},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41587-025-02963-8"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-025-63913-1"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-023-02980-0"},{"key":"e_1_3_4_7_2","doi-asserted-by":"crossref","unstructured":"S. Schmidgall Agent laboratory: Using LLM agents as research assistants. arXiv [Preprint] (2025). https:\/\/doi.org\/10.48550\/arXiv.2501.04227 (Accessed 15 June 2025).","DOI":"10.18653\/v1\/2025.findings-emnlp.320"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-7799(02)01985-6"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1086\/700070"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1165620"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1186\/1759-4499-2-1"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12862"},{"key":"e_1_3_4_13_2","unstructured":"C. Lu The AI scientist: Towards fully automated open-ended scientific discovery. arXiv [Preprint] (2024). https:\/\/doi.org\/10.48550\/arXiv.2408.06292 (Accessed 15 June 2025)."},{"key":"e_1_3_4_14_2","unstructured":"AutoScience Meet Carl: The first AI system to produce academically peer-reviewed research (2025). https:\/\/www.autoscience.ai\/blog\/meet-carl-the-first-ai-system-to-produce-academically-peer-reviewed-research. Accessed 15 June 2025."},{"key":"e_1_3_4_15_2","unstructured":"Team InternAgent et al. InternAgent: When agent becomes the scientist \u2013 Building closed-loop system from hypothesis to verification. arXiv [Preprint] (2025). https:\/\/doi.org\/10.48550\/arXiv.2505.16938. Accessed: 17 September 2026."},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-026-10652-y"},{"key":"e_1_3_4_17_2","unstructured":"N. B. Shah Challenges experiments and computational solutions in peer review (Communications of the ACM) (2022). https:\/\/www.cs.cmu.edu\/nihars\/preprints\/SurveyPeerReview.pdf. Extended Version. Accessed 15 June 2025."},{"key":"e_1_3_4_18_2","unstructured":"P. V. Coveney S. Succi The wall confronting large language models. arXiv [Preprint] (2025). https:\/\/arxiv.org\/abs\/2507.19703 (Accessed 15 June 2025)."},{"key":"e_1_3_4_19_2","unstructured":"G. Son When AI co-scientists fail: Spot-a benchmark for automated verification of scientific research. arXiv [Preprint] (2025). https:\/\/arxiv.org\/abs\/2505.11855 (Accessed 15 June 2025)."},{"key":"e_1_3_4_20_2","unstructured":"S. R. Javaji Can AI validate science? Benchmarking LLMs for accurate scientific claim \u2192 evidence reasoning. arXiv [Preprint] (2025). https:\/\/doi.org\/10.48550\/arXiv.2506.08235 (Accessed 15 June 2025)."},{"key":"e_1_3_4_21_2","doi-asserted-by":"crossref","unstructured":"X. Tang et al. Risks of AI scientists: prioritizing safeguarding over autonomy. Nat. Commun. 16 8317 (2025).","DOI":"10.1038\/s41467-025-63913-1"},{"key":"e_1_3_4_22_2","doi-asserted-by":"crossref","unstructured":"T. Gupta D. Pruthi \u201cAll that glitters is not novel: Plagiarism in AI generated research\u201d in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Association for Computational Linguistics Vienna Austria 2025) pp. 25721\u201325738.","DOI":"10.18653\/v1\/2025.acl-long.1249"},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-025-02616-5"},{"key":"e_1_3_4_24_2","doi-asserted-by":"crossref","unstructured":"H. Shin Mind the blind spots: A focus-level evaluation framework for LLM reviews. arXiv [Preprint] (2025). https:\/\/arxiv.org\/abs\/2502.17086 (Accessed 15 June 2025).","DOI":"10.18653\/v1\/2025.emnlp-main.1805"},{"key":"e_1_3_4_25_2","unstructured":"M. D\u2019Arcy T. Hope L. Birnbaum D. Downey Marg: Multi-agent review generation for scientific papers. arXiv [Preprint] (2024). https:\/\/arxiv.org\/abs\/2401.04259 (Accessed 15 June 2025)."},{"key":"e_1_3_4_26_2","unstructured":"R. Liu N. B. Shah ReviewerGPT? An exploratory study on using large language models for paper reviewing. arXiv [Preprint] (2023). https:\/\/arxiv.org\/abs\/2306.00622 (Accessed 15 June 2025)."},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11192-025-05287-1"},{"key":"e_1_3_4_28_2","unstructured":"M. P. Chitale K. M. Shetye H. Gupta M. Chaudhary V. Varma AutoRev: Automatic peer review system for academic research papers. arXiv [Preprint] (2025). https:\/\/arxiv.org\/abs\/2505.14376 (Accessed 15 June 2025)."},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-025-02172-y"},{"key":"e_1_3_4_30_2","unstructured":"V. Rao A. Kumar H. Lakkaraju N. B. Shah Detecting LLM-generated peer reviews. arXiv [Preprint] (2025). https:\/\/arxiv.org\/abs\/2503.15772 (Accessed 15 June 2025)."},{"key":"e_1_3_4_31_2","doi-asserted-by":"crossref","unstructured":"T. L. Lin Breaking the reviewer: Assessing the vulnerability of large language models in automated peer review under textual adversarial attacks. arXiv [Preprint] (2025). https:\/\/arxiv.org\/abs\/2506.11113 (Accessed 15 June 2025).","DOI":"10.18653\/v1\/2025.findings-emnlp.259"},{"key":"e_1_3_4_32_2","unstructured":"T. Eisenhofer No more reviewer# 2: Subverting automatic paper-reviewer assignment using adversarial learning. arXiv [Preprint] (2023). https:\/\/arxiv.org\/abs\/2303.14443 (Accessed 15 June 2025)."},{"key":"e_1_3_4_33_2","unstructured":"J. Y. J. Hsieh A. Raghunathan N. B. Shah \u201cVulnerability of text-matching in ML\/AI conference reviewer assignments to collusions\u201d in 34th USENIX Security Symposium (USENIX Security 25) (2025) pp. 5189\u20135208."},{"key":"e_1_3_4_34_2","unstructured":"Z. Luo A. Kasirzadeh N. B. Shah AIScientistPitfalls: Code and data accompanying \u201cThe more you automate the less you see: Hidden pitfalls of autonomous AI scientists.\u201d GitHub. https:\/\/github.com\/niharshah\/AIScientistPitfalls. Accessed: 17 September 2026."}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.2610214123","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T19:04:00Z","timestamp":1791227040000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/10.1073\/pnas.2610214123"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,5]]},"references-count":34,"journal-issue":{"issue":"41","published-print":{"date-parts":[[2026,10,13]]}},"alternative-id":["10.1073\/pnas.2610214123"],"URL":"https:\/\/doi.org\/10.1073\/pnas.2610214123","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,5]]},"assertion":[{"value":"2026-03-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-09-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-10-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2610214123"}}