{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T17:52:58Z","timestamp":1772214778021,"version":"3.50.1"},"reference-count":84,"publisher":"IOP Publishing","issue":"3","license":[{"start":{"date-parts":[[2025,9,2]],"date-time":"2025-09-02T00:00:00Z","timestamp":1756771200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2025,9,2]],"date-time":"2025-09-02T00:00:00Z","timestamp":1756771200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"name":"NSF CS","award":["CCF2106707"],"award-info":[{"award-number":["CCF2106707"]}]},{"name":"NSF AST","award":["2307109"],"award-info":[{"award-number":["2307109"]}]},{"DOI":"10.13039\/100006151","name":"Basic Energy Sciences","doi-asserted-by":"crossref","award":["DE-SC0017647"],"award-info":[{"award-number":["DE-SC0017647"]}],"id":[{"id":"10.13039\/100006151","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100001395","name":"Wisconsin Alumni Research Foundation","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100001395","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Mach. Learn.: Sci. Technol."],"published-print":{"date-parts":[[2025,9,30]]},"abstract":"Abstract<\/jats:title>\n \n We introduce a benchmark to evaluate the capability of AI to solve problems in theoretical physics (TP), focusing on high-energy theory and cosmology. The first iteration of our benchmark consists of 57 problems of varying difficulty, from undergraduate to research level. These problems are novel in the sense that they do not come from public problem collections. We evaluate our data set on various open and closed language models, including o3-mini, o1, DeepSeek-R1, GPT-4o and versions of Llama and Qwen. While we find impressive progress in model performance with the most recent models, our research-level difficulty problems are mostly unsolved. We address challenges of auto-verifiability and grading, and discuss common failure modes. While currently state-of-the art models are still of limited use for researchers, our results show that AI assisted TP research may become possible in the near future. We discuss the main obstacles towards this goal and possible strategies to overcome them. The public problems and solutions, results for various models, and updates to the data set and score distribution, are available on the website of the dataset\n tpbench.org<\/jats:ext-link>\n .\n <\/jats:p>","DOI":"10.1088\/2632-2153\/adfcb0","type":"journal-article","created":{"date-parts":[[2025,8,18]],"date-time":"2025-08-18T22:51:48Z","timestamp":1755557508000},"page":"030505","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Theoretical physics benchmark (TPBench)\u2014a dataset and study of AI reasoning capabilities in theoretical physics"],"prefix":"10.1088","volume":"6","author":[{"given":"Daniel J H","family":"Chung","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0006-4989-3753","authenticated-orcid":true,"given":"Zhiqi","family":"Gao","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0002-4720-1320","authenticated-orcid":false,"given":"Yurii","family":"Kvasiuk","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9545-8556","authenticated-orcid":false,"given":"Tianyi","family":"Li","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3777-7791","authenticated-orcid":true,"given":"Moritz","family":"M\u00fcnchmeyer","sequence":"additional","affiliation":[]},{"given":"Maja","family":"Rudolph","sequence":"additional","affiliation":[]},{"given":"Frederic","family":"Sala","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9947-4748","authenticated-orcid":false,"given":"Sai Chaitanya","family":"Tadepalli","sequence":"additional","affiliation":[]}],"member":"266","published-online":{"date-parts":[[2025,9,2]]},"reference":[{"key":"mlstadfcb0bib1","article-title":"Measuring mathematical problem solving with the math dataset","author":"Hendrycks","year":"2021","type":"preprint"},{"key":"mlstadfcb0bib2","article-title":"FrontierMath: a benchmark for evaluating advanced mathematical reasoning in AI","author":"Glazer","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib3","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2023.emnlp-main.468","type":"preprint","article-title":"Have LLMs advanced enough? A challenging problem solving benchmark for large language models","author":"Arora","year":"2023"},{"key":"mlstadfcb0bib4","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2024.acl-long.211","type":"preprint","article-title":"OlympiadBench: a challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems","author":"He","year":"2024"},{"key":"mlstadfcb0bib5","article-title":"Improving physics reasoning in large language models using mixture of refinement agents","author":"Jaiswal","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib6","article-title":"Quantum many-body physics calculations with large language models","author":"Pan","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib7","doi-asserted-by":"crossref","DOI":"10.70777\/si.v2i1.13973","type":"preprint","article-title":"Humanity\u2019s last exam","author":"Phan","year":"2025"},{"key":"mlstadfcb0bib8","doi-asserted-by":"publisher","first-page":"846","DOI":"10.1103\/PhysRevD.50.846","type":"journal-article","article-title":"Some properties of Noether charge and a proposal for dynamical black hole entropy","volume":"50","author":"Iyer","year":"1994","journal-title":"Phys. Rev. D"},{"key":"mlstadfcb0bib9","doi-asserted-by":"publisher","first-page":"1739","DOI":"10.1063\/1.1664507","type":"journal-article","article-title":"Spinor structure of space-times in general relativity. I","volume":"9","author":"Geroch","year":"1968","journal-title":"J. Math. Phys."},{"key":"mlstadfcb0bib10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/BF02099526","type":"journal-article","article-title":"Intersection theory on the moduli space of curves and the matrix Airy function","volume":"147","author":"Kontsevich","year":"1992","journal-title":"Commun. Math. Phys."},{"key":"mlstadfcb0bib11","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1007\/BF01942062","type":"journal-article","article-title":"Proof of the positive mass theorem. 2","volume":"79","author":"Schon","year":"1981","journal-title":"Commun. Math. Phys."},{"key":"mlstadfcb0bib12","article-title":"Quiver Hecke algebras from Floer homology in Couloumb branches","author":"Aganagic","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib13","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1007\/BF01208569","type":"journal-article","article-title":"On Witten\u2019s proof of the positive energy theorem","volume":"84","author":"Parker","year":"1982","journal-title":"Commun. Math. Phys."},{"key":"mlstadfcb0bib14","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1007\/s00222-003-0286-7","type":"journal-article","article-title":"Mirror symmetry, Langlands duality and the Hitchin system","volume":"153","author":"Hausel","year":"2003","journal-title":"Invent. Math."},{"key":"mlstadfcb0bib15","author":"Hardy","year":"1940","type":"book"},{"key":"mlstadfcb0bib16","first-page":"22","type":"journal-article","article-title":"Ten lessons i wish i had been taught","volume":"44","author":"Rota","year":"1997","journal-title":"Not. Am. Math. Soc."},{"key":"mlstadfcb0bib17","article-title":"Can LLMs generate novel research ideas? A large-scale human study with 100+ NLP researchers","author":"Si","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib18","author":"P\u00f3lya","year":"1945","type":"book"},{"key":"mlstadfcb0bib19","article-title":"GPT-4o","author":"OpenAI","year":"2024","type":"other"},{"key":"mlstadfcb0bib20","article-title":"Introducing OpenAI o1-preview","author":"OpenAI","year":"2024","type":"other"},{"key":"mlstadfcb0bib21","article-title":"o3-mini","author":"OpenAI","year":"2025","type":"other"},{"key":"mlstadfcb0bib22","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2023.acl-industry.4","type":"preprint","article-title":"Mathprompter: mathematical reasoning using large language models","author":"Imani","year":"2023"},{"key":"mlstadfcb0bib23","article-title":"Large language models cannot self-correct reasoning yet","author":"Huang","year":"2023","type":"preprint"},{"key":"mlstadfcb0bib24","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2023.findings-acl.551","type":"preprint","article-title":"Do large language models know what they don\u2019t know?","author":"Yin","year":"2023"},{"key":"mlstadfcb0bib25","doi-asserted-by":"crossref","DOI":"10.1162\/tacl_a_00713","type":"preprint","article-title":"When can LLMs actually correct their own mistakes? A critical survey of self-correction of LLMs","author":"Kamoi","year":"2024"},{"key":"mlstadfcb0bib26","article-title":"Chain-of-verification reduces hallucination in large language models","author":"Dhuliawala","year":"2023","type":"preprint"},{"key":"mlstadfcb0bib27","article-title":"SAC3: reliable hallucination detection in black-box language models via semantic-aware cross-check consistency","author":"Zhang","year":"2023","type":"preprint"},{"key":"mlstadfcb0bib28","doi-asserted-by":"crossref","DOI":"10.24963\/ijcai.2024\/381","type":"preprint","article-title":"Llms can find mathematical reasoning mistakes by pedagogical chain-of-thought","author":"Jiang","year":"2024"},{"key":"mlstadfcb0bib29","article-title":"A tale of two fields: neural network-enhanced non-gaussianity search with halos","author":"Kvasiuk","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib30","doi-asserted-by":"publisher","first-page":"JHEP12(2022)108","DOI":"10.1007\/JHEP12(2022)108","type":"journal-article","article-title":"Quantum interference in gravitational particle production","author":"Basso","year":"2022","journal-title":"J. High Energy Phys."},{"key":"mlstadfcb0bib31","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1016\/j.cpc.2016.08.019","type":"journal-article","article-title":"TikZ-Feynman: Feynman diagrams with TikZ","volume":"210","author":"Ellis","year":"2017","journal-title":"Comput. Phys. Commun."},{"key":"mlstadfcb0bib32","article-title":"Meta Llama 3.1","author":"Meta AI","year":"2024","type":"other"},{"key":"mlstadfcb0bib33","article-title":"Qwen2.5","author":"Qwen Team","year":"2024","type":"other"},{"key":"mlstadfcb0bib34","article-title":"Qwen QwQ 32B preview","author":"Qwen Team","year":"2024","type":"other"},{"key":"mlstadfcb0bib35","article-title":"DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning","author":"Guo","year":"2025","type":"preprint"},{"key":"mlstadfcb0bib36","article-title":"DeepSeek LLM: scaling open-source language models with longtermism","author":"Bi","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib37","article-title":"Humans or LLMs as the judge? A study on judgement biases","author":"Chen","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib38","doi-asserted-by":"crossref","DOI":"10.1119\/5.0182627","type":"preprint","article-title":"ChatGPT-4 with code interpreter can be used to solve introductory college-level vector calculus and electromagnetism problems","author":"Kumar","year":"2023"},{"key":"mlstadfcb0bib39","article-title":"MathChat: converse to tackle challenging math problems with LLM agents","author":"Wu","year":"2023","type":"preprint"},{"key":"mlstadfcb0bib40","article-title":"Test-time scaling techniques in theoretical physics\u2014a comparison of methods on the TPBench dataset","author":"Gao","year":"2025","type":"preprint"},{"key":"mlstadfcb0bib41","doi-asserted-by":"publisher","author":"Peskin","year":"1995","DOI":"10.1201\/9780429503559","type":"book"},{"key":"mlstadfcb0bib42","doi-asserted-by":"publisher","author":"Bailin","year":"1994","DOI":"10.1201\/9780367805807","type":"book"},{"key":"mlstadfcb0bib43","doi-asserted-by":"crossref","DOI":"10.1119\/1.13992","type":"book","article-title":"The mathematical theory of black holes","author":"Chandrasekhar","year":"1985"},{"key":"mlstadfcb0bib44","article-title":"Training verifiers to solve math word problems","author":"Cobbe","year":"2021","type":"preprint"},{"key":"mlstadfcb0bib45","first-page":"pp 158","type":"conference-proceedings","article-title":"Program induction by rationale generation: learning to solve and explain algebraic word problems","author":"Ling","year":"2017"},{"key":"mlstadfcb0bib46","article-title":"PutnamBench: evaluating neural theorem-provers on the Putnam mathematical competition","author":"Tsoukalas","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib47","article-title":"MiniF2F: a cross-system benchmark for formal olympiad-level mathematics","author":"Zheng","year":"2022","type":"preprint"},{"key":"mlstadfcb0bib48","article-title":"FIMO: a challenge formal dataset for automated theorem proving","author":"Liu","year":"2023","type":"preprint"},{"key":"mlstadfcb0bib49","article-title":"ProofNet: autoformalizing and formally proving undergraduate-level mathematics","author":"Azerbayev","year":"2023","type":"preprint"},{"key":"mlstadfcb0bib50","article-title":"GPQA: a graduate-level Google-proof Q&A benchmark","author":"Rein","year":"2023","type":"preprint"},{"key":"mlstadfcb0bib51","article-title":"GSM-symbolic: understanding the limitations of mathematical reasoning in large language models","author":"Mirzadeh","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib52","article-title":"Kimi k1. 5: scaling reinforcement learning with LLMs","author":"Team","year":"2025","type":"preprint"},{"key":"mlstadfcb0bib53","article-title":"RedStar: does scaling long-CoT data unlock better slow-reasoning systems?","author":"Xu","year":"2025","type":"preprint"},{"key":"mlstadfcb0bib54","article-title":"Bespoke-Stratos: the unreasonable effectiveness of reasoning distillation","author":"Bespoke Labs","year":"2025","type":"other"},{"key":"mlstadfcb0bib55","article-title":"MetaMath: bootstrap your own mathematical questions for large language models","author":"Yu","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib56","first-page":"pp 15476","type":"conference-proceedings","article-title":"STaR: bootstrapping reasoning with reasoning","volume":"vol 35","author":"Zelikman","year":"2022"},{"key":"mlstadfcb0bib57","article-title":"Quiet-STaR: language models can teach themselves to think before speaking","author":"Zelikman","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib58","article-title":"DeepSeekMath: pushing the limits of mathematical reasoning in open language models","author":"Shao","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib59","article-title":"Self-play fine-tuning converts weak language models to strong language models","author":"Chen","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib60","article-title":"Chain-of-thought prompting elicits reasoning in large language models","author":"Wei","year":"2023","type":"preprint"},{"key":"mlstadfcb0bib61","article-title":"Scaling LLM test-time compute optimally can be more effective than scaling model parameters","author":"Snell","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib62","article-title":"From decoding to meta-generation: inference-time algorithms for large language models","author":"Welleck","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib63","article-title":"s1: simple test-time scaling","author":"Muennighoff","year":"2025","type":"preprint"},{"key":"mlstadfcb0bib64","article-title":"Decomposed prompting: a modular approach for solving complex tasks","author":"Khot","year":"2022","type":"preprint"},{"key":"mlstadfcb0bib65","article-title":"Least-to-most prompting enables complex reasoning in large language models","author":"Zhou","year":"2022","type":"preprint"},{"key":"mlstadfcb0bib66","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2023.emnlp-main.507","type":"preprint","article-title":"Reasoning with language model is planning with world model","author":"Hao","year":"2023"},{"key":"mlstadfcb0bib67","article-title":"Take a step back: evoking reasoning via abstraction in large language models","author":"Zheng","year":"2023","type":"preprint"},{"key":"mlstadfcb0bib68","article-title":"Let\u2019s verify step by step","author":"Lightman","year":"2023","type":"preprint"},{"key":"mlstadfcb0bib69","article-title":"Self-evaluation improves selective generation in large language models","author":"Ren","year":"2023","type":"preprint"},{"key":"mlstadfcb0bib70","article-title":"Boosting of thoughts: trial-and-error problem solving with large language models","author":"Chen","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib71","type":"conference-proceedings","article-title":"Self-refine: iterative refinement with self-feedback","volume":"vol 36","author":"Madaan","year":"2024"},{"key":"mlstadfcb0bib72","article-title":"Analyzing the performance of self-refine on different large language models","author":"Forsman","year":"2024","type":"other"},{"key":"mlstadfcb0bib73","article-title":"Theoretical guarantees on the best-of-n alignment policy","author":"Beirami","year":"2025","type":"preprint"},{"key":"mlstadfcb0bib74","article-title":"Self-consistency improves chain of thought reasoning in language models","author":"Wang","year":"2023","type":"conference-proceedings"},{"key":"mlstadfcb0bib75","first-page":"pp 11809","type":"conference-proceedings","article-title":"Tree of thoughts: deliberate problem solving with large language models","volume":"vol 36","author":"Yao","year":"2023"},{"key":"mlstadfcb0bib76","article-title":"Mutual reasoning makes smaller LLMs stronger problem-solvers","author":"Qi","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib77","article-title":"LLaMA-Berry: pairwise optimization for O1-like olympiad-level mathematical reasoning","author":"Zhang","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib78","article-title":"MindStar: enhancing math reasoning in pre-trained LLMs at inference time","author":"Kang","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib79","first-page":"pp 282","type":"conference-proceedings","article-title":"Bandit based monte-carlo planning","author":"Kocsis","year":"2006"},{"key":"mlstadfcb0bib80","type":"conference-proceedings","article-title":"Toolformer: language models can teach themselves to use tools","volume":"vol 36","author":"Schick","year":"2024"},{"key":"mlstadfcb0bib81","article-title":"Archon: an architecture search framework for inference-time techniques","author":"Saad-Falcon","year":"2024","type":"preprint"},{"key":"mlstadfcb0bib82","article-title":"Program of thoughts prompting: disentangling computation from reasoning for numerical reasoning tasks","author":"Chen","year":"2022","type":"preprint"},{"key":"mlstadfcb0bib83","article-title":"Retrieval-augmented generation for large language models: a survey","author":"Gao","year":"2023","type":"preprint"},{"key":"mlstadfcb0bib84","doi-asserted-by":"publisher","first-page":"666","DOI":"10.1103\/PhysRevLett.71.666","type":"journal-article","article-title":"Entropy and area","volume":"71","author":"Srednicki","year":"1993","journal-title":"Phys. Rev. Lett."}],"container-title":["Machine Learning: Science and Technology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adfcb0","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adfcb0\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adfcb0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adfcb0\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adfcb0\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adfcb0\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adfcb0\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adfcb0\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,2]],"date-time":"2025-09-02T12:05:26Z","timestamp":1756814726000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/2632-2153\/adfcb0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,2]]},"references-count":84,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,9,2]]},"published-print":{"date-parts":[[2025,9,30]]}},"URL":"https:\/\/doi.org\/10.1088\/2632-2153\/adfcb0","relation":{"has-review":[{"id-type":"doi","id":"10.1088\/2632-2153\/ADFCB0\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/ADFCB0\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/ADFCB0\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/ADFCB0\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1088\/2632-2153\/ADFCB0\/v1\/review1","asserted-by":"object"}]},"ISSN":["2632-2153"],"issn-type":[{"value":"2632-2153","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,2]]},"assertion":[{"value":"Theoretical physics benchmark (TPBench)\u2014a dataset and study of AI reasoning capabilities in theoretical physics","name":"article_title","label":"Article Title"},{"value":"Machine Learning: Science and Technology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2025 The Author(s). Published by IOP Publishing Ltd","name":"copyright_information","label":"Copyright Information"},{"value":"2025-04-06","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-08-18","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2025-09-02","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}