{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,1]],"date-time":"2026-11-01T06:36:51Z","timestamp":1793515011350,"version":"4.5.0"},"reference-count":94,"publisher":"National Academy of Sciences","issue":"13","license":[{"start":{"date-parts":[[2025,3,26]],"date-time":"2025-03-26T00:00:00Z","timestamp":1742947200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2025,4]]},"abstract":"<jats:p>AI systems have attained superhuman performance across various domains. If the hidden knowledge encoded in these highly capable systems can be leveraged, human knowledge and performance can be advanced. Yet, this internal knowledge is difficult to extract. Due to the vast space of possible internal representations, searching for meaningful new conceptual knowledge can be like finding a needle in a haystack. Here, we introduce a method that extracts new chess concepts from AlphaZero, an AI system that mastered chess via self-play without human supervision. Our method excavates vectors that represent concepts from AlphaZero\u2019s internal representations using convex optimization, and filters the concepts based on teachability (whether the concept is transferable to another AI agent) and novelty (whether the concept contains information not present in human chess games). These steps ensure that the discovered concepts are useful and meaningful. For the resulting set of concepts, prototypes (chess puzzle\u2013solution pairs) are presented to experts for final validation. In a preliminary human study, four top chess grandmasters (all former or current world chess champions) were evaluated on their ability to solve concept prototype positions. All grandmasters showed improvement after the learning phase, suggesting that the concepts are at the frontier of human understanding. Despite the small scale, our result is a proof of concept demonstrating the possibility of leveraging knowledge from a highly capable AI system to advance the frontier of human knowledge; a development that could bear profound implications and shape how we interact with AI systems across many applications.<\/jats:p>","DOI":"10.1073\/pnas.2406675122","type":"journal-article","created":{"date-parts":[[2025,3,26]],"date-time":"2025-03-26T13:42:05Z","timestamp":1742996525000},"update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":18,"title":["Bridging the human\u2013AI knowledge gap through concept discovery and transfer in AlphaZero"],"prefix":"10.1073","volume":"122","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-6760-5879","authenticated-orcid":false,"given":"Lisa","family":"Schut","sequence":"first","affiliation":[{"name":"Oxford Applied and Theoretical Machine Learning Group","place":["Oxford, United Kingdom"]},{"name":"Department of Computer Science","place":["Oxford, United Kingdom"]},{"name":"University of Oxford","place":["Oxford, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nenad","family":"Toma\u0161ev","sequence":"additional","affiliation":[{"name":"Google DeepMind","place":["London, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4208-6888","authenticated-orcid":false,"given":"Thomas","family":"McGrath","sequence":"additional","affiliation":[{"name":"Goodfire AI","place":["San Francisco"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Demis","family":"Hassabis","sequence":"additional","affiliation":[{"name":"Google DeepMind","place":["London, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ulrich","family":"Paquet","sequence":"additional","affiliation":[{"name":"Google DeepMind","place":["London, United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9938-2915","authenticated-orcid":false,"given":"Been","family":"Kim","sequence":"additional","affiliation":[{"name":"Google DeepMind","place":["Mountain View"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"341","published-online":{"date-parts":[[2025,3,26]]},"reference":[{"key":"e_1_3_4_1_2","unstructured":"J. Adebayo et al. \u201cSanity checks for saliency maps\u201d in Proceedings of the International Conference on Advances in Neural Information Processing Systems 31 S. Bengio et al. Eds. (Curran Associates Inc. Red Hook NY 2018) pp. 9525\u20139536."},{"key":"e_1_3_4_2_2","unstructured":"W. Nie Y. Zhang \u201cA. Patel A theoretical explanation for perplexing behaviors of backpropagation-based visualizations\u201d in Proceedings of the International Conference on Machine Learning J. Dy A. Krause Eds. (PMLR 2018) pp 3809\u20133818."},{"key":"e_1_3_4_3_2","doi-asserted-by":"crossref","unstructured":"B. Bilodeau N. Jaques P. W. Koh B. Kim Impossibility theorems for feature attribution. Proc. Natl. Acad. Sci. U.S.A. 121 e2304406120 (2024).","DOI":"10.1073\/pnas.2304406120"},{"key":"e_1_3_4_4_2","unstructured":"Wired In two moves Alphago and Lee Sedol redefined future (2016). https:\/\/www.wired.com\/2016\/03\/two-moves-alphago-lee-sedol-redefined-future\/. Accessed 29 Jan 2025."},{"key":"e_1_3_4_5_2","unstructured":"Wikipedia contributors Chess rating system. https:\/\/en.wikipedia.org\/wiki\/Chess_rating_system. Accessed 23 July 2023."},{"key":"e_1_3_4_6_2","doi-asserted-by":"crossref","unstructured":"D. Silver et al. A general reinforcement learning algorithm that masters chess shogi and Go through self-play. Science 362 1140\u20131144 (2018).","DOI":"10.1126\/science.aar6404"},{"key":"e_1_3_4_7_2","volume-title":"Mind in Society: The Development of Higher Psychological Processes","author":"Vygotsky L. S.","year":"1978","unstructured":"L. S. Vygotsky, Mind in Society: The Development of Higher Psychological Processes (Cambridge, Mass: Harvard University Press, 1978)."},{"key":"e_1_3_4_8_2","unstructured":"B. Kim \u201cBeyond interpretability: developing a language to shape our relationships with AI\u201d in Keynote talk at 10th International Conference on Learning Representations Virtual (2023)."},{"key":"e_1_3_4_9_2","unstructured":"E. Margolis S. Laurence \u201cConcepts\u201d in The Stanford Encyclopedia of Philosophy E. N. Zalta U. Nodelman Eds. (Metaphysics Research Lab Stanford University 2023)."},{"key":"e_1_3_4_10_2","unstructured":"Y. Goyal A. Feder U. Shalit B. Kim Explaining Classifiers with Causal Concept Effect (CACE). arXiv [Preprint] (2019). https:\/\/arxiv.org\/abs\/1907.07165 (Accessed 1 November 2023)."},{"key":"e_1_3_4_11_2","doi-asserted-by":"crossref","unstructured":"Oxford English Dictionary Concept (n) Sense I.2. (2024). https:\/\/doi.org\/10.1093\/OED\/2908737873. Accessed 5 August 2024.","DOI":"10.1093\/OED\/2908737873"},{"key":"e_1_3_4_12_2","doi-asserted-by":"crossref","unstructured":"T. McGrath et al. Acquisition of chess knowledge in AlphaZero. Proc. Natl. Acad. Sci. U.S.A. 119 e2206625119 (2022).","DOI":"10.1073\/pnas.2206625119"},{"key":"e_1_3_4_13_2","unstructured":"B. Kim et al. \u201cInterpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV)\u201d in International Conference on Machine Learning J. Dy A. Krause Eds. (PMLR 2018) pp. 2668\u20132677."},{"key":"e_1_3_4_14_2","unstructured":"W. Gurnee et al. Finding neurons in a haystack: Case studies with sparse probing. arXiv [Preprint] (2023). https:\/\/arxiv.org\/abs\/2305.01610 (Accessed 1 November 2023)."},{"key":"e_1_3_4_15_2","doi-asserted-by":"crossref","unstructured":"A. Conneau G. Kruszewski G. Lample L. Barrault M. Baroni \u201cWhat you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties\u201d in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) I. Gurevych Y. Miyao Eds. (Association for Computational Linguistics Melbourne Australia 2018) pp. 2126\u20132136.","DOI":"10.18653\/v1\/P18-1198"},{"key":"e_1_3_4_16_2","doi-asserted-by":"crossref","unstructured":"I. Tenney D. Das E. Pavlick \u201cBERT rediscovers the classical NLP pipeline\u201d in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics A. Korhonen D. Traum L. M\u00e0rquez Eds. (Association for Computational Linguistics Florence Italy 2019) pp. 4593\u20134601.","DOI":"10.18653\/v1\/P19-1452"},{"key":"e_1_3_4_17_2","unstructured":"N. Nanda Actually Othello-GPT Has A Linear Emergent World Model (2023). https:\/\/www.neelnanda.io\/mechanistic-interpretability\/othello. Accessed 1 November 2023."},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2008.277"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1996.tb02080.x"},{"key":"e_1_3_4_20_2","doi-asserted-by":"crossref","unstructured":"J. Schrittwieser et al. Mastering Atari Go chess and shogi by planning with a learned model. Nature 588 604\u2013609 (2020).","DOI":"10.1038\/s41586-020-03051-4"},{"key":"e_1_3_4_21_2","unstructured":"A. Ghandeharioun et al. DISSECT: Disentangled simultaneous explanations via concept traversals. arXiv [Preprint] (2021). https:\/\/arxiv.org\/abs\/2105.15164 (Accessed 1 November 2023)."},{"key":"e_1_3_4_22_2","doi-asserted-by":"crossref","unstructured":"X. Li et al. \u201cContrastive Decoding: Open-ended Text Generation as Optimization\u201d in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) A. Rogers J. Boyd-Graber N. Okazaki Eds. (Association for Computational Linguistics Toronto Canada 2023) pp. 12286\u201312312.","DOI":"10.18653\/v1\/2023.acl-long.687"},{"key":"e_1_3_4_23_2","unstructured":"A. Melekhina Interview with GM Chuchelov \u2013 Caruana\u2019s Coach. ChessBase (2014). https:\/\/en.chessbase.com\/post\/interview-with-gm-chuchelov-caruana-s-coach. Accessed 1 August 2023."},{"key":"e_1_3_4_24_2","volume-title":"How to Win at Chess","author":"King D.","year":"2000","unstructured":"D. King, How to Win at Chess (Everyman Chess, 2000)."},{"key":"e_1_3_4_25_2","doi-asserted-by":"crossref","unstructured":"D. Bau B. Zhou A. Khosla A. Oliva A. Torralba \u201cNetwork dissection: Quantifying interpretability of deep visual representations\u201d in Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition R. Chellappa Z. Zhang A. Hoogs Eds. (IEEE Honolulu HI 2017) pp. 6541\u20136549.","DOI":"10.1109\/CVPR.2017.354"},{"key":"e_1_3_4_26_2","unstructured":"D. Alvarez-Melis T. S. Jaakkola \u201cTowards robust interpretability with self-explaining neural networks\u201d in Advances in Neural Information Processing Systems 31 S. Bengio et al. Eds. (Curran Associates Inc. Red Hook NY 2018) pp. 7786\u20137795."},{"key":"e_1_3_4_27_2","unstructured":"P. W. Koh et al. \u201cConcept bottleneck models\u201d in Proceedings of the International Conference on Machine Learning 37 H. Daum\u00e9 III A. Singh Eds. (PMLR 2020) pp. 5338\u20135348."},{"key":"e_1_3_4_28_2","unstructured":"A. Bai C. K. Yeh N. Lin P. Ravikumar C. J. Hsieh \u201cConcept gradient: Concept-based interpretation without linear assumption\u201d in Proceedings of the International Conference on Learning Representations 10 Y. LeCun Ed. (Curran Associates Inc. Red Hook NY 2022)."},{"key":"e_1_3_4_29_2","doi-asserted-by":"crossref","unstructured":"R. Achtibat et al. From attribution maps to human-understandable explanations through Concept Relevance Propagation. Nat Mach Intell 5 1006\u20131019 (2023).","DOI":"10.1038\/s42256-023-00711-8"},{"key":"e_1_3_4_30_2","unstructured":"J. Crabb\u00e9 M. van der Schaar \u201cConcept activation regions: A generalized framework for concept-based explanations\u201d in Advances in Neural Information Processing Systems S. Koyejo et al. Eds. (Curran Associates Inc. 2022) vol. 35 pp. 2590\u20132607."},{"key":"e_1_3_4_31_2","doi-asserted-by":"crossref","unstructured":"M. Graziani V. Andrearczyk H. M\u00fcller \u201cRegression concept vectors for bidirectional explanations in histopathology\u201d in Understanding and Interpreting Machine Learning in Medical Image Computing Applications D. Stoyanov et al. Eds. (Springer 2018) pp. 124\u2013132.","DOI":"10.1007\/978-3-030-02628-8_14"},{"key":"e_1_3_4_32_2","unstructured":"C. Sprague E. B. Wendoloski I. Guch \u201cInterpretable AI for deep learning-based meteorological applications\u201d in American Meteorological Society Annual Meeting (American Meteorological Society Phoenix AZ 2019)."},{"key":"e_1_3_4_33_2","doi-asserted-by":"crossref","unstructured":"J. R. Clough et al. \u201cGlobal and local interpretability for cardiac MRI classification\u201d in International Conference on Medical Image Computing and Computer-Assisted Intervention D. Shen et al. Eds. (Springer 2019) pp. 656\u2013664.","DOI":"10.1007\/978-3-030-32251-9_72"},{"key":"e_1_3_4_34_2","unstructured":"D. Bouchacourt L. Denoyer EDUCE: Explaining model decisions through unsupervised concepts. arXiv [Preprint] (2019). https:\/\/arxiv.org\/abs\/1905.11852 (Accessed 1 November 2023)."},{"key":"e_1_3_4_35_2","doi-asserted-by":"crossref","unstructured":"H. Yeche J. Harrison T. Berthier \u201cUbs: A dimension-agnostic metric for concept vector interpretability applied to radiomics\u201d in Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support K. Suzuki et al. Eds. (Springer 2019) pp. 12\u201320.","DOI":"10.1007\/978-3-030-33850-3_2"},{"key":"e_1_3_4_36_2","unstructured":"S. Sreedharan U. Soni M. Verma S. Srivastava S. Kambhampati \u201cBridging the gap: Providing post-hoc symbolic explanations for sequential decision-making problems with inscrutable representations\u201d in Proceedings of the International Conference on Learning Representations 10 Y. LeCun Ed. (Curran Associates Inc. Red Hook NY 2022)."},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.20378\/irb-47277"},{"key":"e_1_3_4_38_2","doi-asserted-by":"crossref","unstructured":"D. Mincu et al. \u201cConcept-based model explanations for electronic health records\u201d in Proceedings of the Conference on Health Inference and Learning M. Ghassemi T. Naumann E. Pierson Eds. (Association for Computing New York NY 2021) pp. 36\u201346.","DOI":"10.1145\/3450439.3451858"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/TETC.2022.3171314"},{"key":"e_1_3_4_40_2","unstructured":"J. Z. Forde C. Lovering G. Konidaris E. Pavlick M. L. Littman \u201cWhere when & which concepts does AlphaZero learn? Lessons from the game of Hex\u201d in AAAI Workshop on Reinforcement Learning in Games (2022)."},{"key":"e_1_3_4_41_2","doi-asserted-by":"crossref","unstructured":"N. Tomlin A. He D. Klein \"Understanding game-playing agents with natural language annotations\" in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) S. Muresan P. Nakov A. Villavicencio Eds. (Association for Computational Linguistics Dublin Ireland 2022) pp. 797\u2013807.","DOI":"10.18653\/v1\/2022.acl-short.90"},{"key":"e_1_3_4_42_2","first-page":"20554","article-title":"On completeness-aware concept-based explanations in deep neural networks","volume":"33","author":"Yeh C. K.","year":"2020","unstructured":"C. K. Yeh , On completeness-aware concept-based explanations in deep neural networks. Adv. Neural Inf. Process. Syst. 33, 20554\u201320565 (2020).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_4_43_2","unstructured":"A. Ghorbani J. Wexler J. Y. Zou B. Kim \u201cTowards automatic concept-based explanations\u201d in Advances in Neural Information Processing Systems 32 H. Wallach et al. Eds. (Curran Associates Inc. Red Hook NY 2019) pp. 9273\u20139282."},{"key":"e_1_3_4_44_2","doi-asserted-by":"crossref","unstructured":"V. V. Ramaswamy S. S. Y. Kim R. Fong O. Russakovsky \u201cOverlooked factors in concept-based explanations: Dataset choice concept learnability and human capability\u201d in Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (IEEE Vancouver Canada 2023) pp. 10932\u201310941.","DOI":"10.1109\/CVPR52729.2023.01052"},{"key":"e_1_3_4_45_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-020-00265-z"},{"key":"e_1_3_4_46_2","doi-asserted-by":"crossref","unstructured":"S. Sinha M. Huai J. Sun A. Zhang \u201cUnderstanding and enhancing robustness of concept-based models\u201d in Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence (AAAI\u201923\/IAAI\u201923\/EAAI\u201923) B. Willians Y. Chen J. Neville Eds. (AAAI Press 2023) vol. 37 pp. 15127\u201315135.","DOI":"10.1609\/aaai.v37i12.26765"},{"key":"e_1_3_4_47_2","unstructured":"M. T. Bahadori D. E. Heckerman \u201cDebiasing concept-based explanations with causal analysis\u201d in Proceedings of the 9th International Conference on Learning Representations (Curran Associates Inc. Red Hook NY 2024)."},{"key":"e_1_3_4_48_2","unstructured":"Z. Wu K. D\u2019Oosterlinck A. Geiger A. Zur C. Potts \u201cCausal proxy models for concept-based model explanations\u201d in International Conference on Machine Learning A. Krause et al. Eds. (PMLR 2023) pp. 37313\u201337334."},{"key":"e_1_3_4_49_2","unstructured":"A. Mahinpei J. Clark I. Lage F. Doshi-Velez W. Pan Promises and pitfalls of black-box concept learning models. arXiv [Preprint] (2021). https:\/\/doi.org\/10.48550\/arXiv.2106.13314 (Accessed 1 November 2023)."},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3023394"},{"key":"e_1_3_4_51_2","doi-asserted-by":"crossref","unstructured":"A. Heuillet F. Couthouis N. D\u00edaz-Rodr\u00edguez Explainability in deep reinforcement learning. Knowl.-Based Syst. 214 (2021) 10.1016\/j.knosys.2020.106685.","DOI":"10.1016\/j.knosys.2020.106685"},{"key":"e_1_3_4_52_2","unstructured":"C. Glanois et al. A survey on interpretable reinforcement learning. arXiv [Preprint] (2022). https:\/\/arxiv.org\/abs\/2112.13112 (Accessed 1 November 2023)."},{"key":"e_1_3_4_53_2","unstructured":"A. Krajna M. Brcic T. Lipic J. Doncevic Explainability in reinforcement learning: Perspective and position. arXiv [Preprint] (2022). https:\/\/arxiv.org\/abs\/2203.11547 (Accessed 1 November 2023)."},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3527448"},{"key":"e_1_3_4_55_2","unstructured":"S. Milani N. Topin M. Veloso F. Fang A survey of explainable reinforcement learning. arXiv [Preprint] (2022). https:\/\/arxiv.org\/abs\/2202.08434 (Accessed 1 November 2023)."},{"key":"e_1_3_4_56_2","doi-asserted-by":"crossref","unstructured":"R. Dazeley P. Vamplew F. Cruz Explainable reinforcement learning for broad-XAI: a conceptual framework and survey. Neural Comput & Applic 35 16893\u201316916 (2023).","DOI":"10.1007\/s00521-023-08423-1"},{"key":"e_1_3_4_57_2","first-page":"3444","article-title":"Beyond rewards: A hierarchical perspective on offline multiagent behavioral analysis","volume":"35","author":"Omidshafiei S.","year":"2022","unstructured":"S. Omidshafiei, A. Kapishnikov, Y. Assogba, L. Dixon, B. Kim, Beyond rewards: A hierarchical perspective on offline multiagent behavioral analysis. Adv. Neural Inf. Process. Syst. 35, 3444\u20133460 (2022).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_4_58_2","unstructured":"D. Das S. Chernova B. Kim \u201cState2Explanation: concept-based explanations to benefit agent learning and user understanding\u201d in Proceedings of the International Conference on Neural Information Processing Systems 37 A. Oh et al. Eds. (Curran Associates Inc. Red Hook NY 2023) pp. 67156\u201367182."},{"key":"e_1_3_4_59_2","doi-asserted-by":"crossref","unstructured":"J. Gajcin I. Dusparic \u201cReCCoVER: Detecting causal confusion for explainable reinforcement learning\u201d in Proceedings of Explainable and Transparent AI and Multi-Agent Systems D. Calvaresi A. Najjar M. Winikoff K. Fr\u00e4mling Eds. (Springer Cham Switzerland 2022) pp. 38\u201356.","DOI":"10.1007\/978-3-031-15565-9_3"},{"key":"e_1_3_4_60_2","doi-asserted-by":"crossref","unstructured":"R. M. Annasamy K. Sycara \u201cTowards better interpretability in deep q-networks\u201d in Proceedings of the AAAI conference on artificial intelligence P. van Hentenryck Z. Zhou Eds. (AAAI Press Honolulu HI 2019) vol. 33 pp. 4561\u20134569.","DOI":"10.1609\/aaai.v33i01.33014561"},{"key":"e_1_3_4_61_2","unstructured":"S. V. Deshmukh R. Srivatsan S. Vijay J. Subramanian C. Agarwal Counterfactual explanation policies in RL. arXiv [Preprint] (2023). https:\/\/arxiv.org\/abs\/2307.13192 (Accessed 1 November 2023)."},{"key":"e_1_3_4_62_2","unstructured":"M. L. Olson L. Neal F. Li W. K. Wong Counterfactual states for Atari agents via generative deep learning. arXiv [Preprint] (2019). https:\/\/arxiv.org\/abs\/1909.12969 (Accessed 1 November 2023)."},{"key":"e_1_3_4_63_2","unstructured":"Z. Wang et al. \u201cDueling network architectures for deep reinforcement learning\u201d in International Conference on Machine Learning M. Balcan K. Weinberger Eds. (PMLR 2016) pp. 1995\u20132003."},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-019-01228-7"},{"key":"e_1_3_4_65_2","unstructured":"S. Greydanus A. Koul J. Dodge A. Fern \u201cVisualizing and understanding atari agents\u201d in International Conference on Machine Learning J. Dy A. Krause Eds. (PMLR 2018) pp. 1792\u20131801."},{"key":"e_1_3_4_66_2","unstructured":"T. N. Mundhenk B. Y. Chen G. Friedland Efficient saliency maps for explainable AI. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/1911.11293 (Accessed 1 November 2023)."},{"key":"e_1_3_4_67_2","unstructured":"A. Atrey K. Clary D. Jensen \u201cExploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning\u201d in Proceedings of the International Conference on Learning Representations 8 A. Rush Ed. (Curran Associates Inc. Red Hook NY 2020)."},{"key":"e_1_3_4_68_2","first-page":"24","article-title":"Extracting tree-structured representations of trained networks","volume":"8","author":"Craven M.","year":"1995","unstructured":"M. Craven, J. Shavlik, Extracting tree-structured representations of trained networks. Adv. Neural Inf. Process. Syst. 8, 24\u201330 (1995).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_4_69_2","first-page":"2499","article-title":"Verifiable reinforcement learning via policy extraction","volume":"31","author":"Bastani O.","year":"2018","unstructured":"O. Bastani, Y. Pu, A. Solar-Lezama, Verifiable reinforcement learning via policy extraction. Adv. Neural Inf. Process. Syst. 31, 2499\u20132509 (2018).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_4_70_2","unstructured":"A. M. Roth N. Topin P. Jamshidi M. Veloso Conservative q-improvement: Reinforcement learning for an interpretable decision-tree policy. arXiv [Preprint] (2019). https:\/\/arxiv.org\/abs\/1907.01180 (Accessed 1 November 2023)."},{"key":"e_1_3_4_71_2","unstructured":"Y. Coppens et al. \u201cDistilling deep reinforcement learning policies in soft decision trees\u201d in Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence T. Miller R. Weber D. Magazzeni Eds. (2019) pp. 1\u20136."},{"key":"e_1_3_4_72_2","doi-asserted-by":"crossref","unstructured":"G. Liu O. Schulte W. Zhu Q. Li \u201cToward interpretable deep reinforcement learning with linear model u-trees\u201d in Machine Learning and Knowledge Discovery in Databases: European Conference ECML PKDD 2018 Dublin Ireland September 10\u201314 2018 Proceedings Part II 18 M. Berlingerio F. Bonchi T. G\u00e4tner N. Huerly G. Ifrim Eds. (Springer Nature Cham Switzerland 2019 pp. 414\u2013429.","DOI":"10.1007\/978-3-030-10928-8_25"},{"key":"e_1_3_4_73_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2022.03.022"},{"key":"e_1_3_4_74_2","unstructured":"P. Madumal T. Miller L. Sonenberg F. Vetere Distal explanations for model-free explainable reinforcement learning. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2001.10284 (Accessed 1 November 2023)."},{"key":"e_1_3_4_75_2","doi-asserted-by":"crossref","unstructured":"T. Jaunet R. Vuillemot C. Wolf \u201cDRLViz: Understanding decisions and memory in deep reinforcement learning\u201d in Computer Graphics Forum M. Wimmer Ed. (Wiley Online Library 2020) vol. 39 pp. 49\u201361.","DOI":"10.1111\/cgf.13962"},{"key":"e_1_3_4_76_2","unstructured":"A. Koul S. Greydanus A. Fern \u201cLearning finite state representations of recurrent policy networks\u201d in Proceedings of the International Conference on Learning Representations 6 Y. Begio Y. LeCun Eds. (Curran Associates Inc. Red Hook NY 2018)."},{"key":"e_1_3_4_77_2","unstructured":"M. Finkelstein et al. \u201cExplainable reinforcement learning via model transforms (2022)\u201d in Proceedings of International Conference on Advances in Neural Information Processing Systems 35 S. Koyejo et al. Eds. (Curran Associates Inc. Red Hook NY 2022) pp. 34039\u201334051."},{"key":"e_1_3_4_78_2","unstructured":"T. Zahavy N. Ben-Zrihem S. Mannor \u201cGraying the black box: Understanding DQNs\u201d in Proceedings of the International Conference on Machine Learning M. Balcan K. Weinberger Eds. (PMLR 2016) pp. 1899\u20131908."},{"key":"e_1_3_4_79_2","unstructured":"C. Rupprecht C. Ibrahim C. J. Pal Finding and visualizing weaknesses of deep reinforcement learning agents. arXiv [Preprint] (2019). https:\/\/arxiv.org\/abs\/1904.01318 (Accessed 1 November 2023)."},{"key":"e_1_3_4_80_2","doi-asserted-by":"crossref","unstructured":"L. H. Gilpin et al. \u201cExplaining explanations: An overview of interpretability of machine learning\u201d in 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA) F. Bonchi et al. Eds. (IEEE 2018) pp. 80\u201389.","DOI":"10.1109\/DSAA.2018.00018"},{"key":"e_1_3_4_81_2","unstructured":"L. H. Gilpin A. R. Paley M. A. Alam S. Spurlock K. J. Hammond \u201cExplanation\u201d is not a technical term: The problem of ambiguity in XAI. arXiv [Preprint] (2022). https:\/\/arxiv.org\/abs\/2207.00007 (Accessed 1 November 2023)."},{"key":"e_1_3_4_82_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aar6404"},{"key":"e_1_3_4_83_2","unstructured":"N. Toma\u0161ev U. Paquet D. Hassabis V. Kramnik Assessing game balance with AlphaZero: Exploring alternative rule sets in chess. arXiv [Preprint] (2020). https:\/\/arxiv.org\/abs\/2009.04374 (Accessed 1 November 2023)."},{"key":"e_1_3_4_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460349"},{"key":"e_1_3_4_85_2","unstructured":"T. Zahavy et al. Diversifying AI: Towards creative chess with AlphaZero. arXiv [Preprint] (2023). https:\/\/arxiv.org\/abs\/2308.09175 (Accessed 1 November 2023)."},{"key":"e_1_3_4_86_2","unstructured":"LCZero Development Community Leela Chess Zero (2018). lczero.org. Accessed 20 November 2019."},{"key":"e_1_3_4_87_2","unstructured":"P. Gupta et al. \u201cExplain your move: Understanding agent actions using specific and relevant feature attribution\u201d in International Conference on Learning Representations (Curran Associates Inc. Red Hook NY 2020)."},{"key":"e_1_3_4_88_2","doi-asserted-by":"crossref","unstructured":"Y. Kerner \u201cCase-based evaluation in computer chess\u201d in Advances in Case-Based Reasoning J. P. Haton M. Keane M. Manago Eds. (Springer Berlin Heidelberg Berlin Heidelberg 1995) pp. 240\u2013254.","DOI":"10.1007\/3-540-60364-6_40"},{"key":"e_1_3_4_89_2","doi-asserted-by":"crossref","unstructured":"H. Jhamtani V. Gangal E. Hovy G. Neubig T. Berg-Kirkpatrick \u201cLearning to generate move-by-move commentary for chess games from large-scale social forum data\u201d in The 56th Annual Meeting of the Association for Computational Linguistics (ACL) I. Gurevych Y. Miyao Eds. (Melbourne Australia 2018).","DOI":"10.18653\/v1\/P18-1154"},{"key":"e_1_3_4_90_2","unstructured":"DecodeChess Understanding Chess with Explainable AI (2017). https:\/\/decodechess.com\/about\/. Accessed 1 August 2023."},{"key":"e_1_3_4_91_2","unstructured":"X. Feng et al. \u201cChessGPT: Bridging policy learning and language modeling\u201d in Proceedings of the International Conference on Advances in Neural Information Processing Systems 36 A. Oh et al. Eds. (Curran Associates Inc. Red Hook NY 2023) pp. 34039\u201334051."},{"key":"e_1_3_4_92_2","doi-asserted-by":"crossref","unstructured":"A. P\u00e1lsson Y. Bj\u00f6rnsson \u201cUnveiling concepts learned by a world-class chess-playing agent\u201d in International Joint Conference on Artificial Intelligence E. Elkind Ed. (International Joint Conferences on Artificial Intelligence Organization Macao 2023) 4864-4872.","DOI":"10.24963\/ijcai.2023\/541"},{"key":"e_1_3_4_93_2","doi-asserted-by":"crossref","unstructured":"J. Gonz\u00e1lez-D\u00edaz I. Palacios-Huerta Alphazero ideas. SSRN (2022). http:\/\/dx.doi.org\/10.2139\/ssrn.4140916. Accessed 1 November 2023.","DOI":"10.2139\/ssrn.4140916"},{"key":"e_1_3_4_94_2","volume-title":"Game Changer: AlphaZero\u2019s Groundbreaking Chess Strategies and the Promise of AI","author":"Sadler M.","year":"2019","unstructured":"M. Sadler, N. Regan, Game Changer: AlphaZero\u2019s Groundbreaking Chess Strategies and the Promise of AI (New In Chess, 2019)."}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.2406675122","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,26]],"date-time":"2025-03-26T13:44:21Z","timestamp":1742996661000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/10.1073\/pnas.2406675122"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,26]]},"references-count":94,"journal-issue":{"issue":"13","published-print":{"date-parts":[[2025,4]]}},"alternative-id":["10.1073\/pnas.2406675122"],"URL":"https:\/\/doi.org\/10.1073\/pnas.2406675122","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,26]]},"assertion":[{"value":"2024-04-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-10","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2406675122"}}