{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T21:51:55Z","timestamp":1777585915510,"version":"3.51.4"},"reference-count":26,"publisher":"Walter de Gruyter GmbH","issue":"1","license":[{"start":{"date-parts":[[2022,10,1]],"date-time":"2022-10-01T00:00:00Z","timestamp":1664582400000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/3.0"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>We consider an extended notion of reinforcement learning in which the environment can simulate the agent and base its outputs on the agent\u2019s hypothetical behavior. Since good performance usually requires paying attention to whatever things the environment\u2019s outputs are based on, we argue that for an agent to achieve on-average good performance across many such extended environments, it is necessary for the agent to self-reflect. Thus weighted-average performance over the space of all suitably well-behaved extended environments could be considered a way of measuring how self-reflective an agent is. We give examples of extended environments and introduce a simple transformation which experimentally seems to increase some standard RL agents\u2019 performance in a certain type of extended environment.<\/jats:p>","DOI":"10.2478\/jagi-2022-0001","type":"journal-article","created":{"date-parts":[[2022,11,3]],"date-time":"2022-11-03T03:26:26Z","timestamp":1667445986000},"page":"1-24","source":"Crossref","is-referenced-by-count":3,"title":["Extending Environments to Measure Self-reflection in Reinforcement Learning"],"prefix":"10.2478","volume":"13","author":[{"given":"Samuel Allen","family":"Alexander","sequence":"first","affiliation":[{"name":"The U.S. Securities and Exchange Commission"}]},{"given":"Michael","family":"Castaneda","sequence":"additional","affiliation":[{"name":"KX"}]},{"given":"Kevin","family":"Compher","sequence":"additional","affiliation":[{"name":"InQTel"}]},{"given":"Oscar","family":"Martinez","sequence":"additional","affiliation":[{"name":"The U.S. Securities and Exchange Commission"}]}],"member":"374","published-online":{"date-parts":[[2022,11,3]]},"reference":[{"key":"2026042814172096613_j_jagi-2022-0001_ref_001","doi-asserted-by":"crossref","unstructured":"Alexander, S. A., and Hutter, M. 2021. Reward-Punishment Symmetric Universal Intelligence. In CAGI.10.1007\/978-3-030-93758-4_1","DOI":"10.1007\/978-3-030-93758-4_1"},{"key":"2026042814172096613_j_jagi-2022-0001_ref_002","doi-asserted-by":"crossref","unstructured":"Alexander, S. A., and Pedersen, A. P. 2022. Pseudo-visibility: A Game Mechanic Involving Willful Ignorance. In FLAIRS.10.32473\/flairs.v35i.130652","DOI":"10.32473\/flairs.v35i.130652"},{"key":"2026042814172096613_j_jagi-2022-0001_ref_003","unstructured":"Alexander, S. A.; Castaneda, M.; Compher, K.; and Martinez, O. 2022. Extended Environments. https:\/\/github.com\/semitrivial\/ExtendedEnvironments."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_004","doi-asserted-by":"crossref","unstructured":"Alexander, S. A. 2022. Extended subdomains: a solution to a problem of Hern\u00e1ndez-Orallo and Dowe. Preprint (accepted to CAGI-22).10.1007\/978-3-031-19907-3_14","DOI":"10.1007\/978-3-031-19907-3_14"},{"key":"2026042814172096613_j_jagi-2022-0001_ref_005","unstructured":"Bell, J. H.; Linsefors, L.; Oesterheld, C.; and Skalse, J. 2021. Reinforcement Learning in Newcomblike Environments. In NeurIPS."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_006","doi-asserted-by":"crossref","unstructured":"Bellemare, M. G.; Naddaf, Y.; Veness, J.; and Bowling, M. 2013. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research 47:253\u2013279.10.1613\/jair.3912","DOI":"10.1613\/jair.3912"},{"key":"2026042814172096613_j_jagi-2022-0001_ref_007","unstructured":"Beyret, B.; Hern\u00e1ndez-Orallo, J.; Cheke, L.; Halina, M.; Shanahan, M.; and Crosby, M. 2019. The animal-AI environment: Training and testing animal-like artificial cognition. Preprint."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_008","unstructured":"Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; and Zaremba, W. 2016. OpenAI gym. Preprint."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_009","doi-asserted-by":"crossref","unstructured":"Chaslot, G.; Bakkes, S.; Szita, I.; and Spronck, P. 2008. Monte-Carlo Tree Search: A New Framework for Game AI. AIIDE 8:216\u2013217.10.1609\/aiide.v4i1.18700","DOI":"10.1609\/aiide.v4i1.18700"},{"key":"2026042814172096613_j_jagi-2022-0001_ref_010","unstructured":"Chollet, F. 2019. On the measure of intelligence. Preprint."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_011","unstructured":"Cobbe, K.; Hesse, C.; Hilton, J.; and Schulman, J. 2020. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, 2048\u20132056. PMLR."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_012","doi-asserted-by":"crossref","unstructured":"Gavane, V. 2013. A measure of real-time intelligence. Journal of Artificial General Intelligence 4(1):31\u201348.10.2478\/jagi-2013-0003","DOI":"10.2478\/jagi-2013-0003"},{"key":"2026042814172096613_j_jagi-2022-0001_ref_013","unstructured":"Hendrycks, D., and Dietterich, T. 2019. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_014","doi-asserted-by":"crossref","unstructured":"Hern\u00e1ndez-Orallo, J., and Dowe, D. L. 2010. Measuring universal intelligence: Towards an anytime intelligence test. Artificial Intelligence 174(18):1508\u20131539.10.1016\/j.artint.2010.09.006","DOI":"10.1016\/j.artint.2010.09.006"},{"key":"2026042814172096613_j_jagi-2022-0001_ref_015","unstructured":"Hibbard, B. 2008. Adversarial sequence prediction. In CAGI."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_016","unstructured":"Hubinger, E.; van Merwijk, C.; Mikulik, V.; Skalse, J.; and Garrabrant, S. 2019. Risks from learned optimization in advanced machine learning systems. Preprint."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_017","unstructured":"Hutter, M. 2004. Universal artificial intelligence: Sequential decisions based on algorithmic probability. Springer."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_018","doi-asserted-by":"crossref","unstructured":"Legg, S., and Hutter, M. 2007. Universal intelligence: A definition of machine intelligence. Minds and machines 17(4):391\u2013444.10.1007\/s11023-007-9079-x","DOI":"10.1007\/s11023-007-9079-x"},{"key":"2026042814172096613_j_jagi-2022-0001_ref_019","doi-asserted-by":"crossref","unstructured":"Legg, S., and Veness, J. 2013. An approximation of the universal intelligence measure. In Algorithmic Probability and Friends: Bayesian Prediction and Artificial Intelligence. Springer. 236\u2013249.10.1007\/978-3-642-44958-1_18","DOI":"10.1007\/978-3-642-44958-1_18"},{"key":"2026042814172096613_j_jagi-2022-0001_ref_020","unstructured":"Leike, J., and Hutter, M. 2015. Bad universal priors and notions of optimality. In Conference on Learning Theory, 1244\u20131259. PMLR."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_021","doi-asserted-by":"crossref","unstructured":"Li, M., and Vit\u00e1nyi, P. 2008. An introduction to Kolmogorov complexity and its applications. Springer.10.1007\/978-0-387-49820-1","DOI":"10.1007\/978-0-387-49820-1"},{"key":"2026042814172096613_j_jagi-2022-0001_ref_022","unstructured":"Nichol, A.; Pfau, V.; Hesse, C.; Klimov, O.; and Schulman, J. 2018. Gotta Learn Fast: A New Benchmark for Generalization in RL. Preprint."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_023","doi-asserted-by":"crossref","unstructured":"Nozick, R. 1969. Newcomb\u2019s problem and two principles of choice. In Rescher, N., ed., Essays in honor of Carl G. Hempel. Springer. 114\u2013146.10.1007\/978-94-017-1466-2_7","DOI":"10.1007\/978-94-017-1466-2_7"},{"key":"2026042814172096613_j_jagi-2022-0001_ref_024","unstructured":"Raffn, A.; Hill, A.; Ernestus, M.; Gleave, A.; Kanervisto, A.; and Dormann, N. 2019. Stable Baselines3. https:\/\/github.com\/DLR-RM\/stable-baselines3."},{"key":"2026042814172096613_j_jagi-2022-0001_ref_025","doi-asserted-by":"crossref","unstructured":"Sherstan, C.; White, A.; Machado, M. C.; and Pilarski, P. M. 2016. Introspective agents: Confidence measures for general value functions. In Conference on Artificial General Intelligence, 258\u2013261. Springer.10.1007\/978-3-319-41649-6_26","DOI":"10.1007\/978-3-319-41649-6_26"},{"key":"2026042814172096613_j_jagi-2022-0001_ref_026","unstructured":"Yampolskiy, R. V. 2017. Detecting qualia in natural and artificial agents. Preprint."}],"container-title":["Journal of Artificial General Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/reference-global.com\/pdf\/10.2478\/jagi-2022-0001","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T20:03:01Z","timestamp":1777406581000},"score":1,"resource":{"primary":{"URL":"https:\/\/reference-global.com\/article\/10.2478\/jagi-2022-0001"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,1]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,11,3]]},"published-print":{"date-parts":[[2022,10,1]]}},"alternative-id":["10.2478\/jagi-2022-0001"],"URL":"https:\/\/doi.org\/10.2478\/jagi-2022-0001","relation":{},"ISSN":["1946-0163"],"issn-type":[{"value":"1946-0163","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,1]]}}}