{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T16:59:09Z","timestamp":1775321949146,"version":"3.50.1"},"reference-count":50,"publisher":"Maximum Academic Press","license":[{"start":{"date-parts":[[2018,12,18]],"date-time":"2018-12-18T00:00:00Z","timestamp":1545091200000},"content-version":"unspecified","delay-in-days":351,"URL":"https:\/\/www.cambridge.org\/core\/terms"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["The Knowledge Engineering Review"],"published-print":{"date-parts":[[2018]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>We consider task planning for long-living intelligent agents situated in dynamic environments. Specifically, we address the problem of incomplete knowledge of the world due to the addition of new objects with unknown action models. We propose a multilayered agent architecture that uses meta-reasoning to control hierarchical task planning and situated learning, monitor expectations generated by a plan against world observations, forms goals and rewards for the situated reinforcement learner, and learns the missing planning knowledge relevant to the new objects. We use occupancy grids as a low-level representation for the high-level expectations to capture changes in the physical world due to the additional objects, and provide a similarity method for detecting discrepancies between the expectations and the observations at run-time; the meta-reasoner uses these discrepancies to formulate goals and rewards for the learner, and the learned policies are added to the hierarchical task network plan library for future re-use. We describe our experiments in the Minecraft and Gazebo microworlds to demonstrate the efficacy of the architecture and the technique for learning. We test our approach against an ablated reinforcement learning (RL) version, and our results indicate this form of expectation enhances the learning curve for RL while being more generic than propositional representations.<\/jats:p>","DOI":"10.1017\/s0269888918000279","type":"journal-article","created":{"date-parts":[[2018,12,18]],"date-time":"2018-12-18T05:39:37Z","timestamp":1545111577000},"source":"Crossref","is-referenced-by-count":8,"title":["Towards life-long adaptive agents: using metareasoning for combining knowledge-based planning with situated learning"],"prefix":"10.48130","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2154-5889","authenticated-orcid":false,"given":"Priyam","family":"Parashar","sequence":"first","affiliation":[]},{"given":"Ashok K.","family":"Goel","sequence":"additional","affiliation":[]},{"given":"Bradley","family":"Sheneman","sequence":"additional","affiliation":[]},{"given":"Henrik I.","family":"Christensen","sequence":"additional","affiliation":[]}],"member":"27968","published-online":{"date-parts":[[2018,12,18]]},"reference":[{"key":"S0269888918000279_ref37","unstructured":"Ng A. Y. , Harada D. & Russell S. 1999. Policy invariance under reward transformations: theory and application to reward shaping. ICML 99, 278\u2013287."},{"key":"S0269888918000279_ref9","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9780262014809.001.0001"},{"key":"S0269888918000279_ref41","unstructured":"Paisner M. , Maynord M. , Cox M. T. & Perlis D. 2013. Goal-driven autonomy in dynamic environments. In Goal Reasoning: Papers from the ACS Workshop, 79."},{"key":"S0269888918000279_ref29","volume-title":"Case-Based Reasoning: Experiences, Lessons and Future Directions","author":"Leake","year":"1996"},{"key":"S0269888918000279_ref8","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888906000592"},{"key":"S0269888918000279_ref22","volume-title":"Case-Based Planning: Viewing Planning as a Memory Task","author":"Hammond","year":"2012"},{"key":"S0269888918000279_ref12","unstructured":"Dannenhauer D. , Mu\u00f1oz-Avila H. & Cox M. T. 2016. Informed expectations to guide GDA agents in partially observable environments, In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence 2493\u20132499. AAAI Press."},{"key":"S0269888918000279_ref32","doi-asserted-by":"crossref","unstructured":"Murdock J. W. & Goel A. K. 2001. Meta-case-based reasoning: using functional models to adapt case-based agents. In Case-Based Reasoning Research and Development, 407\u2013421. Springer.","DOI":"10.1007\/3-540-44593-5_29"},{"key":"S0269888918000279_ref17","unstructured":"Fitzgerald T. , Bullard K. , Thomaz A. & Goel A. K. 2016. Situated mapping for transfer learning. In Fourth Annual Conference on Advances in Cognitive Systems."},{"key":"S0269888918000279_ref27","doi-asserted-by":"crossref","unstructured":"Koenig N. & Howard A. 2004. Design and use paradigms for gazebo, an open-source multi-robot simulator. In 2004 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566) 3, 2149\u20132154. IEEE.","DOI":"10.1109\/IROS.2004.1389727"},{"key":"S0269888918000279_ref31","doi-asserted-by":"crossref","unstructured":"Mu\u00f1oz-Avila H. , Jaidee U. , Aha D. W. & Carter E. 2010. Goal-d autonomy with case-based reasoning. In Case-Based Reasoning. Research and Development, 228\u2013241. Springer.","DOI":"10.1007\/978-3-642-14274-1_18"},{"key":"S0269888918000279_ref23","unstructured":"Jaidee U. , Mu\u00f1oz-Avila H. & Aha D. W. 2011a. Case-based learning in goal-driven autonomy agents for real-time strategy combat tasks. Proceedings of the ICCBR."},{"key":"S0269888918000279_ref15","doi-asserted-by":"crossref","unstructured":"Efthymiadis K. & Kudenko D. 2015. Knowledge revision for reinforcement learning with abstract MDPs. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems\u2019, AAMAS \u201815, 763\u2013770. International Foundation for Autonomous Agents and Multiagent Systems.","DOI":"10.65109\/BZQX5029"},{"key":"S0269888918000279_ref45","unstructured":"Sutton R. S. & Barto A. G. 1998. Reinforcement learning: An introduction, 1. MIT Press."},{"key":"S0269888918000279_ref16","unstructured":"Erol K. , Hendler J. & Nau D. S. 1994. HTN planning: complexity and expressivity. AAAI 94, 1123\u20131128."},{"key":"S0269888918000279_ref2","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2008.10.024"},{"key":"S0269888918000279_ref46","volume-title":"Probabilistic Robotics","author":"Thrun","year":"2005"},{"key":"S0269888918000279_ref35","doi-asserted-by":"crossref","unstructured":"Nau D. S. , Au T. C. , Ilghami O. , Kuter U. , Murdock J. W. , Wu D. & Yaman F. 2003. SHOP2: An HTN planning system. 1, 379\u2013404.","DOI":"10.1613\/jair.1141"},{"key":"S0269888918000279_ref14","doi-asserted-by":"crossref","unstructured":"Efthymiadis K. & Kudenko D. 2013. Using plan-based reward shaping to learn strategies in StarCraft: Broodwar. In 2013 IEEE Conference on Computational Inteligence in Games (CIG), 1\u20138. IEEE.","DOI":"10.1109\/CIG.2013.6633622"},{"key":"S0269888918000279_ref49","unstructured":"Vattam S. , Klenk M. , Molineaux M. & Aha D. W. 2013. Breadth of Approaches to Goal Reasoning: A Research Survey. Technical Report. Naval Research Lab."},{"key":"S0269888918000279_ref4","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/2376.001.0001","volume-title":"Designing Sociable Robots","author":"Breazeal","year":"2004"},{"key":"S0269888918000279_ref11","unstructured":"Dannenhauer D. & Mu\u00f1oz-Avila H. 2015b. Raising expectations in GDA agents acting in dynamic environments. In Proceedings of the 24th International Conference on Artificial Intelligence, 2241\u20132247. AAAI Press."},{"key":"S0269888918000279_ref36","unstructured":"Nau D. S. , Cao Y. , Lotem A. & Munoz-Avila H. 1999. SHOP: simple hierarchical ordered planner. In Proceedings of the 16th International Joint Conference on Artificial Intelligence, 2, 968\u2013973. IJCAI'99, Morgan Kaufmann Publishers Inc."},{"key":"S0269888918000279_ref34","unstructured":"Murdock J. W. & Goel A. K. 2011. Self-improvement through self-understanding: model-based reflection for agent adaptation. Georgia Institute of Technology."},{"key":"S0269888918000279_ref3","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(96)00047-1"},{"key":"S0269888918000279_ref40","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8640.2009.00344.x"},{"key":"S0269888918000279_ref38","volume-title":"Artificial Intelligence: A New Synthesis","author":"Nilsson","year":"1998"},{"key":"S0269888918000279_ref50","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992698"},{"key":"S0269888918000279_ref33","doi-asserted-by":"publisher","DOI":"10.1080\/09528130701472416"},{"key":"S0269888918000279_ref26","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2011.09.012"},{"key":"S0269888918000279_ref1","first-page":"12","article-title":"A review of recent research in metareasoning and metalearning","volume":"28","author":"Anderson","year":"2007","journal-title":"AI Magazine"},{"key":"S0269888918000279_ref5","doi-asserted-by":"publisher","DOI":"10.1016\/S1364-6613(02)02016-8"},{"key":"S0269888918000279_ref6","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2005.10.009"},{"key":"S0269888918000279_ref7","doi-asserted-by":"crossref","unstructured":"Cox M. T. , Alavi Z. , Dannenhauer D. , Eyorokon V. , Mu\u00f1oz-Avila H. & Perlis D. 2016. MIDCA: a metacognitive, integrated dual-cycle architecture for self-regulated autonomy, In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 3712\u20133718. AAAI Press.","DOI":"10.1609\/aaai.v30i1.9886"},{"key":"S0269888918000279_ref10","first-page":"88","volume-title":"Case-Based Reasoning Research and Development,Lecture Notes in Computer Science 9343","author":"Dannenhauer","year":"2015"},{"key":"S0269888918000279_ref13","doi-asserted-by":"publisher","DOI":"10.1017\/S026988891500017X"},{"key":"S0269888918000279_ref18","doi-asserted-by":"crossref","unstructured":"Goel A. K. & Jones J. 2011. Metareasoning for self-adaptation in intelligent agents. Metareasoning. The MIT Press.","DOI":"10.7551\/mitpress\/9780262014809.003.0010"},{"key":"S0269888918000279_ref19","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2017.44"},{"key":"S0269888918000279_ref20","doi-asserted-by":"crossref","unstructured":"Grounds M. & Kudenko D. 2008. Combining reinforcement learning with symbolic planning. In Adaptive Agents and Multi-Agent Systems III. Adaptation and Multi-Agent Learning, 75\u201386. Springer.","DOI":"10.1007\/978-3-540-77949-0_6"},{"key":"S0269888918000279_ref21","doi-asserted-by":"crossref","unstructured":"Grzes M. & Kudenko D. 2008. Plan-based reward shaping for reinforcement learning. In 2008 4th International IEEE Conference Intelligent Systems, 2, 10\u201322\u201310\u201329. IEEE.","DOI":"10.1109\/IS.2008.4670492"},{"key":"S0269888918000279_ref24","unstructured":"Jaidee U. , Mu\u00f1oz-Avila H. & Aha D. W. 2011b. Integrated learning for goal-driven autonomy. In IJCAI Proceedings\u2014International Joint Conference on Artificial Intelligence, 22, 2450. IJCAI\/AAAI."},{"key":"S0269888918000279_ref25","unstructured":"Jaidee U. , Mu\u00f1oz-Avila H. & Aha D. W. 2012. Learning and reusing goal-specific policies for goal-driven autonomy. In Case-Based Reasoning Research and Development Lecture Notes in Computer Science 7466, B. D. Agudo & I. Watson (eds). Springer, 182\u2013195."},{"key":"S0269888918000279_ref28","volume-title":"Case-Based Reasoning","author":"Kolodner","year":"2014"},{"key":"S0269888918000279_ref30","doi-asserted-by":"publisher","DOI":"10.1109\/TC.1983.1676196"},{"key":"S0269888918000279_ref39","volume-title":"Principles of Artificial Intelligence","author":"Nilsson","year":"2014"},{"key":"S0269888918000279_ref42","doi-asserted-by":"publisher","DOI":"10.4324\/9780203781821"},{"key":"S0269888918000279_ref43","doi-asserted-by":"publisher","DOI":"10.1080\/08839519508945470"},{"key":"S0269888918000279_ref44","doi-asserted-by":"publisher","DOI":"10.1006\/ijhc.1999.0331"},{"key":"S0269888918000279_ref47","unstructured":"Ulam P. , Goel A. K. , Jones J. & Murdock W. 2005. Using model-based reflection to guide reinforcement learning. In Proceedings of the IJCAI 2005 Workshop on Reasoning, Representation and Learning in Computer Games."},{"key":"S0269888918000279_ref48","doi-asserted-by":"crossref","unstructured":"Ulam P. , Jones J. & Goel A. K. 2008. Combining model-based meta-reasoning and reinforcement learning for adapting game-playing agents. In Proceedings of the Fourth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 132\u2013137. AAAI Press.","DOI":"10.1609\/aiide.v4i1.18685"}],"container-title":["The Knowledge Engineering Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0269888918000279","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T15:49:36Z","timestamp":1775317776000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0269888918000279\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018]]},"references-count":50,"alternative-id":["S0269888918000279"],"URL":"https:\/\/doi.org\/10.1017\/s0269888918000279","relation":{},"ISSN":["0269-8889","1469-8005"],"issn-type":[{"value":"0269-8889","type":"print"},{"value":"1469-8005","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018]]},"article-number":"e24"}}