{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T03:21:09Z","timestamp":1740108069499,"version":"3.37.3"},"reference-count":49,"publisher":"Springer Science and Business Media LLC","issue":"23","license":[{"start":{"date-parts":[[2022,12,12]],"date-time":"2022-12-12T00:00:00Z","timestamp":1670803200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2022,12,12]],"date-time":"2022-12-12T00:00:00Z","timestamp":1670803200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["N66001-17-2-4030"],"award-info":[{"award-number":["N66001-17-2-4030"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1619433"],"award-info":[{"award-number":["IIS-1619433"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000183","name":"Army Research Office","doi-asserted-by":"publisher","award":["W911NF2010224"],"award-info":[{"award-number":["W911NF2010224"]}],"id":[{"id":"10.13039\/100000183","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000181","name":"Air Force Office of Scientific Research","doi-asserted-by":"publisher","award":["FA9550-18-1-0462","FA9550-18-1-0462","W911NF2010224"],"award-info":[{"award-number":["FA9550-18-1-0462","FA9550-18-1-0462","W911NF2010224"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neural Comput & Applic"],"published-print":{"date-parts":[[2023,8]]},"DOI":"10.1007\/s00521-022-08119-y","type":"journal-article","created":{"date-parts":[[2022,12,12]],"date-time":"2022-12-12T16:02:56Z","timestamp":1670860976000},"page":"16877-16892","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["RePReL: a unified framework for integrating relational planning and reinforcement learning for effective abstraction in discrete and continuous domains"],"prefix":"10.1007","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7548-3719","authenticated-orcid":false,"given":"Harsha","family":"Kokel","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sriraam","family":"Natarajan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Balaraman","family":"Ravindran","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Prasad","family":"Tadepalli","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,12,12]]},"reference":[{"key":"8119_CR1","unstructured":"Andrychowicz M, Wolski F, Ray A, et\u00a0al (2017) Hindsight experience replay. In: NeurIPS, pp 5048\u20135058"},{"key":"8119_CR2","unstructured":"Ash JT, Adams RP (2020) On warm-starting neural network training. In: Advances in neural information processing systems 33: annual conference on neural information processing systems 2020, NeurIPS 2020, December 6\u201312, 2020, virtual"},{"key":"8119_CR3","unstructured":"Battaglia PW, Hamrick JB, Bapst V et\u00a0al (2018) Relational inductive biases, deep learning, and graph networks. CoRR abs\/1806.01261"},{"key":"8119_CR4","first-page":"213","volume":"3","author":"RI Brafman","year":"2002","unstructured":"Brafman RI, Tennenholtz M (2002) R-max-a general polynomial time algorithm for near-optimal reinforcement learning. JMLR 3:213\u2013231","journal-title":"JMLR"},{"key":"8119_CR5","unstructured":"Das S, Natarajan S, Roy K et\u00a0al (2020) Fitted q-learning for relational domains. CoRR abs\/2006.05595"},{"key":"8119_CR6","unstructured":"Dietterich TG (1998) The maxq method for hierarchical reinforcement learning. In: ICML, pp 118\u2013126"},{"key":"8119_CR7","unstructured":"Dong H, Mao J, Lin T et\u00a0al (2019) Neural logic machines. In: ICLR"},{"key":"8119_CR8","doi-asserted-by":"crossref","unstructured":"Driessens K, Ramon J, Blockeel H (2001) Speeding up relational reinforcement learning through the use of an incremental first order decision tree learner. In: ECML, pp 97\u2013108","DOI":"10.1007\/3-540-44795-4_9"},{"issue":"1\/2","key":"8119_CR9","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1023\/A:1007694015589","volume":"43","author":"S D\u017eeroski","year":"2001","unstructured":"D\u017eeroski S, De Raedt L, Driessens K (2001) Relational reinforcement learning. Mach Learn 43(1\/2):7\u201352","journal-title":"Mach Learn"},{"key":"8119_CR10","doi-asserted-by":"publisher","first-page":"123","DOI":"10.3389\/frobt.2019.00123","volume":"6","author":"M Eppe","year":"2019","unstructured":"Eppe M, Nguyen PDH, Wermter S (2019) From semantics to execution: integrating action planning with reinforcement learning for robotic causal problem-solving. Front Robot AI 6:123","journal-title":"Front Robot AI"},{"key":"8119_CR11","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1613\/jair.5714","volume":"61","author":"R Evans","year":"2018","unstructured":"Evans R, Grefenstette E (2018) Learning explanatory rules from noisy data. JAIR 61:1\u201364","journal-title":"JAIR"},{"key":"8119_CR12","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1613\/jair.1700","volume":"25","author":"A Fern","year":"2006","unstructured":"Fern A, Yoon S, Givan R (2006) Approximate policy iteration with a policy language bias: solving relational Markov decision processes. JAIR 25:75\u2013118","journal-title":"JAIR"},{"key":"8119_CR13","doi-asserted-by":"crossref","unstructured":"Ghallab M, Nau D, Traverso P (2004) Automated Planning: theory and practice. Elsevier","DOI":"10.1016\/B978-155860856-6\/50021-1"},{"issue":"1\u20132","key":"8119_CR14","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1016\/S0004-3702(02)00376-4","volume":"147","author":"R Givan","year":"2003","unstructured":"Givan R, Dean T, Greig M (2003) Equivalence notions and model minimization in Markov decision processes. Artif Intell 147(1\u20132):163\u2013223","journal-title":"Artif Intell"},{"key":"8119_CR15","first-page":"75","volume":"III","author":"M Grounds","year":"2005","unstructured":"Grounds M, Kudenko D (2005) Combining reinforcement learning with symbolic planning. AAMAS III:75\u201386","journal-title":"AAMAS"},{"key":"8119_CR16","unstructured":"Guestrin C, Patrascu R, Schuurmans D (2002) Algorithm-directed exploration for model-based reinforcement learning in factored mdps. In: ICML, pp 235\u2013242"},{"key":"8119_CR17","unstructured":"Guestrin C, Koller D et\u00a0al (2003) Generalizing plans to new environments in relational mdps. In: IJCAI, pp 1003\u20131010"},{"key":"8119_CR18","unstructured":"Haarnoja T, Zhou A, Abbeel P et\u00a0al (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: ICML, pp 1861\u20131870"},{"key":"8119_CR19","doi-asserted-by":"crossref","unstructured":"van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. In: AAAI, pp 2094\u20132100","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"8119_CR20","unstructured":"Igl M, Farquhar G, Luketina J et\u00a0al (2021) Transient non-stationarity and generalisation in deep reinforcement learning. In: International conference on learning representations"},{"key":"8119_CR21","doi-asserted-by":"crossref","unstructured":"Illanes L, Yan X, Icarte RT et\u00a0al (2020) Symbolic plans as high-level instructions for reinforcement learning. ICAPS pp 540\u2013550","DOI":"10.1609\/icaps.v30i1.6750"},{"key":"8119_CR22","unstructured":"Janisch J, Pevn\u00fd T, Lis\u00fd V (2021) Symbolic relational deep reinforcement learning based on graph neural networks. RL4RealLife @ ICML"},{"key":"8119_CR23","doi-asserted-by":"crossref","unstructured":"Jiang Y, Yang F, Zhang S, et\u00a0al (2019) Task-motion planning with reinforcement learning for adaptable mobile service robots. In: IROS, pp 7529\u20137534","DOI":"10.1109\/IROS40897.2019.8967680"},{"key":"8119_CR24","unstructured":"Jiang Z, Luo S (2019) Neural logic reinforcement learning. In: ICML, vol\u00a097. PMLR, pp 3110\u20133119"},{"key":"8119_CR25","unstructured":"Jiang Z, Minervini P, Jiang M, et\u00a0al (2021) Grid-to-graph: flexible spatial relational inductive biases for reinforcement learning. In: AAMAS. ACM, pp 674\u2013682"},{"key":"8119_CR26","doi-asserted-by":"crossref","unstructured":"Kimura D, Ono M, Chaudhury S, et\u00a0al (2021) Neuro-symbolic reinforcement learning with first-order logic. In: EMNLP, pp 3505\u20133511","DOI":"10.18653\/v1\/2021.emnlp-main.283"},{"issue":"1","key":"8119_CR27","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1609\/icaps.v31i1.16001","volume":"31","author":"H Kokel","year":"2021","unstructured":"Kokel H, Manoharan A, Natarajan S et al (2021) Reprel: integrating relational planning and reinforcement learning for effective abstraction. ICAPS 31(1):533\u2013541","journal-title":"ICAPS"},{"key":"8119_CR28","unstructured":"Kokel H, Manoharan A, Natarajan S, et\u00a0al (2021b) Dynamic probabilistic logic models for effective abstractions in RL. CoRR abs\/2110.08318"},{"key":"8119_CR29","doi-asserted-by":"crossref","unstructured":"Konidaris G, Kaelbling LP, Lozano-Perez T (2018) From skills to symbols: Learning symbolic representations for abstract high-level planning. JAIR","DOI":"10.1613\/jair.5575"},{"key":"8119_CR30","unstructured":"Li L, Walsh TJ, Littman ML (2006) Towards a unified theory of state abstraction for mdps. In: ISAIM, p\u00a05"},{"key":"8119_CR31","doi-asserted-by":"crossref","unstructured":"Li R, Jabri A, Darrell T, et\u00a0al (2020) Towards practical multi-object manipulation using relational reinforcement learning. In: ICRA. IEEE, pp 4051\u20134058","DOI":"10.1109\/ICRA40945.2020.9197468"},{"key":"8119_CR32","unstructured":"Lyle C, Rowland M, Dabney W (2022) Understanding and preventing capacity loss in reinforcement learning. In: International Conference on Learning Representations"},{"key":"8119_CR33","doi-asserted-by":"crossref","unstructured":"Lyu D, Yang F, Liu B, et\u00a0al (2019) SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In: AAAI, pp 2970\u20132977","DOI":"10.1609\/aaai.v33i01.33012970"},{"key":"8119_CR34","doi-asserted-by":"crossref","unstructured":"Manfredotti CE (2009) Modeling and inference with relational dynamic bayesian networks. In: CCAI, pp 287\u2013290","DOI":"10.1007\/978-3-642-01818-3_44"},{"key":"8119_CR35","doi-asserted-by":"crossref","unstructured":"Natarajan S, Tadepalli P, et\u00a0al (2005) Learning first-order probabilistic models with combining rules. In: ICML, pp 609\u2013616","DOI":"10.1145\/1102351.1102428"},{"issue":"1\u20133","key":"8119_CR36","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1007\/s10472-009-9138-5","volume":"54","author":"S Natarajan","year":"2008","unstructured":"Natarajan S, Tadepalli P et al (2008) Learning first-order probabilistic models with combining rules. Ann Math Artif Intell 54(1\u20133):223\u2013256","journal-title":"Ann Math Artif Intell"},{"key":"8119_CR37","unstructured":"Nau D, Cao Y, Lotem A, et\u00a0al (1999) Shop: Simple hierarchical ordered planner. In: IJCAI, pp 968\u2013975"},{"key":"8119_CR38","unstructured":"Parr R, Russell SJ (1998) Reinforcement learning with hierarchies of machines. In: NeurIPS, pp 1043\u20131049"},{"key":"8119_CR39","unstructured":"Plappert M, Andrychowicz M, Ray A, et\u00a0al (2018) Multi-goal reinforcement learning: challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464"},{"key":"8119_CR40","unstructured":"Ravindran B, Barto AG (2003) Smdp homomorphisms: an algebraic approach to abstraction in semi markov decision processes. In: IJCAI, pp 1011\u20131018"},{"key":"8119_CR41","unstructured":"Ravindran B, Barto AG (2003) SMDP homomorphisms: an algebraic approach to abstraction in semi-markov decision processes. In: IJCAI. Morgan Kaufmann, pp 1011\u20131018"},{"key":"8119_CR42","unstructured":"Riegel R, Gray AG, Luus FPS, et\u00a0al (2020) Logical neural networks. CoRR abs\/2006.13155"},{"issue":"6419","key":"8119_CR43","doi-asserted-by":"publisher","first-page":"1140","DOI":"10.1126\/science.aar6404","volume":"362","author":"D Silver","year":"2018","unstructured":"Silver D, Hubert T et al (2018) A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science 362(6419):1140\u20131144","journal-title":"Science"},{"key":"8119_CR44","unstructured":"Sutton RS, Precup D, Singh SP (1998) Intra-option learning about temporally abstract actions. In: ICML, pp 556\u2013564"},{"issue":"1\u20132","key":"8119_CR45","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1016\/S0004-3702(99)00052-1","volume":"112","author":"RS Sutton","year":"1999","unstructured":"Sutton RS, Precup D, Singh SP (1999) Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning. Artif Intell 112(1\u20132):181\u2013211","journal-title":"Artif Intell"},{"key":"8119_CR46","unstructured":"Vlasselaer J, Meert W, et\u00a0al (2014) Efficient probabilistic inference for dynamic relational models. In: StarAI @ AAAI"},{"key":"8119_CR47","doi-asserted-by":"crossref","unstructured":"Yang F, Lyu D, Liu B, et\u00a0al (2018) Peorl: integrating symbolic planning and hierarchical reinforcement learning for robust decision-making. IJCAI pp 4860\u20134866","DOI":"10.24963\/ijcai.2018\/675"},{"key":"8119_CR48","unstructured":"Zambaldi V, Raposo D, et\u00a0al (2019) Deep reinforcement learning with relational inductive biases. In: ICLR"},{"key":"8119_CR49","doi-asserted-by":"crossref","unstructured":"Zhang L, Li X, Wang M, et\u00a0al (2021) Off-policy differentiable logic reinforcement learning. In: ECML PKDD, pp 617\u2013632","DOI":"10.1007\/978-3-030-86520-7_38"}],"container-title":["Neural Computing and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-022-08119-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00521-022-08119-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00521-022-08119-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T19:06:12Z","timestamp":1689188772000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00521-022-08119-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,12]]},"references-count":49,"journal-issue":{"issue":"23","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["8119"],"URL":"https:\/\/doi.org\/10.1007\/s00521-022-08119-y","relation":{},"ISSN":["0941-0643","1433-3058"],"issn-type":[{"type":"print","value":"0941-0643"},{"type":"electronic","value":"1433-3058"}],"subject":[],"published":{"date-parts":[[2022,12,12]]},"assertion":[{"value":"11 March 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 November 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 December 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The research leading to these results received funding from federal grants as mentioned in the acknowledgments. Specifically, HK and SN received the support of ARO (award W911NF2010224) and AFOSR (award FA9550-18-1-0462). PT received the support of DARPA (contract N66001-17-2-4030) and NSF (grant IIS-1619433). No conflict of interest exists with this work.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}