{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,23]],"date-time":"2025-11-23T15:05:22Z","timestamp":1763910322177,"version":"build-2065373602"},"publisher-location":"Cham","reference-count":39,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783032083326","type":"print"},{"value":"9783032083333","type":"electronic"}],"license":[{"start":{"date-parts":[[2025,10,19]],"date-time":"2025-10-19T00:00:00Z","timestamp":1760832000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,10,19]],"date-time":"2025-10-19T00:00:00Z","timestamp":1760832000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>It is often desirable to constrain reinforcement learning (RL) policies to align with societal norms and individual preferences in order to better represent users\u2019 intentions and expectations. In order to adequately deal with exceptions and conflicts between competing norms\/preferences, it is useful for such constraints to be defeasible; and to enable transparent justification for decisions, as well as allowing them to be more easily re-used in different variations of a particular problem setting, it is also helpful to express them in a high-level, symbolic, human-understandable form. However, manually specifying these norms can be challenging, typically requiring familiarity with a logical formalism and extensive knowledge of the application domain. To address this, we propose a human-in-the-loop approach that learns ethical constraints by combining autonomous exploration with expert imitation from simple user-provided examples of the intended behaviour. Our system infers a list of high-level logical constraints along with an RL policy that adheres to them, while optionally allowing users to interactively refine constraints, resolve conflicts, manage ethical contradictions and provide counter-examples. The system interleaves an Approximate Q-learning (AQL) component for goal-directed exploration and an interactive Inductive Logic Programming (ILP) module for symbolic constraint inference, thus ensuring learned norms remain transparent, auditable, and editable. We evaluate our method in a Pacman environment, demonstrating that the learned logical constraints achieve normative compliance comparable to existing approaches, without requiring manual rule specification. Furthermore, we demonstrate the value of human intervention via interactive ILP mechanisms, and experimentally show that it accelerates convergence to accurate and efficient normative frameworks. This work therefore contributes a novel approach to learning symbolic, defeasible constraints for RL policies, and introduces interactive mechanisms that allow constraints to be tailored, refined, and overridden in accordance with user intentions.<\/jats:p>","DOI":"10.1007\/978-3-032-08333-3_3","type":"book-chapter","created":{"date-parts":[[2025,10,18]],"date-time":"2025-10-18T05:22:28Z","timestamp":1760764948000},"page":"41-63","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["A Human-in-the-Loop Approach to Learning Social Norms and Behavioural Policies"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9641-4157","authenticated-orcid":false,"given":"Oliver","family":"Deane","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0221-1501","authenticated-orcid":false,"given":"Oliver","family":"Ray","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,10,19]]},"reference":[{"key":"3_CR1","doi-asserted-by":"crossref","unstructured":"Alshiekh, M., Bloem, R., Ehlers, R., K\u00f6nighofer, B., Niekum, S., Topcu, U.: Safe reinforcement learning via shielding. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol.\u00a032 (2018)","DOI":"10.1609\/aaai.v32i1.11797"},{"key":"3_CR2","unstructured":"Altman, E.: Constrained Markov Decision Processes. Routledge (1999)"},{"key":"3_CR3","unstructured":"Andrighetto, G., Governatori, G., Noriega, P., van\u00a0der Torre, L.: Normative multi-agent systems. Dagstuhl Reports 2(3) (2012)"},{"issue":"7","key":"3_CR4","doi-asserted-by":"publisher","first-page":"2593","DOI":"10.1007\/s10994-023-06311-2","volume":"112","author":"M Baert","year":"2023","unstructured":"Baert, M., Leroux, S., Simoens, P.: Inverse reinforcement learning through logic constraint inference. Mach. Learn. 112(7), 2593\u20132618 (2023)","journal-title":"Mach. Learn."},{"issue":"4","key":"3_CR5","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1007\/s10994-024-06653-5","volume":"114","author":"M Baert","year":"2025","unstructured":"Baert, M., Mazzaglia, P., Leroux, S., Simoens, P.: Maximum causal entropy inverse constrained reinforcement learning. Mach. Learn. 114(4), 103 (2025)","journal-title":"Mach. Learn."},{"key":"3_CR6","unstructured":"Berreby, F., Bourgne, G., Ganascia, J.G.: A declarative modular framework for representing and applying ethical principles. In: 16th Conference on Autonomous Agents and MultiAgent Systems (2017)"},{"key":"3_CR7","doi-asserted-by":"crossref","unstructured":"Boella, G., van\u00a0der Torre, L., Verhagen, H., et\u00a0al.: Normative multi-agent systems. In: Dagstuhl seminar proceedings, vol.\u00a07122 (2007)","DOI":"10.1007\/978-3-540-75254-7_5"},{"key":"3_CR8","doi-asserted-by":"publisher","first-page":"765","DOI":"10.1613\/jair.1.13507","volume":"74","author":"A Cropper","year":"2022","unstructured":"Cropper, A., Duman\u010di\u0107, S.: Inductive logic programming at 30: a new introduction. J. Artif. Intell. Res. 74, 765\u2013850 (2022)","journal-title":"J. Artif. Intell. Res."},{"key":"3_CR9","doi-asserted-by":"crossref","unstructured":"Deane, O., Ray, O.: Interactive model refinement in relational domains with inductive logic programming. In: Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 127\u2013129 (2023)","DOI":"10.1145\/3581754.3584150"},{"key":"3_CR10","unstructured":"Deane, O., Ray, O.: Neuro-symbolic inverse constrained reinforcement learning. In: Proceedings of the 19th International Conference on Neurosymbolic Learning and Reasoning (2025). in Press"},{"key":"3_CR11","doi-asserted-by":"crossref","unstructured":"Gil, Y.: Learning by experimentation: incremental refinement of incomplete planning domains. In: Machine Learning Proceedings 1994, pp. 87\u201395. Elsevier (1994)","DOI":"10.1016\/B978-1-55860-335-6.50019-2"},{"key":"3_CR12","unstructured":"Jansen, N., K\u00f6nighofer, B., Junges, S., Serban, A.C., Bloem, R.: Safe reinforcement learning via probabilistic shields. arXiv preprint arXiv:1807.06096 (2018)"},{"key":"3_CR13","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1613\/jair.301","volume":"4","author":"LP Kaelbling","year":"1996","unstructured":"Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237\u2013285 (1996)","journal-title":"J. Artif. Intell. Res."},{"issue":"6","key":"3_CR14","doi-asserted-by":"publisher","first-page":"4909","DOI":"10.1109\/TITS.2021.3054625","volume":"23","author":"BR Kiran","year":"2021","unstructured":"Kiran, B.R., et al.: Deep reinforcement learning for autonomous driving: a survey. IEEE Trans. Intell. Transp. Syst. 23(6), 4909\u20134926 (2021)","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"3_CR15","unstructured":"Liu, G., Xu, S., Liu, S., Gaurav, A., Subramanian, S.G., Poupart, P.: A comprehensive survey on inverse constrained reinforcement learning: definitions, progress and challenges. arXiv preprint arXiv:2409.07569 (2024)"},{"issue":"1","key":"3_CR16","doi-asserted-by":"publisher","first-page":"22","DOI":"10.3390\/robotics10010022","volume":"10","author":"R Liu","year":"2021","unstructured":"Liu, R., Nageotte, F., Zanne, P., de Mathelin, M., Dresp-Langley, B.: Deep reinforcement learning for the control of robotic manipulation: a focussed mini-review. Robotics 10(1), 22 (2021)","journal-title":"Robotics"},{"key":"3_CR17","unstructured":"Malik, S., Anwar, U., Aghasi, A., Ahmed, A.: Inverse constrained reinforcement learning. In: International Conference on Machine Learning, pp. 7390\u20137399. PMLR (2021)"},{"key":"3_CR18","doi-asserted-by":"crossref","unstructured":"McPherson, D.L., Stocking, K.C., Sastry, S.S.: Maximum likelihood constraint inference from stochastic demonstrations. In: 2021 IEEE Conference on Control Technology and Applications (CCTA), pp. 1208\u20131213. IEEE (2021)","DOI":"10.1109\/CCTA48906.2021.9658862"},{"key":"3_CR19","doi-asserted-by":"crossref","unstructured":"Melo, F.S., Ribeiro, M.I.: Q-learning with linear function approximation. In: International Conference on Computational Learning Theory, pp. 308\u2013322. Springer (2007)","DOI":"10.1007\/978-3-540-72927-3_23"},{"key":"3_CR20","first-page":"295","volume":"8","author":"S Muggleton","year":"1991","unstructured":"Muggleton, S.: New generation computing. Induc. Logic Prog. 8, 295\u2013318 (1991)","journal-title":"Induc. Logic Prog."},{"key":"3_CR21","doi-asserted-by":"crossref","unstructured":"Neufeld, E.A., Bartocci, E., Ciabattoni, A., Governatori, G.: A normative supervisor for reinforcement learning agents. In: CADE, pp. 565\u2013576 (2021)","DOI":"10.1007\/978-3-030-79876-5_32"},{"issue":"4\/5","key":"3_CR22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1147\/JRD.2019.2940428","volume":"63","author":"R Noothigattu","year":"2019","unstructured":"Noothigattu, R., et al.: Teaching AI agents ethical values using reinforcement learning and policy orchestration. IBM J. Res. Dev. 63(4\/5), 1\u20132 (2019)","journal-title":"IBM J. Res. Dev."},{"key":"3_CR23","doi-asserted-by":"crossref","unstructured":"Nute, D.: Defeasible deontic logic, vol.\u00a0263. Springer Science & Business Media (1997)","DOI":"10.1007\/978-94-015-8851-5"},{"key":"3_CR24","doi-asserted-by":"crossref","unstructured":"Pnueli, A.: The temporal logic of programs. In: 18th annual symposium on foundations of computer science (SFCS 1977), pp. 46\u201357. IEEE (1977)","DOI":"10.1109\/SFCS.1977.32"},{"key":"3_CR25","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1016\/j.artint.2015.06.005","volume":"227","author":"H Prakken","year":"2015","unstructured":"Prakken, H., Sartor, G.: Law and logic: a review from an argumentation perspective. Artif. Intell. 227, 214\u2013245 (2015)","journal-title":"Artif. Intell."},{"issue":"3","key":"3_CR26","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1016\/j.jal.2008.10.007","volume":"7","author":"O Ray","year":"2009","unstructured":"Ray, O.: Nonmonotonic abductive inductive learning. J. Appl. Log. 7(3), 329\u2013340 (2009)","journal-title":"J. Appl. Log."},{"key":"3_CR27","doi-asserted-by":"crossref","unstructured":"Ray, O., Moyle, S.: Towards expert-guided elucidation of cyber attacks through interactive inductive logic programming. In: 2021 13th International Conference on Knowledge and Systems Engineering (KSE), pp.\u00a01\u20137. IEEE (2021)","DOI":"10.1109\/KSE53942.2021.9648769"},{"key":"3_CR28","doi-asserted-by":"crossref","unstructured":"Rossi, F., Mattei, N.: Building ethically bounded ai. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol.\u00a033, pp. 9785\u20139789 (2019)","DOI":"10.1609\/aaai.v33i01.33019785"},{"key":"3_CR29","first-page":"1","volume":"19","author":"S Russell","year":"2022","unstructured":"Russell, S.: Artificial intelligence and the problem of control. Persp. Digital Hum. 19, 1\u2013322 (2022)","journal-title":"Persp. Digital Hum."},{"key":"3_CR30","unstructured":"Scobee, D.R., Sastry, S.S.: Maximum likelihood constraint inference for inverse reinforcement learning. arXiv preprint arXiv:1909.05477 (2019)"},{"issue":"1","key":"3_CR31","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1016\/0004-3702(94)00071-8","volume":"74","author":"M Shanahan","year":"1995","unstructured":"Shanahan, M.: Default reasoning about spatial occupancy. Artif. Intell. 74(1), 147\u2013163 (1995)","journal-title":"Artif. Intell."},{"key":"3_CR32","doi-asserted-by":"crossref","unstructured":"Silver, D., et\u00a0al.: Mastering the game of go without human knowledge. Nature 550(7676), 354\u2013359 (2017)","DOI":"10.1038\/nature24270"},{"key":"3_CR33","unstructured":"Srinivasan, A.: The aleph manual. Machine Learning at the Computing Laboratory (2001)"},{"key":"3_CR34","unstructured":"Subramanian, S.G., Liu, G., Elmahgiubi, M., Rezaee, K., Poupart, P.: Confidence aware inverse constrained reinforcement learning. arXiv preprint arXiv:2406.16782 (2024)"},{"issue":"1","key":"3_CR35","first-page":"103","volume":"21","author":"S Thrun","year":"2000","unstructured":"Thrun, S., Littman, M.L.: Reinforcement learning: an introduction. AI Mag. 21(1), 103\u2013103 (2000)","journal-title":"AI Mag."},{"key":"3_CR36","doi-asserted-by":"crossref","unstructured":"Varghese, D., Barroso-Bergada, D., Bohan, D.A., Tamaddoni-Nezhad, A.: Efficient abductive learning of microbial interactions using meta inverse entailment. In: International Conference on Inductive Logic Programming, pp. 127\u2013141. Springer (2022)","DOI":"10.1007\/978-3-031-55630-2_10"},{"key":"3_CR37","doi-asserted-by":"crossref","unstructured":"Walker, T., et al.: Automating the ILP setup task: converting user advice about specific examples into general background knowledge. In: Inductive Logic Programming: 20th International Conference, ILP 2010, Florence, Italy, June 27-30, 2010. Revised Papers 20, pp. 253\u2013268. Springer (2011)","DOI":"10.1007\/978-3-642-21295-6_28"},{"key":"3_CR38","first-page":"279","volume":"8","author":"CJ Watkins","year":"1992","unstructured":"Watkins, C.J., Dayan, P.: Q-learning. Machine learning 8, 279\u2013292 (1992)","journal-title":"Q-learning. Machine learning"},{"key":"3_CR39","doi-asserted-by":"crossref","unstructured":"Zhang, S., Stone, P.: CORPP: commonsense reasoning and probabilistic planning, as applied to dialog with a mobile robot. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol.\u00a029 (2015)","DOI":"10.1609\/aaai.v29i1.9385"}],"container-title":["Communications in Computer and Information Science","Explainable Artificial Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-032-08333-3_3","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,18]],"date-time":"2025-10-18T05:22:40Z","timestamp":1760764960000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-032-08333-3_3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,19]]},"ISBN":["9783032083326","9783032083333"],"references-count":39,"URL":"https:\/\/doi.org\/10.1007\/978-3-032-08333-3_3","relation":{},"ISSN":["1865-0929","1865-0937"],"issn-type":[{"value":"1865-0929","type":"print"},{"value":"1865-0937","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,19]]},"assertion":[{"value":"19 October 2025","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"Authors have no known conflicts of interest.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Disclosure of Interests"}},{"value":"xAI","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"World Conference on Explainable Artificial Intelligence","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Istanbul","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"T\u00fcrkiye","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2025","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"9 July 2025","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"11 July 2025","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"3","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"xai2025","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/xaiworldconference.com\/2025\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}