{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T14:40:33Z","timestamp":1768833633600,"version":"3.49.0"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2021,5,29]],"date-time":"2021-05-29T00:00:00Z","timestamp":1622246400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,5,29]],"date-time":"2021-05-29T00:00:00Z","timestamp":1622246400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001631","name":"University College Dublin","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001631","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Appl Intell"],"published-print":{"date-parts":[[2022,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In multi-agent systems, goal achievement is challenging when agents operate in ever-changing environments and face unseen situations, where not all the goals are known or predefined. In such cases, agents need to identify the changes and adapt their behaviour, by evolving their goals or even generating new goals to address the emerging requirements. Learning and practical reasoning techniques have been used to enable agents with limited knowledge to adapt to new circumstances. However, they depend on the availability of large amounts of data, require long exploration periods, and cannot help agents to set new goals. Furthermore, the accuracy of agents\u2019 actions is improved by introducing added intelligence through integrating conceptual features extracted from ontologies. However, the concerns related to taking suitable actions when unseen situations occur are not addressed. This paper proposes a new Automatic Goal Generation Model (AGGM) that enables agents to create new goals to handle unseen situations and to adapt to their ever-changing environment on a real-time basis. AGGM is compared to Q-learning, SARSA, and Deep Q Network in a Traffic Signal Control System case study. The results show that AGGM outperforms the baseline algorithms in unseen situations while handling the seen situations as well as the baseline algorithms.<\/jats:p>","DOI":"10.1007\/s10489-021-02449-5","type":"journal-article","created":{"date-parts":[[2021,5,29]],"date-time":"2021-05-29T21:02:44Z","timestamp":1622322164000},"page":"1808-1824","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Using ontology to guide reinforcement learning agents in unseen situations"],"prefix":"10.1007","volume":"52","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0983-301X","authenticated-orcid":false,"given":"Saeedeh","family":"Ghanadbashi","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fatemeh","family":"Golpayegani","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,5,29]]},"reference":[{"issue":"2","key":"2449_CR1","first-page":"3","volume":"39","author":"DW Aha","year":"2018","unstructured":"Aha DW (2018) Goal reasoning: Foundations, emerging applications, and prospects. AI Mag 39 (2):3\u201324","journal-title":"AI Mag"},{"key":"2449_CR2","unstructured":"Alegre LN (2019) SUMO-RL https:\/\/github.com\/LucasAlegre\/sumo-rl"},{"key":"2449_CR3","doi-asserted-by":"crossref","unstructured":"Almeida Falbo R, Menezes CS, Rocha ARC (1998) A systematic approach for building ontologies. In: Ibero-american conference on artificial intelligence (IBERAMIA). Springer, pp 349\u2013360","DOI":"10.1007\/3-540-49795-1_31"},{"key":"2449_CR4","doi-asserted-by":"crossref","unstructured":"Bailey JM, Golpayegani F, Clarke S (2019) Comasig: a collaborative multi-agent signal control to support senior drivers. In: IEEE Intelligent transportation systems conference (ITSC). IEEE, pp 1239\u20131244","DOI":"10.1109\/ITSC.2019.8917531"},{"issue":"3-4","key":"2449_CR5","first-page":"428","volume":"2","author":"J Broersen","year":"2002","unstructured":"Broersen J, Dastani M, Hulstijn J, van der Torre L (2002) Goal generation in the BOID architecture. Cognit Sci Quarter (CSQ) 2(3-4):428\u2013447","journal-title":"Cognit Sci Quarter (CSQ)"},{"key":"2449_CR6","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1016\/j.neucom.2012.12.001","volume":"108","author":"G Caruana","year":"2013","unstructured":"Caruana G, Li M, Liu Y (2013) An ontology enhanced parallel SVM for scalable spam filter training. Neurocomputing 108:45\u201357","journal-title":"Neurocomputing"},{"key":"2449_CR7","doi-asserted-by":"crossref","unstructured":"Cunnington D, Manotas I, Law M, de Mel G, Calo S, Bertino E, Russo A (2019) A generative policy model for connected and autonomous vehicles. In: IEEE Intelligent transportation systems conference (ITSC). IEEE, pp 1558\u20131565","DOI":"10.1109\/ITSC.2019.8916782"},{"key":"2449_CR8","doi-asserted-by":"crossref","unstructured":"Dignum F, Conte R (1997) Intentional agents and goal formation. In: International workshop on agent theories, architectures, and languages (ATAL). Springer, pp 231\u2013243","DOI":"10.1007\/BFb0026762"},{"key":"2449_CR9","unstructured":"Ding Y, Florensa C, Abbeel P, Phielipp M (2019) Goal-conditioned imitation learning. In: Conference on neural information processing systems (NIPS), pp 15,298\u201315,309"},{"key":"2449_CR10","doi-asserted-by":"publisher","first-page":"28,573","DOI":"10.1109\/ACCESS.2018.2831228","volume":"6","author":"A Dorri","year":"2018","unstructured":"Dorri A, Kanhere SS, Jurdak R (2018) Multi-agent systems: A survey. IEEE Access 6:28,573\u201328,593","journal-title":"IEEE Access"},{"key":"2449_CR11","unstructured":"Eysenbach B, Gu S, Ibarz J, Levine S (2017) Leave no trace: Learning to reset for safe and autonomous reinforcement learning. Computing Research Repository (CoRR). arXiv:1711.06782"},{"key":"2449_CR12","unstructured":"Florensa C, Held D, Wulfmeier M, Zhang M, Abbeel P (2017) Reverse curriculum generation for reinforcement learning. In: Annual conference on robot learning (coRL). PMLR, pp 482\u2013495"},{"key":"2449_CR13","doi-asserted-by":"crossref","unstructured":"Fong ACM, Hong G, Fong B (2019) Augmented intelligence with ontology of semantic objects. In: International conference on contemporary computing and informatics (IC3i). IEEE, pp 1\u20134","DOI":"10.1109\/IC3I46837.2019.9055577"},{"key":"2449_CR14","unstructured":"Fran\u00e7ois-Lavet V, Fonteneau R, Ernst D (2015) How to discount deep reinforcement learning: Towards new dynamic strategies. Computing Research Repository (CoRR). arXiv:1512.02011"},{"issue":"3","key":"2449_CR15","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3319402","volume":"10","author":"F Golpayegani","year":"2019","unstructured":"Golpayegani F, Dusparic I, Clarke S (2019) Using social dependence to enable neighbourly behaviour in open multi-agent systems. ACM Trans Intell Syst Technol (TIST) 10(3):1\u201331","journal-title":"ACM Trans Intell Syst Technol (TIST)"},{"key":"2449_CR16","unstructured":"Haber N, Mrowca D, Fei-Fei L, Yamins DL (2018) Learning to play with intrinsically-motivated, self-aware agents. In: Conference on neural information processing systems (NIPS), pp 8388\u20138399"},{"key":"2449_CR17","unstructured":"Hadfield-Menell D, Milli S, Abbeel P, Russell SJ, Dragan A (2017) Inverse reward design. In: Conference on neural information processing systems (NIPS), pp 6765\u20136774"},{"issue":"1","key":"2449_CR18","doi-asserted-by":"publisher","first-page":"9","DOI":"10.3233\/SW-180320","volume":"10","author":"A Haller","year":"2019","unstructured":"Haller A, Janowicz K, Cox SJ, Lefran\u00e7ois M., Taylor K, Le Phuoc D, Lieberman J, Garc\u00eda-castro R, Atkinson R, Stadler C (2019) The modular SSN ontology: A joint W3C and OGC standard specifying the semantics of sensors, observations, sampling, and actuation. Semantic Web 10(1):9\u201332","journal-title":"Semantic Web"},{"issue":"79","key":"2449_CR19","first-page":"1","volume":"21","author":"I Horrocks","year":"2004","unstructured":"Horrocks I, Patel-Schneider PF, Boley H, Tabet S, Grosof B, Dean M (2004) SWRL: A Semantic web rule language combining OWL and ruleML. W3C Member Submission 21(79):1\u201331","journal-title":"W3C Member Submission"},{"key":"2449_CR20","unstructured":"Jaidee U, Mu\u00f1oz-Avila H, Aha DW (2011) Integrated learning for goal-driven autonomy. In: International joint conference on artificial intelligence (IJCAI). IJCAI\/AAAI, pp 2450\u20132455"},{"key":"2449_CR21","doi-asserted-by":"crossref","unstructured":"Johnson B, Floyd MW, Coman A, Wilson MA, Aha DW (2018) Goal reasoning and trusted autonomy. In: Foundations of trusted autonomy. Springer, Cham, pp 47\u201366","DOI":"10.1007\/978-3-319-64816-3_3"},{"key":"2449_CR22","unstructured":"Kondrakunta S, Gogineni VR, Molineaux M, Munoz-Avila H, Oxenham M, Cox MT (2018) Toward problem recognition, explanation and goal formulation. In: Goal reasoning workshop at IJCAI\/FAIM"},{"issue":"8","key":"2449_CR23","first-page":"901","volume":"30","author":"S Krau\u00df","year":"1997","unstructured":"Krau\u00df S (1997) Towards a unified view of microscopic traffic flow theories. Int Federat Autom Control (IFAC) Proc 30(8):901\u2013905","journal-title":"Int Federat Autom Control (IFAC) Proc"},{"issue":"7","key":"2449_CR24","first-page":"105","volume":"7","author":"Z Liu","year":"2007","unstructured":"Liu Z (2007) A survey of intelligence methods in urban traffic signal control. Int J Comput Sci Netw Secur (IJCSNS) 7(7):105\u2013112","journal-title":"Int J Comput Sci Netw Secur (IJCSNS)"},{"key":"2449_CR25","doi-asserted-by":"crossref","unstructured":"Lopez PA, Behrisch M, Bieker-Walz L, Erdmann J, Fl\u00f6tter\u00f6d YP, Hilbrich R, L\u00fccken L, Rummel J, Wagner P, WieBner E (2018) Microscopic traffic simulation using sumo. In: IEEE Intelligent transportation systems conference (ITSC). IEEE, pp 2575\u20132582","DOI":"10.1109\/ITSC.2018.8569938"},{"key":"2449_CR26","unstructured":"Luck M, d\u2019Inverno M (1995) Goal generation and adoption in hierarchical agent models. In: Australasian joint conference on artificial intelligence (AJCAI). World scientific"},{"key":"2449_CR27","unstructured":"Maynord M, Cox MT, Paisner M, Perlis D (2013) Data-driven goal generation for integrated cognitive systems. In: AAAI Fall symposium series. AAAI Press"},{"key":"2449_CR28","doi-asserted-by":"crossref","unstructured":"Mazak A, Schandl B, Lanzenberger M (2010) Iweightings: Enhancing structure-based ontology alignment by enriching models with importance weighting. In: International conference on complex, intelligent and software intensive systems (CISIS). IEEE, pp 992\u2013997","DOI":"10.1109\/CISIS.2010.164"},{"key":"2449_CR29","doi-asserted-by":"crossref","unstructured":"Monticolo D, Lahoud I, Bonjour E (2012) Distributed knowledge extracted by a MAS using ontology alignment methods. In: International conference on computer & information science (ICCIS). IEEE, pp 386\u2013391","DOI":"10.1109\/ICCISci.2012.6297276"},{"key":"2449_CR30","doi-asserted-by":"crossref","unstructured":"Morignot P, Nashashibi F (2012) An ontology-based approach to relax traffic regulation for autonomous vehicle assistance. Computing Research Repository (CoRR). arXiv:1212.0768","DOI":"10.2316\/P.2013.793-024"},{"key":"2449_CR31","doi-asserted-by":"crossref","unstructured":"Motta JA, Capus L, Tourigny N (2016) Vence: a new machine learning method enhanced by ontological knowledge to extract summaries. In: Science and information (SAI) computing conference. IEEE, pp 61\u201370","DOI":"10.1109\/SAI.2016.7555963"},{"issue":"4","key":"2449_CR32","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1145\/2757001.2757003","volume":"1","author":"MA Musen","year":"2015","unstructured":"Musen MA (2015) The prot\u0117g\u0117 project: A look back and a look forward. AI Matters 1(4):4\u201312","journal-title":"AI Matters"},{"key":"2449_CR33","unstructured":"Nguyen TT, Nguyen ND, Nahavandi S (2018) Deep reinforcement learning for multi-agent systems: A review of challenges, solutions and applications. Computing Research Repository (CoRR). arXiv:1812.11794"},{"key":"2449_CR34","unstructured":"Noy NF, McGuinness DL et al (2001) Ontology development 101: A guide to creating your first ontology. Tech. rep., Stanford Knowledge Systems Laboratory. https:\/\/protege.stanford.edu\/publications\/ontology_development\/ontology101.pdf"},{"key":"2449_CR35","unstructured":"Powell J, Molineaux M, Aha DW (2011) Active and interactive discovery of goal selection knowledge. In: International florida artificial intelligence research society (FLAIRS) conference. AAAI Press"},{"key":"2449_CR36","doi-asserted-by":"crossref","unstructured":"Rezzai M, Dachry W, Moutaouakkil F, Medromi H (2018) Design and realization of a new architecture based on multi-agent systems and reinforcement learning for traffic signal control. In: International conference on multimedia computing and systems (ICMCS). IEEE, pp 1\u20136","DOI":"10.1109\/ICMCS.2018.8525896"},{"key":"2449_CR37","doi-asserted-by":"crossref","unstructured":"Sewak M (2019) Deep Q Network (DQN), double DQN, and dueling DQN. In: Deep reinforcement learning. Springer, pp 95\u2013108","DOI":"10.1007\/978-981-13-8285-7_8"},{"issue":"10","key":"2449_CR38","first-page":"271","volume":"2","author":"T Sharma","year":"2012","unstructured":"Sharma T, Tiwari N, Kelkar D (2012) Study of difference between forward and backward reasoning. Int J Emerg Technol Adv Eng (IJETAE) 2(10):271\u2013273","journal-title":"Int J Emerg Technol Adv Eng (IJETAE)"},{"key":"2449_CR39","unstructured":"Stojanovic L (2004) Methods and tools for ontology evolution. Ph.D. thesis, Karlsruhe Institute of Technology, Germany. http:\/\/digbib.ubka.uni-karlsruhe.de\/volltexte\/1000003270"},{"key":"2449_CR40","volume-title":"Reinforcement learning: An introduction","author":"RS Sutton","year":"2018","unstructured":"Sutton RS, Barto AG (2018) Reinforcement learning: An introduction. MIT Press, Cambridge"},{"key":"2449_CR41","unstructured":"Thanh-Tung D, Flood B, Wilson C, Sheahan C, Bao-Lam D (2006) Ontology-MAS for modelling and robust controlling enterprises. In: International conference on theories and applications of computer science (ICTACS), pp 116\u2013123"},{"key":"2449_CR42","doi-asserted-by":"crossref","unstructured":"Tom\u00e1s VR, Garcia LA (2005) A cooperative multiagent system for traffic management and control. In: International joint conference on autonomous agents and multiagent systems (AAMAS). ACM, pp 52\u201359","DOI":"10.1145\/1082473.1082804"},{"key":"2449_CR43","doi-asserted-by":"crossref","unstructured":"Wang Y, Yang X, Liang H, Liu Y (2018) A review of the self-adaptive traffic signal control system based on future traffic environment. J Adv Transport (JAT) 1\u201312","DOI":"10.1155\/2018\/1096123"}],"container-title":["Applied Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-021-02449-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10489-021-02449-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10489-021-02449-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,1,24]],"date-time":"2022-01-24T01:16:34Z","timestamp":1642986994000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10489-021-02449-5"}},"subtitle":["A traffic signal control system case study"],"short-title":[],"issued":{"date-parts":[[2021,5,29]]},"references-count":43,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,1]]}},"alternative-id":["2449"],"URL":"https:\/\/doi.org\/10.1007\/s10489-021-02449-5","relation":{},"ISSN":["0924-669X","1573-7497"],"issn-type":[{"value":"0924-669X","type":"print"},{"value":"1573-7497","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,29]]},"assertion":[{"value":"20 April 2021","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 May 2021","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Conflict of Interests"}}]}}