{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T10:20:45Z","timestamp":1777890045461,"version":"3.51.4"},"reference-count":36,"publisher":"SAGE Publications","issue":"4","license":[{"start":{"date-parts":[[2024,10,4]],"date-time":"2024-10-04T00:00:00Z","timestamp":1728000000000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SW"],"published-print":{"date-parts":[[2024,10,4]]},"abstract":"<jats:p>Computational agents support humans in many areas of life and are therefore found in heterogeneous contexts. This means that agents operate in rapidly changing environments and can be confronted with huge state and action spaces. In order to perform services and carry out activities satisfactorily, i.e. in a goal-oriented manner, agents require prior knowledge and therefore have to develop and pursue context-dependent policies. The problem here is that prescribing policies in advance is limited and inflexible, especially in dynamically changing environments. Moreover, the context (i.e. the external and internal state) of an agent determines its choice of actions. Since the environments in which agents operate can be stochastic and complex in terms of the number of states and feasible actions, activities are usually modelled in a simplified way by Markov decision processes so that, for example, agents with reinforcement learning are able to learn policies, i.e. state-action pairs, that help to capture the context and act accordingly to optimally perform activities. However, training policies for all possible contexts using reinforcement learning is time-consuming. A requirement and challenge for agents is to learn strategies quickly and respond immediately in cross-context environments and applications, e.g., the Internet, service robotics, cyber-physical systems. In this work, we propose a novel simulation-based approach that enables a) the representation of heterogeneous contexts through knowledge graphs and entity embeddings and b) the context-aware composition of policies on demand by ensembles of agents running in parallel. The evaluation we conducted with the \u201cVirtual Home\u201d dataset indicates that agents with a need to switch seamlessly between different contexts, e.g. in a home environment, can request on-demand composed policies that lead to the successful completion of context-appropriate activities without having to learn these policies in lengthy training steps and episodes, in contrast to agents that use reinforcement learning. The presented approach enables both context-aware and cross-context applicability of untrained computational agents. Furthermore, the source code of the approach as well as the generated data, i.e. the trained embeddings and the semantic representation of domestic activities, is open source and openly accessible on Github and Figshare.<\/jats:p>","DOI":"10.3233\/sw-233531","type":"journal-article","created":{"date-parts":[[2024,1,9]],"date-time":"2024-01-09T12:13:10Z","timestamp":1704802390000},"page":"1443-1471","source":"Crossref","is-referenced-by-count":2,"title":["Context-aware composition of agent policies by Markov decision process entity embeddings and agent ensembles"],"prefix":"10.1177","volume":"15","author":[{"given":"Nicole","family":"Merkle","sequence":"first","affiliation":[{"name":"Institute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ralf","family":"Mikut","sequence":"additional","affiliation":[{"name":"Institute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","reference":[{"key":"10.3233\/SW-233531_ref1","unstructured":"R.\u00a0Agarwal, M.C.\u00a0Machado, P.S.\u00a0Castro and M.G.\u00a0Bellemare, Contrastive behavioral similarity embeddings for generalization in reinforcement learning, in: International Conference on Learning Representations, 2021."},{"key":"10.3233\/SW-233531_ref2","unstructured":"C.\u00a0Allen and T.\u00a0Hospedales, Analogies explained: Towards understanding word embeddings, in: Proceedings of the 36th International Conference on Machine Learning, K.\u00a0Chaudhuri and R.\u00a0Salakhutdinov, eds, Proceedings of Machine Learning Research, Vol.\u00a097, PMLR, Long Beach, California, USA, 2019, pp.\u00a0223\u2013231, http:\/\/proceedings.mlr.press\/v97\/allen19a.html."},{"key":"10.3233\/SW-233531_ref3","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.IR.8318"},{"key":"10.3233\/SW-233531_ref4","doi-asserted-by":"publisher","DOI":"10.1109\/ICHR.2005.1573593"},{"key":"10.3233\/SW-233531_ref5","doi-asserted-by":"crossref","unstructured":"S.P.\u00a0Chatzis and D.\u00a0Kosmopoulos, A partially-observable Markov decision process for dealing with dynamically changing environments, in: Artificial Intelligence Applications and Innovations, L.\u00a0Iliadis, I.\u00a0Maglogiannis and H.\u00a0Papadopoulos, eds, Springer Berlin Heidelberg, Berlin, Heidelberg, 2014, pp.\u00a0111\u2013120. ISBN 978-3-662-44654-6.","DOI":"10.1007\/978-3-662-44654-6_11"},{"key":"10.3233\/SW-233531_ref6","unstructured":"M.\u00a0Dennis, N.\u00a0Jaques, E.\u00a0Vinitsky, A.\u00a0Bayen, S.J.\u00a0Russell, A.\u00a0Critch and S.\u00a0Levine, Emergent complexity and zero-shot transfer via unsupervised environment design, ArXiv (2020), arXiv:2012.02096."},{"issue":"12","key":"10.3233\/SW-233531_ref7","doi-asserted-by":"publisher","first-page":"3318","DOI":"10.1002\/hbm.25019","article-title":"Dark control: The default mode network as a reinforcement learning agent","volume":"41","author":"Dohmatob","year":"2020","journal-title":"Human Brain Mapping"},{"issue":"7","key":"10.3233\/SW-233531_ref8","doi-asserted-by":"crossref","first-page":"866","DOI":"10.1016\/j.robot.2010.03.007","article-title":"Probabilistic policy reuse for inter-task transfer learning","volume":"58","author":"Fern\u00e1ndez","year":"2010","journal-title":"Robotics and Autonomous Systems"},{"issue":"3","key":"10.3233\/SW-233531_ref9","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1109\/PROC.1973.9030","article-title":"The Viterbi algorithm","volume":"61","author":"Forney","year":"1973","journal-title":"Proceedings of the IEEE"},{"issue":"1","key":"10.3233\/SW-233531_ref10","first-page":"1437","article-title":"A comprehensive survey on safe reinforcement learning","volume":"16","author":"Garc\u0131a","year":"2015","journal-title":"Journal of Machine Learning Research"},{"issue":"6","key":"10.3233\/SW-233531_ref11","doi-asserted-by":"publisher","first-page":"2298","DOI":"10.1109\/TAC.2018.2866455","article-title":"Learning policies for Markov decision processes from data","volume":"64","author":"Hanawal","year":"2019","journal-title":"IEEE Transactions on Automatic Control"},{"issue":"3","key":"10.3233\/SW-233531_ref12","doi-asserted-by":"publisher","first-page":"1949","DOI":"10.1109\/LRA.2018.2808367","article-title":"Autonomous detection and experimental validation of affordances","volume":"3","author":"Kaiser","year":"2018","journal-title":"IEEE Robotics and Automation Letters"},{"key":"10.3233\/SW-233531_ref13","doi-asserted-by":"publisher","DOI":"10.1109\/CoG47356.2020.9231687"},{"key":"10.3233\/SW-233531_ref14","unstructured":"D.P.\u00a0Kingma and J.\u00a0Ba, Adam: A method for stochastic optimization, in: 3rd International Conference on Learning Representations, ICLR 2015, Y.\u00a0Bengio and Y.\u00a0LeCun, eds, Conference Track Proceedings, San Diego, CA, USA, May 7\u20139, 2015, 2015, http:\/\/arxiv.org\/abs\/1412.6980."},{"key":"10.3233\/SW-233531_ref15","doi-asserted-by":"publisher","DOI":"10.1109\/ICHI.2017.45"},{"key":"10.3233\/SW-233531_ref16","unstructured":"K.\u00a0Lu, R.\u00a0Li, X.\u00a0Chen, Z.\u00a0Zhao and H.\u00a0Zhang, Reinforcement learning-powered semantic communication via semantic similarity, ArXiv (2021), arXiv:2108.12121."},{"key":"10.3233\/SW-233531_ref17","doi-asserted-by":"publisher","DOI":"10.6084\/m9.figshare.22215079"},{"key":"10.3233\/SW-233531_ref19","unstructured":"T.\u00a0Mikolov, I.\u00a0Sutskever, K.\u00a0Chen, G.S.\u00a0Corrado and J.\u00a0Dean, Distributed representations of words and phrases and their compositionality, in: Advances in Neural Information Processing Systems, C.J.\u00a0Burges, L.\u00a0Bottou, M.\u00a0Welling, Z.\u00a0Ghahramani and K.Q.\u00a0Weinberger, eds, Vol.\u00a026, Curran Associates, Inc., 2013, https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2013\/file\/9aa42b31882ec039965f3c4923ce901b-Paper.pdf."},{"key":"10.3233\/SW-233531_ref20","doi-asserted-by":"crossref","unstructured":"L.\u00a0Miralles-Pechu\u00e1n, F.\u00a0Jim\u00e9nez, H.\u00a0Ponce and L.\u00a0Mart\u00ednez-Villase\u00f1or, A methodology based on deep Q-learning\/genetic algorithms for optimizing COVID-19 pandemic government actions, in: Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM \u201920, Association for Computing Machinery, New York, NY, USA, 2020, pp.\u00a01135\u20131144. ISBN 9781450368599.","DOI":"10.1145\/3340531.3412179"},{"issue":"7540","key":"10.3233\/SW-233531_ref22","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"issue":"4","key":"10.3233\/SW-233531_ref23","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1016\/j.robot.2004.07.005","article-title":"Integrating a flexible anthropomorphic, robot hand into the control, system of a humanoid robot","volume":"48","author":"Osswald","year":"2004","journal-title":"Robotics and Autonomous Systems"},{"key":"10.3233\/SW-233531_ref24","doi-asserted-by":"publisher","DOI":"10.1109\/SASO.2016.19"},{"key":"10.3233\/SW-233531_ref25","doi-asserted-by":"publisher","DOI":"10.4249\/scholarpedia.3698"},{"key":"10.3233\/SW-233531_ref26","unstructured":"D.\u00a0Poole and A.\u00a0Mackworth, Artificial Intelligence: Foundations of Computational Agents, 2nd edn, Cambridge University Press, Cambridge, UK, 2017. ISBN 978-0-521-51900-7."},{"key":"10.3233\/SW-233531_ref27","doi-asserted-by":"crossref","unstructured":"X.\u00a0Puig, K.\u00a0Ra, M.\u00a0Boben, J.\u00a0Li, T.\u00a0Wang, S.\u00a0Fidler and A.\u00a0Torralba, Virtualhome: Simulating household activities via programs, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp.\u00a08494\u20138502.","DOI":"10.1109\/CVPR.2018.00886"},{"issue":"1","key":"10.3233\/SW-233531_ref28","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1109\/MASSP.1986.1165342","article-title":"An introduction to hidden Markov models","volume":"3","author":"Rabiner","year":"1986","journal-title":"IEEE ASSP Magazine"},{"issue":"2","key":"10.3233\/SW-233531_ref29","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1007\/s10707-019-00345-2","article-title":"Using word embeddings to generate data-driven human agent decision-making from natural language","volume":"23","author":"Runck","year":"2019","journal-title":"GeoInformatica"},{"key":"10.3233\/SW-233531_ref30","unstructured":"F.L.D.\u00a0Silva, G.\u00a0Warnell, A.H.R.\u00a0Costa and P.\u00a0Stone, Agents teaching agents: A survey on inter-agent transfer learning, in: Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS \u201920, International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 2020, pp.\u00a02165\u20132167. ISBN 9781450375184."},{"issue":"7587","key":"10.3233\/SW-233531_ref31","doi-asserted-by":"publisher","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"10.3233\/SW-233531_ref32","first-page":"1","article-title":"A revealing introduction to hidden Markov models","author":"Stamp","year":"2004","journal-title":"Science"},{"key":"10.3233\/SW-233531_ref33","unstructured":"R.S.\u00a0Sutton and A.G.\u00a0Barto, Reinforcement Learning: An Introduction, 2nd edn, The MIT Press, 2018, http:\/\/incompleteideas.net\/book\/the-book-2nd.html."},{"key":"10.3233\/SW-233531_ref34","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1109\/CONFLUENCE.2018.8442725","volume-title":"Cognitive Robotics in Artificial Intelligence, in: 2018 8th International Conference on Cloud Computing","author":"Verma","year":"2018"},{"key":"10.3233\/SW-233531_ref35","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1016\/j.ins.2016.05.002","article-title":"A multi-agent reinforcement learning approach to dynamic service composition","volume":"363","author":"Wang","year":"2016","journal-title":"Information Sciences"},{"key":"10.3233\/SW-233531_ref37","doi-asserted-by":"publisher","DOI":"10.1109\/IROS51168.2021.9636660"},{"key":"10.3233\/SW-233531_ref38","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331203"},{"key":"10.3233\/SW-233531_ref39","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401171"}],"container-title":["Semantic Web"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/SW-233531","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T05:27:15Z","timestamp":1777613235000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/SW-233531"}},"subtitle":[],"editor":[{"given":"Agnieszka","family":"Lawrynowicz","sequence":"additional","affiliation":[{"name":"Poznan University of Technology, Poland"}],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2024,10,4]]},"references-count":36,"journal-issue":{"issue":"4"},"URL":"https:\/\/doi.org\/10.3233\/sw-233531","relation":{},"ISSN":["2210-4968","1570-0844"],"issn-type":[{"value":"2210-4968","type":"electronic"},{"value":"1570-0844","type":"print"}],"subject":[],"published":{"date-parts":[[2024,10,4]]}}}