{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,28]],"date-time":"2026-08-28T06:14:30Z","timestamp":1787897670986,"version":"build-2784847793"},"reference-count":51,"publisher":"Cambridge University Press (CUP)","issue":"1300","license":[{"start":{"date-parts":[[2022,2,21]],"date-time":"2022-02-21T00:00:00Z","timestamp":1645401600000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Aeronaut. j."],"published-print":{"date-parts":[[2022,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The high operational cost of aircraft, limited availability of air space, and strict safety regulations make training of fighter pilots increasingly challenging. By integrating Live, Virtual, and Constructive simulation resources, efficiency and effectiveness can be improved. In particular, if constructive simulations, which provide synthetic agents operating synthetic vehicles, were used to a higher degree, complex training scenarios could be realised at low cost, the need for support personnel could be reduced, and training availability could be improved. In this work, inspired by the recent improvements of techniques for artificial intelligence, we take a user perspective and investigate how intelligent, learning agents could help build future training systems. Through a domain analysis, a user study, and practical experiments, we identify important agent capabilities and characteristics, and then discuss design approaches and solution concepts for training systems to utilise learning agents for improved training value.<\/jats:p>","DOI":"10.1017\/aer.2022.8","type":"journal-article","created":{"date-parts":[[2022,2,21]],"date-time":"2022-02-21T14:13:16Z","timestamp":1645452796000},"page":"907-931","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":20,"title":["Design of simulation-based pilot training systems using machine learning agents"],"prefix":"10.1017","volume":"126","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4144-4893","authenticated-orcid":false,"given":"J.","family":"K\u00e4llstr\u00f6m","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"R.","family":"Granlund","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"F.","family":"Heintz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"56","published-online":{"date-parts":[[2022,2,21]]},"reference":[{"key":"S0001924022000082_ref43","doi-asserted-by":"crossref","unstructured":"[43] Bacon, P.L. , Harb, J. and Precup, D. The option-critic architecture, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017, pp 1726\u20131734.","DOI":"10.1609\/aaai.v31i1.10916"},{"key":"S0001924022000082_ref20","doi-asserted-by":"publisher","DOI":"10.3390\/electronics7110279"},{"key":"S0001924022000082_ref34","unstructured":"[34] Kingma, D.P. and Ba, J. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, 2014."},{"key":"S0001924022000082_ref3","volume-title":"Agent-based Models","author":"Gilbert","year":"2019"},{"key":"S0001924022000082_ref11","first-page":"1","article-title":"A framework for describing interaction between human operators and autonomous, automated, and manual control systems","author":"Lundberg","year":"2020","journal-title":"Cognition, Technology and Work"},{"key":"S0001924022000082_ref35","unstructured":"[35] K\u00e4llstr\u00f6m, J. Adaptive agent-based simulation for individualized training, Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), May 2020, pp 2193\u20132195."},{"key":"S0001924022000082_ref10","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1724-z"},{"key":"S0001924022000082_ref24","first-page":"1","article-title":"UAV maneuvering target tracking in uncertain environments based on deep reinforcement learning and meta-learning","volume":"12","author":"Li","year":"2020","journal-title":"Remote Sensing"},{"key":"S0001924022000082_ref39","unstructured":"[39] Ho, J. and Ermon, S. Generative adversarial imitation learning, Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016, pp 4572\u20134580."},{"key":"S0001924022000082_ref41","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(99)00052-1"},{"key":"S0001924022000082_ref42","unstructured":"[42] Smith, M. , Hoof, H. and Pineau, J. An inference-based policy gradient method for learning options, Proceedings of the International Conference on Machine Learning, 2018, pp 4703\u20134712."},{"key":"S0001924022000082_ref23","first-page":"1","article-title":"Deep reinforcement learning approach with multiple experience pools for UAV\u2019s autonomous motion planning in complex unknown environments","volume":"20","author":"Hu","year":"2020","journal-title":"Sensors"},{"key":"S0001924022000082_ref18","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-10-6463-0_24"},{"key":"S0001924022000082_ref8","doi-asserted-by":"publisher","DOI":"10.1038\/nature24270"},{"key":"S0001924022000082_ref46","doi-asserted-by":"crossref","unstructured":"[46] Hayes, C.F. , R\u0103dulescu, R. , Bargiacchi, E. , K\u00e4llstr\u00f6m, J. , Macfarlane, M. , Reymond, M. , Verstraeten, T. , Zintgraf, L.M. , Dazeley, R. , Heintz, F. , Howley, E. , Irissappane, A.A. , Mannion, P. , Now\u00e9, A. , Ramos, G. , Restelli, M. , Vamplew, P. and Roijers, D.M. A Practical Guide to Multi-Objective Reinforcement Learning and Planning, arXiv preprint arXiv:2103.09568, 2021.","DOI":"10.1007\/s10458-022-09552-y"},{"key":"S0001924022000082_ref31","unstructured":"[31] Matplotlib matplotlib.pyplot.boxplot, https:\/\/matplotlib.org\/stable\/api\/_as_gen\/matplotlib.pyplot.boxplot.html."},{"key":"S0001924022000082_ref1","doi-asserted-by":"crossref","unstructured":"[1] Page, E.H. and Smith, R. Introduction to military training simulation: a guide for discrete event simulationists, Proceedings of the Winter Simulation Conference, 1998, vol. 1, pp 53\u201360.","DOI":"10.1109\/WSC.1998.744899"},{"key":"S0001924022000082_ref37","first-page":"1","article-title":"Algorithms for inverse reinforcement learning","volume":"1","author":"Ng","year":"2000","journal-title":"ICML"},{"key":"S0001924022000082_ref45","unstructured":"[45] Vezhnevets, A.S. , Osindero, S. , Schaul, T. , Heess, N. , Jaderberg, M. , Silver, D. and Kavukcuoglu, K. Feudal networks for hierarchical reinforcement learning, Proceedings of the International Conference on Machine Learning, 2017, pp 3540\u20133549."},{"key":"S0001924022000082_ref6","doi-asserted-by":"publisher","DOI":"10.3384\/ecp19162011"},{"key":"S0001924022000082_ref28","doi-asserted-by":"publisher","DOI":"10.1109\/SMC42975.2020.9283492"},{"key":"S0001924022000082_ref26","first-page":"6379","article-title":"Multi-agent actor-critic for mixed cooperative-competitive environments","author":"Lowe","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"S0001924022000082_ref50","unstructured":"[50] Florensa, C. , Held, D. , Wulfmeier, M. , Zhang, M. and Abbeel, P. Reverse curriculum generation for reinforcement learning, Proceedings of the Conference on Robot Learning, 2017, pp 482\u2013495."},{"key":"S0001924022000082_ref5","volume-title":"Deep Learning","author":"Goodfellow","year":"2016"},{"key":"S0001924022000082_ref4","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"2018"},{"key":"S0001924022000082_ref29","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.2967225"},{"key":"S0001924022000082_ref33","doi-asserted-by":"publisher","DOI":"10.1201\/9780429489105"},{"key":"S0001924022000082_ref49","doi-asserted-by":"publisher","DOI":"10.1109\/TCIAIG.2012.2186810"},{"key":"S0001924022000082_ref12","unstructured":"[12] Lillicrap, T.P. , Hunt, J.J. , Pritzel, A. , Heess, N. , Erez, T. , Tassa, Y. , Silver, D. and Wierstra, D. Continuous control with deep reinforcement learning, Proceedings of the 4th International Conference on Learning Representations (ICLR), May 2016."},{"key":"S0001924022000082_ref2","unstructured":"[2] Roessingh, J.J. and Verhaaf, G.G. Training effectiveness of embedded training in a (multi-) fighter environment, Tech Rep, National Aerospace Lab Amsterdam (Netherlands), 2009."},{"key":"S0001924022000082_ref16","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992698"},{"key":"S0001924022000082_ref15","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"S0001924022000082_ref7","doi-asserted-by":"publisher","DOI":"10.1038\/nature16961"},{"key":"S0001924022000082_ref14","unstructured":"[14] Schulman, J. , Wolski, F. , Dhariwal, P. , Radford, A. and Klimov, O. Proximal policy optimization algorithms, arXiv preprint arXiv:1707.06347, 2017."},{"key":"S0001924022000082_ref17","unstructured":"[17] Rijken, R. and Toubman, A. The future of autonomous air combat behavior, Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2016, pp 3089\u20133094."},{"key":"S0001924022000082_ref13","unstructured":"[13] Andrychowicz, M. , Wolski, F. , Ray, A. , Schneider, J. , Fong, R. , Welinder, P. , McGrew, B. , Tobin, J. , Abbeel, P. and Zaremba, W. Hindsight experience replay, Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, pp 5055\u20135065."},{"key":"S0001924022000082_ref21","doi-asserted-by":"publisher","DOI":"10.1109\/ICCA.2019.8899703"},{"key":"S0001924022000082_ref32","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1080\/00031305.1978.10479236","article-title":"Variations of box plots","volume":"32","author":"McGill","year":"1978","journal-title":"Am. Stat."},{"key":"S0001924022000082_ref38","doi-asserted-by":"publisher","DOI":"10.1145\/1015330.1015430"},{"key":"S0001924022000082_ref44","article-title":"DAC: The double actor-critic architecture for learning options","volume":"32","author":"Zhang","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"key":"S0001924022000082_ref47","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-019-09433-x"},{"key":"S0001924022000082_ref25","doi-asserted-by":"publisher","DOI":"10.3390\/rs12040640"},{"key":"S0001924022000082_ref48","unstructured":"[48] K\u00e4llstr\u00f6m, J. and Heintz, F. Tunable dynamics in agent-based simulation using multi-objective reinforcement learning, Proceeding of the Adaptive and Learning Agents Workshop (ALA-19) at AAMAS, May 2019, pp 1\u20137."},{"key":"S0001924022000082_ref27","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2933454"},{"key":"S0001924022000082_ref36","doi-asserted-by":"publisher","DOI":"10.1613\/jair.3987"},{"key":"S0001924022000082_ref9","doi-asserted-by":"publisher","DOI":"10.1126\/science.aau6249"},{"key":"S0001924022000082_ref22","doi-asserted-by":"publisher","DOI":"10.3390\/electronics9071121"},{"key":"S0001924022000082_ref19","doi-asserted-by":"publisher","DOI":"10.1109\/CAC.2018.8623434"},{"key":"S0001924022000082_ref30","doi-asserted-by":"publisher","DOI":"10.3390\/app10155198"},{"key":"S0001924022000082_ref40","unstructured":"[40] Rabinowitz, N. , Perbet, F. , Song, F. , Zhang, C. , Eslami, S.A. and Botvinick, M. Machine theory of mind, Proceedings of the International Conference on Machine Learning, 2018, pp 4218\u20134227."},{"key":"S0001924022000082_ref51","unstructured":"[51] Czarnecki, W. , Jayakumar, S. , Jaderberg, M. , Hasenclever, L. , Teh, Y.W. , Heess, N. , Osindero, S. and Pascanu, R. Mix & match agent curricula for reinforcement learning, Proceedings of the International Conference on Machine Learning, 2018, pp 1087\u20131095."}],"container-title":["The Aeronautical Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S0001924022000082","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,18]],"date-time":"2024-09-18T23:15:29Z","timestamp":1726701329000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S0001924022000082\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,21]]},"references-count":51,"journal-issue":{"issue":"1300","published-print":{"date-parts":[[2022,6]]}},"alternative-id":["S0001924022000082"],"URL":"https:\/\/doi.org\/10.1017\/aer.2022.8","relation":{},"ISSN":["0001-9240","2059-6464"],"issn-type":[{"value":"0001-9240","type":"print"},{"value":"2059-6464","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,21]]},"assertion":[{"value":"\u00a9 The Author(s), 2022. Published by Cambridge University Press on behalf of Royal Aeronautical Society","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http:\/\/creativecommons.org\/licenses\/by\/4.0\/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.","name":"license","label":"License","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}