{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T03:20:04Z","timestamp":1783653604139,"version":"3.55.0"},"reference-count":95,"publisher":"SAGE Publications","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["AIC"],"published-print":{"date-parts":[[2022,9,20]]},"abstract":"<jats:p>We are concerned with the construction, formal verification, and safety assurance of dependable multiagent systems. For the case where the system (agents and their environment) can be explicitly modelled, we develop formal verification methods over several logic languages, such as temporal epistemic logic and strategy logic, to reason about the knowledge and strategy of the agents. For the case where the system cannot be explicitly modelled, we study multiagent deep reinforcement learning, aiming to develop efficient and scalable learning methods for cooperative multiagent tasks. In addition to these, we develop (both formal and simulation-based) verification methods for the neural network based perception agent that is trained with supervised learning, considering its safety and robustness against attacks from an adversarial agent, and other approaches (such as explainable AI, reliability assessment, and safety argument) for the analysis and assurance of the learning components. Our ultimate objective is to combine formal methods, machine learning, and reliability engineering to not only develop dependable learning-enabled multiagent systems but also provide rigorous methods for the verification and assurance of such systems.<\/jats:p>","DOI":"10.3233\/aic-220128","type":"journal-article","created":{"date-parts":[[2022,9,6]],"date-time":"2022-09-06T11:28:11Z","timestamp":1662463691000},"page":"407-420","source":"Crossref","is-referenced-by-count":2,"title":["Dependable learning-enabled multiagent systems"],"prefix":"10.1177","volume":"35","author":[{"given":"Xiaowei","family":"Huang","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Liverpool, Liverpool, U.K."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bei","family":"Peng","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Liverpool, Liverpool, U.K."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xingyu","family":"Zhao","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Liverpool, Liverpool, U.K."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","reference":[{"key":"10.3233\/AIC-220128_ref1","doi-asserted-by":"publisher","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking inside the black-box: A survey on explainable artificial intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"issue":"1","key":"10.3233\/AIC-220128_ref2","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1023\/A:1008739929481","article-title":"Reactive modules","volume":"15","author":"Alur","year":"1999","journal-title":"Formal Methods in System Design"},{"key":"10.3233\/AIC-220128_ref3","doi-asserted-by":"publisher","DOI":"10.1109\/ITSC.2008.4732641"},{"key":"10.3233\/AIC-220128_ref4","doi-asserted-by":"publisher","DOI":"10.23919\/DATE.2019.8715114"},{"key":"10.3233\/AIC-220128_ref5","doi-asserted-by":"crossref","unstructured":"S.\u00a0Bensalem, C.-H.\u00a0Cheng, X.\u00a0Huang, P.\u00a0Katsaros, A.\u00a0Molin, D.\u00a0Nickovic and D.\u00a0Peled, Formal specification for learning-enabled autonomous systems, in: FoMLAS2022, 2022.","DOI":"10.1007\/978-3-031-21222-2_8"},{"key":"10.3233\/AIC-220128_ref6","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-84996-086-1_4"},{"issue":"9","key":"10.3233\/AIC-220128_ref8","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1109\/MC.2019.2914775","article-title":"Disruptive innovations and disruptive assurance: Assuring machine learning and autonomy","volume":"52","author":"Bloomfield","year":"2019","journal-title":"Computer"},{"key":"10.3233\/AIC-220128_ref10","unstructured":"D.\u00a0Borsa, A.\u00a0Barreto, J.\u00a0Quan, D.\u00a0Mankowitz, R.\u00a0Munos, H.\u00a0Van Hasselt, D.\u00a0Silver and T.\u00a0Schaul, Universal successor features approximators, in: International Conference on Learning Representations, 2019."},{"key":"10.3233\/AIC-220128_ref12","doi-asserted-by":"crossref","unstructured":"K.\u00a0Cai, C.X.\u00a0Lu and X.\u00a0Huang, STUN: Self-teaching uncertainty estimation for place recognition, in: IROS2022, 2022.","DOI":"10.1109\/IROS47612.2022.9981546"},{"key":"10.3233\/AIC-220128_ref13","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.49"},{"key":"10.3233\/AIC-220128_ref14","unstructured":"Y.\u00a0Chow, O.\u00a0Nachum, E.A.\u00a0Du\u00e9\u00f1ez-Guzm\u00e1n and M.\u00a0Ghavamzadeh, A Lyapunov-based approach to safe reinforcement learning, in: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montr\u00e9al, Canada, December 3\u20138, 2018, 2018, pp.\u00a08103\u20138112."},{"issue":"4","key":"10.3233\/AIC-220128_ref15","first-page":"611","article-title":"Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning","volume":"6","author":"d\u2019Avila Garcez","year":"2019","journal-title":"Journal of Applied Logics"},{"key":"10.3233\/AIC-220128_ref16","doi-asserted-by":"crossref","unstructured":"Y.\u00a0Dong, X.\u00a0Zhao and X.\u00a0Huang, Dependability analysis of deep reinforcement learning based robotics and autonomous systems through probabilistic model checking, in: IROS2022, 2022.","DOI":"10.1109\/IROS47612.2022.9981794"},{"issue":"4","key":"10.3233\/AIC-220128_ref17","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1007\/s004460050038","article-title":"Knowledge-based programs","volume":"10","author":"Fagin","year":"1997","journal-title":"Distributed Computing"},{"key":"10.3233\/AIC-220128_ref18","doi-asserted-by":"crossref","unstructured":"M.\u00a0Fisher, R.C.\u00a0Cardoso, E.C.\u00a0Collins, C.\u00a0Dadswell, L.A.\u00a0Dennis, C.\u00a0Dixon, M.\u00a0Farrell, A.\u00a0Ferrando, X.\u00a0Huang, M.\u00a0Jump, G.\u00a0Kourtis, A.\u00a0Lisitsa, M.\u00a0Luckcuck, S.\u00a0Luo, V.\u00a0Page, F.\u00a0Papacchini and M.\u00a0Webster, An overview of verification and validation challenges for inspection robots, Robotics 10(2) (2021).","DOI":"10.3390\/robotics10020067"},{"key":"10.3233\/AIC-220128_ref19","doi-asserted-by":"crossref","unstructured":"J.\u00a0Foerster, G.\u00a0Farquhar, T.\u00a0Afouras, N.\u00a0Nardelli and S.\u00a0Whiteson, Counterfactual multi-agent policy gradients, in: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","DOI":"10.1609\/aaai.v32i1.11794"},{"key":"10.3233\/AIC-220128_ref20","unstructured":"T.\u00a0Gupta, A.\u00a0Mahajan, B.\u00a0Peng, W.\u00a0B\u00f6hmer and S.\u00a0Whiteson, Uneven: Universal value exploration for multi-agent reinforcement learning, in: International Conference on Machine Learning, PMLR, 2021, pp.\u00a03930\u20133941."},{"issue":"12","key":"10.3233\/AIC-220128_ref21","doi-asserted-by":"publisher","first-page":"1402","DOI":"10.1109\/32.62448","article-title":"Partition testing does not inspire confidence","volume":"16","author":"Hamlet","year":"1990","journal-title":"IEEE Tran. on Software Engineering"},{"issue":"6","key":"10.3233\/AIC-220128_ref22","doi-asserted-by":"publisher","first-page":"750","DOI":"10.1007\/s10458-019-09421-1","article-title":"A survey and critique of multiagent deep reinforcement learning","volume":"33","author":"Hernandez-Leal","year":"2019","journal-title":"Autonomous Agents and Multi-Agent Systems"},{"issue":"8","key":"10.3233\/AIC-220128_ref23","doi-asserted-by":"publisher","first-page":"666","DOI":"10.1145\/359576.359585","article-title":"Communicating sequential processes","volume":"21","author":"Hoare","year":"1978","journal-title":"Commun. ACM"},{"key":"10.3233\/AIC-220128_ref24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TR.2021.3080664","article-title":"Coverage-guided testing for recurrent neural networks","author":"Huang","year":"2021","journal-title":"IEEE Transactions on Reliability"},{"key":"10.3233\/AIC-220128_ref25","doi-asserted-by":"publisher","DOI":"10.1109\/IROS45743.2020.9340720"},{"key":"10.3233\/AIC-220128_ref26","unstructured":"X.\u00a0Huang, Bounded planning for strategic goals with incomplete information and perfect recall, in: AAMAS 2013, Saint Paul, MN, USA, May 6\u201310, 2013, 2013, pp.\u00a0885\u2013892."},{"key":"10.3233\/AIC-220128_ref27","unstructured":"X.\u00a0Huang, Diagnosability in concurrent probabilistic systems, in: AAMAS 2013, 2013, pp.\u00a0853\u2013860."},{"key":"10.3233\/AIC-220128_ref28","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1016\/j.artint.2015.01.005","article-title":"Bounded model checking of strategy ability with perfect recall","volume":"222","author":"Huang","year":"2015","journal-title":"Artif. Intell."},{"key":"10.3233\/AIC-220128_ref29","unstructured":"X.\u00a0Huang, Q.\u00a0Chen, J.\u00a0Meng and K.\u00a0Su, Reconfigurability in reactive multiagent systems, in: IJCAI 2016, 2016, pp.\u00a0315\u2013321."},{"key":"10.3233\/AIC-220128_ref30","unstructured":"X.\u00a0Huang, Q.\u00a0Chen and K.\u00a0Su, The complexity of model checking succinct multiagent systems, in: IJCAI 2015, 2015, pp.\u00a01076\u20131082."},{"key":"10.3233\/AIC-220128_ref31","doi-asserted-by":"crossref","unstructured":"X.\u00a0Huang, Q.\u00a0Chen and K.\u00a0Su, Strengthening agents strategic ability with communication, in: AAAI 2016, 2016, pp.\u00a02509\u20132515.","DOI":"10.1609\/aaai.v30i1.10121"},{"key":"10.3233\/AIC-220128_ref32","doi-asserted-by":"crossref","unstructured":"X.\u00a0Huang, G.\u00a0Jin and W.\u00a0Ruan, Machine Learning Safety, Springer, 2022.","DOI":"10.1007\/978-981-19-6814-3"},{"key":"10.3233\/AIC-220128_ref33","doi-asserted-by":"publisher","DOI":"10.1016\/j.cosrev.2020.100270"},{"key":"10.3233\/AIC-220128_ref34","doi-asserted-by":"crossref","unstructured":"X.\u00a0Huang and M.\u00a0Kwiatkowska, Model checking probabilistic knowledge: A PSPACE case, in: AAAI 2016, 2016, pp.\u00a02516\u20132522.","DOI":"10.1609\/aaai.v30i1.10122"},{"issue":"4","key":"10.3233\/AIC-220128_ref35","doi-asserted-by":"publisher","first-page":"21:1","DOI":"10.1145\/3329123","article-title":"Reasoning about cognitive trust in stochastic multiagent systems","volume":"20","author":"Huang","year":"2019","journal-title":"ACM Trans. Comput. Log."},{"key":"10.3233\/AIC-220128_ref36","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-63387-9_1"},{"key":"10.3233\/AIC-220128_ref37","doi-asserted-by":"crossref","unstructured":"X.\u00a0Huang and M.Z.\u00a0Kwiatkowska, Reasoning about cognitive trust in stochastic multiagent systems, in: AAAI 2017, 2017, pp.\u00a03768\u20133774.","DOI":"10.1609\/aaai.v31i1.11050"},{"key":"10.3233\/AIC-220128_ref38","unstructured":"X.\u00a0Huang and C.\u00a0Luo, A logic of probabilistic knowledge and strategy, in: AAMAS 2013, 2013, pp.\u00a0845\u2013852."},{"key":"10.3233\/AIC-220128_ref39","unstructured":"X.\u00a0Huang, C.\u00a0Luo and R.\u00a0van der Meyden, Improved bounded model checking for a fair branching-time temporal epistemic logic, in: AAMAS2010, W.\u00a0van der Hoek, G.A.\u00a0Kaminka, Y.\u00a0Lesp\u00e9rance, M.\u00a0Luck and S.\u00a0Sen, eds, 2010, pp.\u00a01403\u20131404."},{"key":"10.3233\/AIC-220128_ref40","doi-asserted-by":"crossref","unstructured":"X.\u00a0Huang, C.\u00a0Luo and R.\u00a0van der Meyden, Symbolic model checking of probabilistic knowledge, in: TARK-2011, 2011, pp.\u00a0177\u2013186.","DOI":"10.1145\/2000378.2000399"},{"key":"10.3233\/AIC-220128_ref41","unstructured":"X.\u00a0Huang, P.\u00a0Maupin and R.\u00a0van der Meyden, Model checking knowledge in pursuit evasion games, in: IJCAI2011, T.\u00a0Walsh, ed., 2011, pp.\u00a0240\u2013245."},{"key":"10.3233\/AIC-220128_ref42","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/153"},{"key":"10.3233\/AIC-220128_ref43","unstructured":"X.\u00a0Huang, J.\u00a0Ruan, Q.\u00a0Chen and K.\u00a0Su, Normative multiagent systems: A dynamic generalization, in: IJCAI2016, 2016, pp.\u00a01123\u20131129."},{"key":"10.3233\/AIC-220128_ref44","doi-asserted-by":"crossref","unstructured":"X.\u00a0Huang, W.\u00a0Ruan, Q.\u00a0Tang and X.\u00a0Zhao, Bridging formal methods and machine learning with global optimisation, in: ICFEM2022, 2022.","DOI":"10.1007\/978-3-031-17244-1_1"},{"key":"10.3233\/AIC-220128_ref45","unstructured":"X.\u00a0Huang, K.\u00a0Su and C.\u00a0Zhang, Probabilistic alternating-time temporal logic of incomplete information and synchronous perfect recall, in: AAAI 2012, 2012."},{"key":"10.3233\/AIC-220128_ref46","unstructured":"X.\u00a0Huang and R.\u00a0van der Meyden, The complexity of epistemic model checking: Clock semantics and branching time, in: ECAI 2010, H.\u00a0Coelho, R.\u00a0Studer and M.J.\u00a0Wooldridge, eds, Vol.\u00a0215, 2010, pp.\u00a0549\u2013554."},{"key":"10.3233\/AIC-220128_ref47","unstructured":"X.\u00a0Huang and R.\u00a0van der Meyden, Synthesizing strategies for epistemic goals by epistemic model checking: An application to pursuit evasion games, in: AAAI 2012, 2012."},{"key":"10.3233\/AIC-220128_ref48","unstructured":"X.\u00a0Huang and R.\u00a0van der Meyden, Symbolic synthesis of knowledge-based program implementations with synchronous semantics, in: TARK2013, B.C.\u00a0Schipper, ed., 2013."},{"key":"10.3233\/AIC-220128_ref49","doi-asserted-by":"crossref","unstructured":"X.\u00a0Huang and R.\u00a0van der Meyden, Symbolic model checking epistemic strategy logic, in: AAAI2014, C.E.\u00a0Brodley and P.\u00a0Stone, eds, 2014, pp.\u00a01426\u20131432.","DOI":"10.1609\/aaai.v28i1.8894"},{"key":"10.3233\/AIC-220128_ref50","unstructured":"X.\u00a0Huang and R.\u00a0van der Meyden, A temporal logic of strategic knowledge, in: KR2014, C.\u00a0Baral, G.D.\u00a0Giacomo and T.\u00a0Eiter, eds, 2014."},{"key":"10.3233\/AIC-220128_ref51","doi-asserted-by":"crossref","unstructured":"X.\u00a0Huang and R.\u00a0van der Meyden, Symbolic synthesis for epistemic specifications with observational semantics, in: TACAS 2014, E.\u00a0\u00c1brah\u00e1m and K.\u00a0Havelund, eds, 2014, pp.\u00a0455\u2013469.","DOI":"10.1007\/978-3-642-54862-8_39"},{"issue":"4","key":"10.3233\/AIC-220128_ref52","doi-asserted-by":"publisher","first-page":"26:1","DOI":"10.1145\/3233769","article-title":"An epistemic strategy logic","volume":"19","author":"Huang","year":"2018","journal-title":"ACM Trans. Comput. Log."},{"key":"10.3233\/AIC-220128_ref53","unstructured":"S.\u00a0Iqbal, C.A.S.\u00a0De Witt, B.\u00a0Peng, W.\u00a0B\u00f6hmer, S.\u00a0Whiteson and F.\u00a0Sha, Randomized entity-wise factorization for multi-agent reinforcement learning, in: International Conference on Machine Learning, PMLR, 2021, pp.\u00a04596\u20134606."},{"key":"10.3233\/AIC-220128_ref54","doi-asserted-by":"crossref","unstructured":"G.\u00a0Jin, X.\u00a0Yi, W.\u00a0Huang, S.\u00a0Schewe and X.\u00a0Huang, Enhancing adversarial training with second-order statistics of weights, in: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp.\u00a015273\u201315283.","DOI":"10.1109\/CVPR52688.2022.01484"},{"key":"10.3233\/AIC-220128_ref55","unstructured":"G.\u00a0Jin, X.\u00a0Yi, L.\u00a0Zhang, L.\u00a0Zhang, S.\u00a0Schewe and X.\u00a0Huang, How does weight correlation affect the generalisation ability of deep neural networks, in: NeurIPS\u201920, 2020."},{"key":"10.3233\/AIC-220128_ref56","unstructured":"C.W.\u00a0Johnson, The increasing risks of risk assessment: On the rise of artificial intelligence and non-determinism in safety-critical systems, in: The 26th Safety-Critical Systems Symposium, Safety-Critical Systems Club, York, UK, 2018, p.\u00a015."},{"key":"10.3233\/AIC-220128_ref57","unstructured":"D.\u00a0Koller and R.\u00a0Parr, Computing factored value functions for policies in structured MDPs, in: Proceedings of IJCAI, 1999, pp.\u00a01332\u20131339."},{"issue":"3\u20134","key":"10.3233\/AIC-220128_ref58","doi-asserted-by":"publisher","first-page":"373","DOI":"10.1177\/0278364907085560","article-title":"Distributed self-reconfiguration of M-TRAN III modular robotic system","volume":"27","author":"Kurokawa","year":"2008","journal-title":"The International Journal of Robotics Research"},{"issue":"4","key":"10.3233\/AIC-220128_ref59","first-page":"45","article-title":"Operability studies and hazard analysis","volume":"70","author":"Lawley","year":"1974","journal-title":"Chem. Eng. Prog."},{"key":"10.3233\/AIC-220128_ref60","unstructured":"N.G.\u00a0Leveson, Engineering a Safer World: Systems Thinking Applied to Safety, The MIT Press, 2016."},{"key":"10.3233\/AIC-220128_ref61","doi-asserted-by":"crossref","unstructured":"J.\u00a0Li, J.\u00a0Liu, P.\u00a0Yang, L.\u00a0Chen, X.\u00a0Huang and L.\u00a0Zhang, Analyzing deep neural networks with symbolic propagation: Towards higher precision and faster verification, in: SAS2019, Springer, 2019, pp.\u00a0296\u2013319.","DOI":"10.1007\/978-3-030-32304-2_15"},{"key":"10.3233\/AIC-220128_ref62","doi-asserted-by":"crossref","unstructured":"R.\u00a0Li, J.\u00a0Li, C.-C.\u00a0Huang, P.\u00a0Yang, X.\u00a0Huang, L.\u00a0Zhang, B.\u00a0Xue and H.\u00a0Hermanns, PRODeep: A platform for robustness verification of deep neural networks, in: Proc. of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC\/FSE 2020, ACM, New York, NY, USA, 2020, pp.\u00a01630\u20131634.","DOI":"10.1145\/3368089.3417918"},{"key":"10.3233\/AIC-220128_ref63","doi-asserted-by":"publisher","DOI":"10.1145\/336512.336551"},{"key":"10.3233\/AIC-220128_ref65","unstructured":"R.\u00a0Lowe, Y.\u00a0Wu, A.\u00a0Tamar, J.\u00a0Harb, O.P.\u00a0Abbeel and I.\u00a0Mordatch, Multi-agent actor-critic for mixed cooperative-competitive environments, in: Advances in Neural Information Processing Systems, 2017, pp.\u00a06379\u20136390."},{"key":"10.3233\/AIC-220128_ref66","unstructured":"X.\u00a0Lyu, Y.\u00a0Xiao, B.\u00a0Daley and C.\u00a0Amato, Contrasting centralized and decentralized critics in multi-agent reinforcement learning, in: Proceedings of the 20th International Conference on Autonomous Agents and Multi-Agent Systems, 2021."},{"key":"10.3233\/AIC-220128_ref67","unstructured":"A.\u00a0Mahajan, T.\u00a0Rashid, M.\u00a0Samvelyan and S.\u00a0Whiteson, Maven: Multi-agent variational exploration, in: Advances in Neural Information Processing Systems, 2019, pp.\u00a07611\u20137622."},{"key":"10.3233\/AIC-220128_ref68","doi-asserted-by":"crossref","unstructured":"R.\u00a0Milner, A Calculus of Communicating Systems, Springer, 1980.","DOI":"10.1007\/3-540-10235-3"},{"issue":"2","key":"10.3233\/AIC-220128_ref70","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1109\/52.199724","article-title":"Operational profiles in software-reliability engineering","volume":"10","author":"Musa","year":"1993","journal-title":"IEEE Software"},{"key":"10.3233\/AIC-220128_ref71","doi-asserted-by":"crossref","unstructured":"F.A.\u00a0Oliehoek, C.\u00a0Amato et al., A Concise Introduction to Decentralized POMDPs, Vol.\u00a01, Springer, 2016.","DOI":"10.1007\/978-3-319-28929-8_1"},{"key":"10.3233\/AIC-220128_ref72","unstructured":"OpenAI, openai\/baselines, OpenAI, 2020, original-date: 2017-05-24T01:58:13Z."},{"key":"10.3233\/AIC-220128_ref73","unstructured":"L.\u00a0Pan, T.\u00a0Rashid, B.\u00a0Peng, L.\u00a0Huang and S.\u00a0Whiteson, Regularized softmax deep multi-agent Q-learning, in: Advances in Neural Information Processing Systems, 2021."},{"issue":"6","key":"10.3233\/AIC-220128_ref74","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1109\/TEVC.2006.880330","article-title":"Biasing coevolutionary search for optimal multiagent behaviors","volume":"10","author":"Panait","year":"2006","journal-title":"IEEE Transactions on Evolutionary Computation"},{"key":"10.3233\/AIC-220128_ref75","unstructured":"B.\u00a0Peng, T.\u00a0Rashid, C.\u00a0Schroeder de Witt, P.-A.\u00a0Kamienny, P.\u00a0Torr, W.\u00a0B\u00f6hmer and S.\u00a0Whiteson, Facmac: Factored multi-agent centralised policy gradients, in: Advances in Neural Information Processing Systems, 2021."},{"key":"10.3233\/AIC-220128_ref76","unstructured":"Y.\u00a0Qi, P.R.\u00a0Conmy, W.\u00a0Huang, X.\u00a0Zhao and X.\u00a0Huang, A hierarchical HAZOP-like safety analysis for learning-enabled systems, in: AISafety\u201922 Workshop at IJCAI\u201922, 2022."},{"key":"10.3233\/AIC-220128_ref77","unstructured":"T.\u00a0Rashid, G.\u00a0Farquhar, B.\u00a0Peng and S.\u00a0Whiteson, Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning, in: Advances in Neural Information Processing Systems, 2020, pp.\u00a010199\u201310210."},{"key":"10.3233\/AIC-220128_ref78","unstructured":"T.\u00a0Rashid, M.\u00a0Samvelyan, C.S.\u00a0Witt, G.\u00a0Farquhar, J.\u00a0Foerster and S.\u00a0Whiteson, QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning, in: International Conference on Machine Learning, 2018, pp.\u00a04292\u20134301."},{"key":"10.3233\/AIC-220128_ref79","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"10.3233\/AIC-220128_ref80","doi-asserted-by":"crossref","unstructured":"W.\u00a0Ruan, X.\u00a0Huang and M.\u00a0Kwiatkowska, Reachability analysis of deep neural networks with provable guarantees, in: Proc. of the 27th Int. Joint Conf. on Artificial Intelligence, IJCAI-18, 2018, pp.\u00a02651\u20132659.","DOI":"10.24963\/ijcai.2018\/368"},{"key":"10.3233\/AIC-220128_ref81","doi-asserted-by":"crossref","unstructured":"W.\u00a0Ruan, M.\u00a0Wu, Y.\u00a0Sun, X.\u00a0Huang, D.\u00a0Kroening and M.\u00a0Kwiatkowska, Global robustness evaluation of deep neural networks with provable guarantees for the Hamming distance, in: Proc. of the 28th Int. Joint Conf. on Artificial Intelligence, IJCAI-19, 2019, pp.\u00a05944\u20135952.","DOI":"10.24963\/ijcai.2019\/824"},{"issue":"3","key":"10.3233\/AIC-220128_ref83","doi-asserted-by":"publisher","first-page":"197","DOI":"10.3233\/AIC-210084","article-title":"Neuro-symbolic artificial intelligence","volume":"34","author":"Sarker","year":"2021","journal-title":"AI Communications"},{"key":"10.3233\/AIC-220128_ref85","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58604-1_24"},{"key":"10.3233\/AIC-220128_ref86","doi-asserted-by":"crossref","unstructured":"Y.\u00a0Sun, X.\u00a0Huang, D.\u00a0Kroening, J.\u00a0Sharp, M.\u00a0Hill and R.\u00a0Ashmore, Structural test coverage criteria for deep neural networks, in: ICSE\u201919 \u2013 Companion, IEEE Press, Piscataway, NJ, USA, 2019, pp.\u00a0320\u2013321.","DOI":"10.1109\/ICSE-Companion.2019.00134"},{"key":"10.3233\/AIC-220128_ref87","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238172"},{"key":"10.3233\/AIC-220128_ref88","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9196932"},{"key":"10.3233\/AIC-220128_ref90","unstructured":"T.\u00a0Wang, T.\u00a0Gupta, A.\u00a0Mahajan, B.\u00a0Peng, S.\u00a0Whiteson and C.\u00a0Zhang, Rode: Learning roles to decompose multi-agent tasks, in: International Conference on Learning Representations, 2021."},{"key":"10.3233\/AIC-220128_ref91","unstructured":"T.\u00a0Wang, R.\u00a0Liao, J.\u00a0Ba and S.\u00a0Fidler, NerveNet: Learning structured policy with graph neural networks, in: 6th International Conference on Learning Representations, ICLR, 2018."},{"key":"10.3233\/AIC-220128_ref92","unstructured":"S.\u00a0Webb, T.\u00a0Rainforth, Y.W.\u00a0Teh and M.P.\u00a0Kumar, A statistical approach to assessing neural network robustness, in: 7th Int. Conf. Learning Representations (ICLR\u201919), OpenReview.net, New Orleans, LA, USA, 2019."},{"key":"10.3233\/AIC-220128_ref93","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-89960-2_22"},{"key":"10.3233\/AIC-220128_ref94","doi-asserted-by":"publisher","first-page":"298","DOI":"10.1016\/j.tcs.2019.05.046","article-title":"A game-based approximate verification of deep neural networks with provable guarantees","volume":"807","author":"Wu","year":"2020","journal-title":"Theoretical Computer Science"},{"key":"10.3233\/AIC-220128_ref95","doi-asserted-by":"crossref","unstructured":"P.\u00a0Xu, W.\u00a0Ruan and X.\u00a0Huang, Quantifying safety risks of deep neural networks, Complex & Intelligent Systems (2022).","DOI":"10.1007\/s40747-022-00790-x"},{"issue":"3","key":"10.3233\/AIC-220128_ref96","doi-asserted-by":"publisher","first-page":"407","DOI":"10.1007\/s00165-021-00548-1","article-title":"Enhancing robustness verification for deep neural networks via symbolic propagation","volume":"33","author":"Yang","year":"2021","journal-title":"Form. Asp. Comput."},{"issue":"2","key":"10.3233\/AIC-220128_ref97","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1109\/6.981854","article-title":"Modular robots","volume":"39","author":"Yim","year":"2002","journal-title":"IEEE Spectrum"},{"key":"10.3233\/AIC-220128_ref98","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-54549-9_16"},{"key":"10.3233\/AIC-220128_ref99","unstructured":"X.\u00a0Zhao, W.\u00a0Huang, A.\u00a0Banks, V.\u00a0Cox, D.\u00a0Flynn, S.\u00a0Schewe and X.\u00a0Huang, Assessing the reliability of deep learning classifiers through robustness evaluation and operational profiles, in: AISafety\u201921 Workshop at IJCAI\u201921, Vol.\u00a02916, 2021."},{"key":"10.3233\/AIC-220128_ref100","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2112.00646"},{"key":"10.3233\/AIC-220128_ref101","unstructured":"X.\u00a0Zhao, W.\u00a0Huang, X.\u00a0Huang, V.\u00a0Robu and D.\u00a0Flynn, BayLIME: Bayesian local interpretable model-agnostic explanations, in: Proc. of the 37th Conference on Uncertainty in Artificial Intelligence, C.\u00a0de\u00a0Campos and M.H.\u00a0Maathuis, eds, UAI\u201921, Vol.\u00a0161, PMLR, 2021, pp.\u00a0887\u2013896."},{"key":"10.3233\/AIC-220128_ref102","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2020.106393"},{"key":"10.3233\/AIC-220128_ref103","unstructured":"M.\u00a0Zhou, Z.\u00a0Liu, P.\u00a0Sui, Y.\u00a0Li and Y.Y.\u00a0Chung, Learning implicit credit assignment for cooperative multi-agent reinforcement learning, in: Advances in Neural Information Processing Systems, 2020, pp.\u00a011853\u201311864."}],"container-title":["AI Communications"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/AIC-220128","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T18:28:05Z","timestamp":1777400885000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/AIC-220128"}},"subtitle":[],"editor":[{"given":"Stefano V.","family":"Albrecht","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]},{"given":"Michael","family":"Woolridge","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2022,9,20]]},"references-count":95,"journal-issue":{"issue":"4"},"URL":"https:\/\/doi.org\/10.3233\/aic-220128","relation":{},"ISSN":["1875-8452","0921-7126"],"issn-type":[{"value":"1875-8452","type":"electronic"},{"value":"0921-7126","type":"print"}],"subject":[],"published":{"date-parts":[[2022,9,20]]}}}