{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T16:25:13Z","timestamp":1778603113691,"version":"3.51.4"},"reference-count":26,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T00:00:00Z","timestamp":1714348800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Neurosci."],"abstract":"<jats:p>In recent years, with the rapid development of network applications and the increasing demand for high-quality network service, quality-of-service (QoS) routing has emerged as a critical network technology. The application of machine learning techniques, particularly reinforcement learning and graph neural network, has garnered significant attention in addressing this problem. However, existing reinforcement learning methods lack research on the causal impact of agent actions on the interactive environment, and graph neural network fail to effectively represent link features, which are pivotal for routing optimization. Therefore, this study quantifies the causal influence between the intelligent agent and the interactive environment based on causal inference techniques, aiming to guide the intelligent agent in improving the efficiency of exploring the action space. Simultaneously, graph neural network is employed to embed node and link features, and a reward function is designed that comprehensively considers network performance metrics and causality relevance. A centralized reinforcement learning method is proposed to effectively achieve QoS-aware routing in Software-Defined Networking (SDN). Finally, experiments are conducted in a network simulation environment, and metrics such as packet loss, delay, and throughput all outperform the baseline.<\/jats:p>","DOI":"10.3389\/fncom.2024.1393025","type":"journal-article","created":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T04:34:20Z","timestamp":1714365260000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Reinforcement learning-based SDN routing scheme empowered by causality detection and GNN"],"prefix":"10.3389","volume":"18","author":[{"given":"Yuanhao","family":"He","sequence":"first","affiliation":[]},{"given":"Geyang","family":"Xiao","sequence":"additional","affiliation":[]},{"given":"Jun","family":"Zhu","sequence":"additional","affiliation":[]},{"given":"Tao","family":"Zou","sequence":"additional","affiliation":[]},{"given":"Yuan","family":"Liang","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2024,4,29]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/ICNP52444.2021.9651930","article-title":"\u201cIs machine learning ready for traffic engineering optimization?\u201d","volume-title":"2021 IEEE 29th International Conference on Network Protocols (ICNP)","author":"Bernardez","year":"2021"},{"key":"B2","doi-asserted-by":"publisher","first-page":"870","DOI":"10.1109\/TNSM.2020.3036911","article-title":"Intelligent routing based on reinforcement learning for software-defined networking","volume":"18","author":"Casas-Velasco","year":"2021","journal-title":"IEEE Trans. Netw. Serv. Manag"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.01452","article-title":"Causal reinforcement learning: a survey","author":"Deng","year":"2023","journal-title":"arXiv"},{"key":"B4","doi-asserted-by":"publisher","first-page":"1673","DOI":"10.1109\/TNSM.2021.3077249","article-title":"Generative adversarial network-based transfer reinforcement learning for routing with prior knowledge","volume":"18","author":"Dong","year":"2021","journal-title":"IEEE Trans. Netw. Serv. Manag"},{"key":"B5","doi-asserted-by":"publisher","first-page":"3080","DOI":"10.1109\/TNET.2023.3269983","article-title":"Routenet-fermi: network modeling with graph neural networks","volume":"31","author":"Ferriol-Galm's","year":"2023","journal-title":"IEEE\/ACM Trans. Netw"},{"key":"B6","article-title":"\u201cSoft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor,\u201d","volume-title":"Proceedings of the 35th International Conference on Machine Learning, ICML","author":"Haarnoja","year":"2018"},{"key":"B7","doi-asserted-by":"publisher","first-page":"1731","DOI":"10.1109\/TMC.2021.3105963","article-title":"Rthop: real-time hop-by-hop mobile network routing by decentralized learning with semantic attention","volume":"22","author":"He","year":"2023","journal-title":"IEEE Trans. Mob. Comp"},{"key":"B8","article-title":"\u201cA graph reinforcement learning routing scheme based on nodes and edges co-embedding,\u201d","volume-title":"ICC 2024","author":"He","year":"2024"},{"key":"B9","first-page":"9260","article-title":"\u201cAction-sufficient state representation learning for control with structural constraints,\u201d","volume-title":"International Conference on Machine Learning","author":"Huang","year":"2022"},{"key":"B10","first-page":"2656","article-title":"\u201cCensnet: convolution with edge-node switching in graph neural networks,\u201d","volume-title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19","author":"Jiang","year":"2019"},{"key":"B11","doi-asserted-by":"publisher","first-page":"7075","DOI":"10.1109\/TPAMI.2020.3029762","article-title":"Co-embedding of nodes and edges with graph neural networks","volume":"45","author":"Jiang","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"B12","doi-asserted-by":"publisher","first-page":"2337","DOI":"10.1109\/TPDS.2023.3284651","article-title":"Scalable deep reinforcement learning-based online routing for multi-type service requirements","volume":"34","author":"Liu","year":"2023","journal-title":"IEEE Trans. Parallel Distrib. Syst"},{"key":"B13","first-page":"1","article-title":"\u201cDRL-or: deep reinforcement learning-based online routing for multi-type service requirements,\u201d","volume-title":"IEEE INFOCOM 2021","author":"Liu","year":"2021"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv:1812.10576","article-title":"Deconfounding reinforcement learning in observational settings","author":"Lu","year":"2018","journal-title":"arXiv"},{"key":"B15","article-title":"\u201cTowards safe online reinforcement learning in computer systems,\u201d","volume-title":"NeurIPS Machine Learning for Systems Workshop","author":"Mao","year":"2019"},{"key":"B16","first-page":"3976","article-title":"\u201cCounterfactual data augmentation using locally factored dynamics,\u201d","volume-title":"Advances in Neural Information Processing Systems 33","author":"Pitis","year":"2020"},{"key":"B17","doi-asserted-by":"publisher","first-page":"2260","DOI":"10.1109\/JSAC.2020.3000405","article-title":"Routenet: leveraging graph neural networks for network modeling and optimization in SDN","volume":"38","author":"Rusek","year":"2019","journal-title":"IEEE J. Sel. Areas Commun"},{"key":"B18","first-page":"22905","article-title":"\u201cCausal influence detection for improving efficiency in reinforcement learning,\u201d","volume-title":"Advances in Neural Information Processing Systems 34","author":"Seitzer","year":"2021"},{"key":"B19","first-page":"9848","article-title":"\u201cCausal curiosity: RL agents discovering self-supervised experiments for causal representation learning,\u201d","volume-title":"International conference on machine learning","author":"Sontakke","year":"2021"},{"key":"B20","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1109\/TFUZZ.2023.3302293","article-title":"Variable separation-based fuzzy optimal control for multiagent systems in nonstrict-feedback form","volume":"34","author":"Su","year":"2023","journal-title":"IEEE Trans Fuzzy Syst"},{"key":"B21","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1109\/TNET.2021.3126933","article-title":"Enabling scalable routing in software-defined networks with deep reinforcement learning on critical nodes","volume":"30","author":"Sun","year":"2022","journal-title":"IEEE\/ACM Trans. Netw"},{"key":"B22","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1109\/TNSE.2023.3295079","article-title":"Distributed optimal energy management for we-energy considering operation security","volume":"11","author":"Teng","year":"2023","journal-title":"IEEE Trans. Netw. Sci. Eng"},{"key":"B23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/IWQOS52092.2021.9521350","article-title":"\u201cRobustness-aware real-time SFC routing update in multi-tenant clouds,\u201d","volume-title":"2021 IEEE\/ACM 29th International Symposium on Quality of Service (IWQOS)","author":"Tu","year":"2021"},{"key":"B24","doi-asserted-by":"publisher","first-page":"1230","DOI":"10.1109\/TNET.2021.3137418","article-title":"A robustness-aware real-time SFC routing update scheme in multi-tenant clouds","volume":"30","author":"Tu","year":"2022","journal-title":"IEEE\/ACM Trans. Netw"},{"key":"B25","doi-asserted-by":"publisher","first-page":"1639","DOI":"10.1109\/TCYB.2022.3232599","article-title":"Convergence analysis of value iteration adaptive dynamic programming for continuous-time nonlinear systems","volume":"54","author":"Xiao","year":"2023","journal-title":"IEEE Trans. Cybern"},{"key":"B26","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1109\/COMST.2018.2866942","article-title":"A survey of machine learning techniques applied to software defined networking (SDN): research issues and challenges","volume":"21","author":"Xie","year":"2019","journal-title":"IEEE Commun. Surv. Tutor"}],"container-title":["Frontiers in Computational Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2024.1393025\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T04:34:27Z","timestamp":1714365267000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2024.1393025\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,29]]},"references-count":26,"alternative-id":["10.3389\/fncom.2024.1393025"],"URL":"https:\/\/doi.org\/10.3389\/fncom.2024.1393025","relation":{},"ISSN":["1662-5188"],"issn-type":[{"value":"1662-5188","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,29]]},"article-number":"1393025"}}