{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,27]],"date-time":"2025-10-27T11:02:47Z","timestamp":1761562967684,"version":"build-2065373602"},"reference-count":33,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2023,4,28]],"date-time":"2023-04-28T00:00:00Z","timestamp":1682640000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Funda\u00e7\u00e3o para a Ci\u00eancia e Tecnologia (FCT)","award":["UIDB\/04466\/2020","UIDP\/04466\/2020","UIDB\/50008\/2020"],"award-info":[{"award-number":["UIDB\/04466\/2020","UIDP\/04466\/2020","UIDB\/50008\/2020"]}]},{"name":"FCT","award":["UIDB\/04466\/2020","UIDP\/04466\/2020","UIDB\/50008\/2020"],"award-info":[{"award-number":["UIDB\/04466\/2020","UIDP\/04466\/2020","UIDB\/50008\/2020"]}]},{"name":"European Union (EU)","award":["UIDB\/04466\/2020","UIDP\/04466\/2020","UIDB\/50008\/2020"],"award-info":[{"award-number":["UIDB\/04466\/2020","UIDP\/04466\/2020","UIDB\/50008\/2020"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>To handle the exponential growth of data-intensive network edge services and automatically solve new challenges in routing management, machine learning is steadily being incorporated into software-defined networking solutions. In this line, the article presents the design of a piecewise-stationary Bayesian multi-armed bandit approach for the online optimum end-to-end dynamic routing of data flows in the context of programmable networking systems. This learning-based approach has been analyzed with simulated and emulated data, showing the proposal\u2019s ability to sequentially and proactively self-discover the end-to-end routing path with minimal delay among a considerable number of alternatives, even when facing abrupt changes in transmission delay distributions due to both variable congestion levels on path network devices and dynamic delays to transmission links.<\/jats:p>","DOI":"10.3390\/a16050233","type":"journal-article","created":{"date-parts":[[2023,4,28]],"date-time":"2023-04-28T09:54:53Z","timestamp":1682675693000},"page":"233","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["A Bayesian Multi-Armed Bandit Algorithm for Dynamic End-to-End Routing in SDN-Based Networks with Piecewise-Stationary Rewards"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4357-1546","authenticated-orcid":false,"given":"Pedro","family":"Santana","sequence":"first","affiliation":[{"name":"ISCTE\u2014University Institute of Lisbon (ISCTE-IUL), Av. das For\u00e7as Armadas, 1649-026 Lisboa, Portugal"},{"name":"ISTAR\u2014Information Sciences and Technologies and Architecture Research Center, Av. das For\u00e7as Armadas, 1649-026 Lisboa, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3516-8781","authenticated-orcid":false,"given":"Jos\u00e9","family":"Moura","sequence":"additional","affiliation":[{"name":"ISCTE\u2014University Institute of Lisbon (ISCTE-IUL), Av. das For\u00e7as Armadas, 1649-026 Lisboa, Portugal"},{"name":"Instituto de Telecomunica\u00e7\u00f5es (IT), Av. das For\u00e7as Armadas, 1649-026 Lisboa, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2023,4,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s42452-020-03335-4","article-title":"Modeling cooperative behavior for resilience in cyber-physical systems using SDN and NFV","volume":"2","author":"Moura","year":"2020","journal-title":"SN Appl. Sci."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"104582","DOI":"10.1109\/ACCESS.2021.3099092","article-title":"A Survey on Machine Learning Techniques for Routing Optimization in SDN","volume":"9","author":"Amin","year":"2021","journal-title":"IEEE Access"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"16","DOI":"10.1186\/s13174-018-0087-2","article-title":"A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities","volume":"9","author":"Boutaba","year":"2018","journal-title":"J. Internet Serv. Appl."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3039","DOI":"10.1109\/COMST.2019.2926625","article-title":"Artificial Neural Networks-Based Machine Learning for Wireless Networks: A Tutorial","volume":"21","author":"Chen","year":"2019","journal-title":"IEEE Commun. Surv. Tutorials"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1251","DOI":"10.1109\/COMST.2020.2964534","article-title":"Machine Learning for Resource Management in Cellular and IoT Networks: Potentials, Current Solutions, and Open Challenges","volume":"22","author":"Hussain","year":"2020","journal-title":"IEEE Commun. Surv. Tutorials"},{"key":"ref_6","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/2200000068","article-title":"Introduction to multi-armed bandits","volume":"12","author":"Slivkins","year":"2019","journal-title":"Found. Trends\u00ae Mach. Learn."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Lattimore, T., and Szepesv\u00e1ri, C. (2020). Bandit Algorithms, Cambridge University Press.","DOI":"10.1017\/9781108571401"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Bouneffouf, D., Rish, I., and Aggarwal, C. (2020, January 19\u201324). Survey on applications of multi-armed and contextual bandits. Proceedings of the 2020 IEEE Congress on Evolutionary Computation (CEC), Glasgow, UK.","DOI":"10.1109\/CEC48606.2020.9185782"},{"key":"ref_10","unstructured":"Mariano, P., Almeida, S.M., and Santana, P. (2021). Energy Sources, Part A: Recovery, Utilization, and Environmental Effects, Taylor and Francis."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1578","DOI":"10.1109\/COMST.2021.3073009","article-title":"Survey on machine learning for intelligent end-to-end communication toward 6G: From network access, routing to traffic control and streaming adaption","volume":"23","author":"Tang","year":"2021","journal-title":"IEEE Commun. Surv. Tutorials"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Tariq, Z.U.A., Baccour, E., Erbad, A., Guizani, M., and Hamdi, M. (2022, January 18\u201320). Network Intrusion Detection for Smart Infrastructure using Multi-armed Bandit based Reinforcement Learning in Adversarial Environment. Proceedings of the 2022 International Conference on Cyber Warfare and Security (ICCWS), Warsaw, Poland.","DOI":"10.1109\/ICCWS56285.2022.9998440"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1720","DOI":"10.1109\/TIFS.2020.3042049","article-title":"Self-configurable cyber-physical intrusion detection for smart homes using reinforcement learning","volume":"16","author":"Heartfield","year":"2020","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1879","DOI":"10.1109\/TNET.2018.2856302","article-title":"Multi-armed bandit in action: Optimizing performance in dynamic hybrid networks","volume":"26","author":"Henri","year":"2018","journal-title":"IEEE\/ACM Trans. Netw."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1109\/TNSM.2020.3044674","article-title":"Joint Switch\u2013Controller Association and Control Devolution for SDN Systems: An Integrated Online Perspective of Control and Learning","volume":"18","author":"Huang","year":"2020","journal-title":"IEEE Trans. Netw. Serv. Manag."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"174773","DOI":"10.1109\/ACCESS.2020.3025432","article-title":"QR-SDN: Towards reinforcement learning states, actions, and rewards for direct flow routing in software-defined networks","volume":"8","author":"Rischke","year":"2020","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"4807","DOI":"10.1109\/TNSM.2021.3132491","article-title":"DRSIR: A Deep Reinforcement Learning Approach for Routing in Software-Defined Networking","volume":"19","author":"Rendon","year":"2022","journal-title":"IEEE Trans. Netw. Serv. Manag."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1023\/A:1013689704352","article-title":"Finite-time analysis of the multiarmed bandit problem","volume":"47","author":"Auer","year":"2002","journal-title":"Mach. Learn."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1137\/S0097539701398375","article-title":"The nonstochastic multiarmed bandit problem","volume":"32","author":"Auer","year":"2002","journal-title":"SIAM J. Comput."},{"key":"ref_20","unstructured":"Garivier, A., and Capp\u00e9, O. (2011, January 9\u201311). The KL-UCB algorithm for bounded stochastic bandits and beyond. Proceedings of the 24th Annual Conference on Learning Theory, Budapest, Hungary."},{"key":"ref_21","unstructured":"Kocsis, L., and Szepesv\u00e1ri, C. (2006, January 10\u201312). Discounted ucb. Proceedings of the 2nd PASCAL Challenges Workshop, Graz, Austria."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Garivier, A., and Moulines, E. (2011, January 5\u20137). On upper-confidence bound policies for switching bandit problems. Proceedings of the 22nd International Conference on Algorithmic Learning Theory (ALT), Espoo, Finland. Proceedings 22.","DOI":"10.1007\/978-3-642-24412-4_16"},{"key":"ref_23","unstructured":"Cao, Y., Wen, Z., Kveton, B., and Xie, Y. (2019, January 16\u201318). Nearly optimal adaptive procedure with change detection for piecewise-stationary bandit. Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, PMLR, Okinawa, Japan."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"6711","DOI":"10.1109\/TIT.2021.3081508","article-title":"Multi-armed bandits with correlated arms","volume":"67","author":"Gupta","year":"2021","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_25","unstructured":"(2023, February 25). Network Analysis. Available online: https:\/\/networkx.org\/."},{"key":"ref_26","unstructured":"Cohen, B. (1988). Statistical Power Analysis for the Behavioral Sciences, Lawrence Erlbaum Associates. [2nd ed.]."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1093\/biomet\/25.3-4.285","article-title":"On the likelihood that one unknown probability exceeds another in view of the evidence of two samples","volume":"25","author":"Thompson","year":"1933","journal-title":"Biometrika"},{"key":"ref_28","unstructured":"(2023, February 25). SMPyBandits. Available online: https:\/\/pypi.org\/project\/SMPyBandits\/."},{"key":"ref_29","unstructured":"(2023, February 25). Mininet. Available online: https:\/\/github.com\/mininet\/mininet."},{"key":"ref_30","unstructured":"(2023, February 25). Ryu. Available online: https:\/\/github.com\/faucetsdn\/ryu."},{"key":"ref_31","first-page":"2312","article-title":"Improved algorithms for linear stochastic bandits","volume":"24","year":"2011","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Cardoso, P., Moura, J., and Marinheiro, R.N. (2023). Elastic Provisioning of Network and Computing Resources at the Edge for IoT Services. Sensors, 23.","DOI":"10.3390\/s23052762"},{"key":"ref_33","unstructured":"Cardoso, P., Moura, J., and Marinheiro, R. (2023, April 02). Software-Defined Elastic Provisioning of IoT Edge Computing Virtual Resources. CoRR 2023. Available online: https:\/\/arxiv.org\/abs\/2003.11999."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/5\/233\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:25:44Z","timestamp":1760124344000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/5\/233"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,28]]},"references-count":33,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2023,5]]}},"alternative-id":["a16050233"],"URL":"https:\/\/doi.org\/10.3390\/a16050233","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2023,4,28]]}}}