{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T16:09:27Z","timestamp":1781194167103,"version":"3.54.1"},"reference-count":38,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T00:00:00Z","timestamp":1721865600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"2023 Information Society Innovation Fund (ISIF Asia)"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Bufferbloat is one of the leading causes of high data transmission latency and jitter on the Internet, which severely impacts the performance of low-latency interactive applications such as online streaming, cloud-based gaming\/applications, Internet of Things (IoT) applications, voice over IP (VoIP), real-time video conferencing, and so forth. There is currently a pressing need for developing Transmission Control Protocol (TCP) congestion control algorithms and bottleneck queue management schemes that can collaboratively control\/reduce end-to-end latency, thus ensuring optimal quality of service (QoS) and quality of experience (QoE) for users. This paper introduces a novel solution by experimentally integrate the low latency, low loss, and scalable throughput (L4S) architecture (specified by the IETF in RFC 9330) in FreeBSD framework with the asynchronous advantage actor-critic (A3C) reinforcement learning algorithm. The first phase involves incorporating a modified dual-queue coupled active queue management (AQM) system for L4S into the FreeBSD networking stack, enhancing queue management and mitigating latency and packet loss. The second phase employs A3C to adjust and fine-tune the system performance dynamically. Finally, we evaluate the proposed solution\u2019s effectiveness through comprehensive experiments, comparing it with traditional AQM-based systems. This paper contributes to the advancement of machine learning (ML) for transport protocol research in the field. The experimental implementation and results presented in this paper are made available through our GitHub repositories.<\/jats:p>","DOI":"10.3390\/fi16080265","type":"journal-article","created":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T14:31:07Z","timestamp":1721917867000},"page":"265","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Active Queue Management in L4S with Asynchronous Advantage Actor-Critic: A FreeBSD Networking Stack Perspective"],"prefix":"10.3390","volume":"16","author":[{"given":"Deol","family":"Satish","sequence":"first","affiliation":[{"name":"IoT & Software Engineering Research Lab, School of Information Technology, Deakin University, Geelong, VIC 3220, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9699-9418","authenticated-orcid":false,"given":"Jonathan","family":"Kua","sequence":"additional","affiliation":[{"name":"IoT & Software Engineering Research Lab, School of Information Technology, Deakin University, Geelong, VIC 3220, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5819-765X","authenticated-orcid":false,"given":"Shiva Raj","family":"Pokhrel","sequence":"additional","affiliation":[{"name":"IoT & Software Engineering Research Lab, School of Information Technology, Deakin University, Geelong, VIC 3220, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,7,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1145\/2063166.2071893","article-title":"Bufferbloat: Dark Buffers in the Internet: Networks without effective AQM may again be vulnerable to congestion collapse","volume":"9","author":"Gettys","year":"2011","journal-title":"Queue"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1109\/90.251892","article-title":"Random early detection gateways for congestion avoidance","volume":"1","author":"Floyd","year":"1993","journal-title":"IEEE\/ACM Trans. Netw."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1842","DOI":"10.1109\/COMST.2017.2685630","article-title":"A survey of rate adaptation techniques for dynamic adaptive streaming over HTTP","volume":"19","author":"Kua","year":"2017","journal-title":"IEEE Commun. Surv. Tutorials"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3344381","article-title":"Adaptive Chunklets and AQM for higher-performance content streaming","volume":"15","author":"Kua","year":"2019","journal-title":"Acm Trans. Multimed. Comput. Commun. Appl. (TOMM)"},{"key":"ref_5","unstructured":"Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys, J., and Dumazet, E. (2024, July 18). The Flow Queue Codel Packet Scheduler and Active Queue Management Algorithm. Technical Report. Available online: https:\/\/www.rfc-editor.org\/rfc\/rfc8290.html."},{"key":"ref_6","unstructured":"Pan, R., Natarajan, P., Baker, F., and White, G. (2024, July 18). Proportional Integral Controller Enhanced (PIE): A Lightweight Control Scheme to Address the Bufferbloat Problem. RFC 8033. Available online: https:\/\/www.rfc-editor.org\/info\/rfc8033."},{"key":"ref_7","unstructured":"White, G., and Pan, R. (2024, July 18). Active Queue Management (AQM) Based on Proportional Integral Controller Enhanced (PIE) for Data-Over-Cable Service Interface Specifications (DOCSIS) Cable Modems. RFC 8034. Available online: https:\/\/www.rfc-editor.org\/info\/rfc8034."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Cardozo, T.B., da Silva, A.P.C., Vieira, A.B., and Ziviani, A. (2014, January 17\u201320). Bufferbloat systematic analysis. Proceedings of the 2014 International Telecommunications Symposium (ITS), Sao Paulo, Brazil.","DOI":"10.1109\/ITS.2014.6947975"},{"key":"ref_9","unstructured":"Ahammed, G., and Banu, R. (2010). Anakyzing the performance of active queue management algorithms. arXiv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1399","DOI":"10.1109\/JIOT.2017.2722683","article-title":"Using active queue management to assist IoT application flows in home broadband networks","volume":"4","author":"Kua","year":"2017","journal-title":"IEEE Internet Things J."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Kua, J., Branch, P., and Armitage, G. (2020, January 16\u201319). Detecting bottleneck use of pie or fq-codel active queue management during dash-like content streaming. Proceedings of the 2020 IEEE 45th Conference on Local Computer Networks (LCN), Sydney, Australia.","DOI":"10.1109\/LCN48667.2020.9314804"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Amol, D., and Rajesh, P. (2014). A review on active queue management techniques of congestion control. Proceedings of the 2014 International Conference on Electronic Systems, Signal Processing and Computing Technologies, IEEE.","DOI":"10.1109\/ICESC.2014.34"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1145\/2209249.2209264","article-title":"Controlling queue delay","volume":"55","author":"Nichols","year":"2012","journal-title":"Commun. ACM"},{"key":"ref_14","unstructured":"Hoeiland-Joergensen, T., McKenney, P., Taht, D., Ghettys, J., and Dumazet, E. (2024, July 18). Flowqueue-Codel: Draft-Hoeiland-Joergensen-Aqm-fq-Codel-00. Available online: https:\/\/datatracker.ietf.org\/doc\/draft-ietf-aqm-fq-codel\/00\/."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Ramakrishnan, G., Bhasi, M., Saicharan, V., Monis, L., Patil, S.D., and Tahiliani, M.P. (2019, January 14\u201317). FQ-PIE queue discipline in the Linux kernel: Design, implementation and challenges. Proceedings of the 2019 IEEE 44th LCN Symposium on Emerging Topics in Networking (LCN Symposium), Osnabr\u00fcck, Germany.","DOI":"10.1109\/LCNSymposium47956.2019.9000684"},{"key":"ref_16","unstructured":"Al-Saadi, R., and Armitage, G. (2016). Dummynet AQM v0. 2\u2013CoDel, FQ-CoDel, PIE and FQ-PIE for FreeBSD\u2019s ipfw\/Dummynet Framework, Centre for Advanced Internet Architectures, Swinburne University of Technology. Tech. Rep. A 160418."},{"key":"ref_17","unstructured":"Ramakrishnan, K., Floyd, S., and Black, D. (2024, July 18). The Addition of Explicit Congestion Notification (ECN) to IP. Technical Report. Available online: https:\/\/www.rfc-editor.org\/rfc\/rfc3168.html."},{"key":"ref_18","unstructured":"De Schepper, K., Albisser, O., Tilmans, O., and Briscoe, B. (2022). Dual Queue Coupled AQM: Deployable Very Low Queuing Delay for All. arXiv."},{"key":"ref_19","unstructured":"Schepper, K.D., Briscoe, B., and White, G. (2024, July 18). Dual-Queue Coupled Active Queue Management (AQM) for Low Latency, Low Loss, and Scalable Throughput (L4S). RFC 9332. Available online: https:\/\/www.rfc-editor.org\/info\/rfc9332."},{"key":"ref_20","first-page":"1726","article-title":"On designing improved controllers for AQM routers supporting TCP flows","volume":"Volume 3","author":"Hollot","year":"2001","journal-title":"Proceedings of the IEEE INFOCOM 2001 Conference on Computer Communications, Twentieth Annual Joint Conference of the IEEE 61 Computer and Communications Society (Cat. No.01CH37213)"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Szygu\u0142a, J., Doma\u0144ski, A., Doma\u0144ska, J., Marek, D., Filus, K., and Mendla, S. (2021). Supervised Learning of Neural Networks for Active Queue Management in the Internet. Sensors, 21.","DOI":"10.3390\/s21154979"},{"key":"ref_22","first-page":"1169","article-title":"QRED: A Q-learning-based active queue management scheme","volume":"19","author":"Su","year":"2018","journal-title":"J. Internet Technol."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Liu, J., and Wei, D. (2022, January 18\u201321). Active Queue Management Based on Q-Learning Traffic Predictor. Proceedings of the 2022 International Conference on Cyber-Physical Social Intelligence (ICCSI), Nanjing, China.","DOI":"10.1109\/ICCSI55536.2022.9970698"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Gomez, C.A., Wang, X., and Shami, A. (2019, January 9\u201313). Intelligent active queue management using explicit congestion notification. Proceedings of the 2019 IEEE Global Communications Conference (GLOBECOM), Waikoloa, HI, USA.","DOI":"10.1109\/GLOBECOM38437.2019.9013475"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"108515","DOI":"10.1016\/j.comnet.2021.108515","article-title":"An intelligent scheme for congestion control: When active queue management meets deep reinforcement learning","volume":"200","author":"Ma","year":"2021","journal-title":"Comput. Netw."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1007\/s10922-021-09603-x","article-title":"Deep reinforcement learning based active queue management for iot networks","volume":"29","author":"Kim","year":"2021","journal-title":"J. Netw. Syst. Manag."},{"key":"ref_27","unstructured":"Fawaz, H., Zeghlache, D., Pham, Q.T.A., J\u00e9r\u00e9mie, L., and Medagliani, P. (2021, January 13\u201316). Deep Reinforcement Learning for Smart Queue Management. Proceedings of the NETSYS 2021: Conference on Networked Systems 2021, L\u00fcbeck, Germany. Available online: https:\/\/hal.archives-ouvertes.fr\/hal-03546621."},{"key":"ref_28","unstructured":"Albisser, O., De Schepper, K., Briscoe, B., Tilmans, O., and Steen, H. (2019, January 20\u201322). DUALPI2\u2014Low Latency, Low Loss and Scalable Throughput (L4S) AQM. Proceedings of the Linux Netdev 0x13, Prague, Czech Republic. Available online: https:\/\/www.netdevconf.org\/0x13\/session.html?talk-DUALPI2-AQM."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Briscoe, B., Schepper, K.D., Bagnulo, M., and White, G. (2024, July 18). Low Latency, Low Loss, and Scalable Throughput (L4S) Internet Service: Architecture. RFC 9330. Available online: https:\/\/www.rfc-editor.org\/info\/rfc9330.","DOI":"10.17487\/RFC9330"},{"key":"ref_30","unstructured":"De Schepper, K., Bondarenko, O., Tsang, I.J., and Briscoe, B. (2016, January 12\u201315). Pi2: A linearized aqm for both classic and scalable tcp. Proceedings of the 12th International on Conference on emerging Networking EXperiments and Technologies, Irvine, CA, USA."},{"key":"ref_31","unstructured":"Briscoe, B. (2021). PI2 Parameters. arXiv."},{"key":"ref_32","unstructured":"Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016, January 19\u201324). Asynchronous methods for deep reinforcement learning. Proceedings of the International Conference on Machine Learning. PMLR, New York, NY, USA."},{"key":"ref_33","first-page":"A3C","article-title":"Vision Enhanced Asynchronous Advantage Actor-Critic on Racing Games","volume":"4","author":"Palamuttam","year":"2017","journal-title":"Methods"},{"key":"ref_34","unstructured":"Stewart, L., and Healy, J. (2024, July 18). Characterising the Behaviour and Performance of SIFTR v1. 1.0. Technical Report, CAIA, Tech. Rep. Available online: http:\/\/caia.swinburne.edu.au\/reports\/070824A\/CAIA-TR-070824A.pdf."},{"key":"ref_35","unstructured":"The Tcpdump Group (2024, July 12). Tcpdump. Available online: https:\/\/www.tcpdump.org\/."},{"key":"ref_36","unstructured":"Dpkt Contributors (2024, July 12). Dpkt. Available online: https:\/\/pypi.org\/project\/dpkt\/."},{"key":"ref_37","unstructured":"Wireshark Foundation (2024, July 12). Wireshark. Available online: https:\/\/www.wireshark.org\/."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Pokhrel, S.R., Kua, J., Satish, D., Ozer, S., Howe, J., and Walid, A. (2024). DDPG-MPCC: An Experience Driven Multipath Performance Oriented Congestion Control. Future Internet, 16.","DOI":"10.3390\/fi16020037"}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/16\/8\/265\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:23:28Z","timestamp":1760109808000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/16\/8\/265"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,25]]},"references-count":38,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2024,8]]}},"alternative-id":["fi16080265"],"URL":"https:\/\/doi.org\/10.3390\/fi16080265","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,25]]}}}