{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T14:54:32Z","timestamp":1788879272307,"version":"build-2803163510"},"reference-count":24,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2022,1,17]],"date-time":"2022-01-17T00:00:00Z","timestamp":1642377600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGMETRICS Perform. Eval. Rev."],"published-print":{"date-parts":[[2022,1,17]]},"abstract":"<jats:p>We approach the task of network congestion control in datacenters using Reinforcement Learning (RL). Successful congestion control algorithms can dramatically improve latency and overall network throughput. Until today, no such learning-based algorithms have shown practical potential in this domain. Evidently, the most popular recent deployments rely on rule-based heuristics that are tested on a predetermined set of benchmarks. Consequently, these heuristics do not generalize well to newly-seen scenarios. Contrarily, we devise an RL-based algorithm with the aim of generalizing to different configurations of real-world datacenter networks. We overcome challenges such as partial-observability, nonstationarity, and multi-objectiveness. We further propose a policy gradient algorithm that leverages the analytical structure of the reward function to approximate its derivative and improve stability. We show that this scheme outperforms alternative popular RL approaches, and generalizes to scenarios that were not seen during training. Our experiments, conducted on a realistic simulator that emulates communication networks' behavior, exhibit improved performance concurrently on the multiple considered metrics compared to the popular algorithms deployed today in real datacenters. Our algorithm is being productized to replace heuristics in some of the largest datacenters in the world.<\/jats:p>","DOI":"10.1145\/3512798.3512815","type":"journal-article","created":{"date-parts":[[2022,1,20]],"date-time":"2022-01-20T13:13:23Z","timestamp":1642684403000},"page":"43-46","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":33,"title":["Reinforcement Learning for Datacenter Congestion Control"],"prefix":"10.1145","volume":"49","author":[{"given":"Chen","family":"Tessler","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuval","family":"Shpigelman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gal","family":"Dalal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amit","family":"Mandelbaum","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Doron","family":"Haritan Kazakov","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Benjamin","family":"Fuhrer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gal","family":"Chechik","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shie","family":"Mannor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,1,20]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1851182.1851192"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1030194.1015499"},{"key":"e_1_2_1_3_1","volume-title":"Openai gym. arXiv preprint arXiv:1606.01540","author":"Brockman G.","year":"2016","unstructured":"Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint arXiv:1606.01540, 2016."},{"key":"e_1_2_1_4_1","first-page":"343","volume-title":"15th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 18)","author":"Dong M.","year":"2018","unstructured":"Dong, M., Meng, T., Zarchy, D., Arslan, E., Gilad, Y., Godfrey, B., and Schapira, M. {PCC} vivace: Online-learning congestion control. In 15th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 18), pp. 343--356, 2018."},{"key":"e_1_2_1_5_1","first-page":"3050","volume-title":"International Conference on Machine Learning","author":"Jay N.","year":"2019","unstructured":"Jay, N., Rotman, N., Godfrey, B., Schapira, M., and Tamar, A. A deep reinforcement learning perspective on internet congestion control. In International Conference on Machine Learning, pp. 3050--3059, 2019."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3387514.3406591"},{"key":"e_1_2_1_7_1","volume-title":"Ai safety gridworlds. arXiv preprint arXiv:1711.09883","author":"Leike J.","year":"2017","unstructured":"Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt, T., Lefrancq, A., Orseau, L., and Legg, S. Ai safety gridworlds. arXiv preprint arXiv:1711.09883, 2017."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341302.3342085"},{"key":"e_1_2_1_9_1","volume-title":"Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971","author":"Lillicrap T. P.","year":"2015","unstructured":"Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015."},{"issue":"3","key":"e_1_2_1_10_1","first-page":"385","article-title":"Multiobjective reinforcement learning: A comprehensive overview","volume":"45","author":"Liu C.","year":"2014","unstructured":"Liu, C., Xu, X., and Hu, D. Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(3):385-- 398, 2014.","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics: Systems"},{"key":"e_1_2_1_11_1","volume-title":"Simple random search provides a competitive approach to reinforcement learning. arXiv preprint arXiv:1803.07055","author":"Mania H.","year":"2018","unstructured":"Mania, H., Guy, A., and Recht, B. Simple random search provides a competitive approach to reinforcement learning. arXiv preprint arXiv:1803.07055, 2018."},{"key":"e_1_2_1_12_1","volume-title":"A geometric approach to multicriterion reinforcement learning. Journal of machine learning research, 5(Apr):325--360","author":"Mannor S.","year":"2004","unstructured":"Mannor, S. and Shimkin, N. A geometric approach to multicriterion reinforcement learning. Journal of machine learning research, 5(Apr):325--360, 2004."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2829988.2787510"},{"key":"e_1_2_1_14_1","volume-title":"Human-level control through deep reinforcement learning. nature, 518(7540):529--533","author":"Mnih V.","year":"2015","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540):529--533, 2015."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1002\/9780470316887"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.17487\/RFC3168"},{"key":"e_1_2_1_17_1","volume-title":"Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347","author":"Schulman J.","year":"2017","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017."},{"key":"e_1_2_1_18_1","volume-title":"ICML","author":"Silver D.","year":"2014","unstructured":"Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In ICML, 2014."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-27645-3_12"},{"key":"e_1_2_1_20_1","volume-title":"International Conference on Learning Representations","author":"Tessler C.","year":"2018","unstructured":"Tessler, C., Mankowitz, D. J., and Mannor, S. Reward constrained policy optimization. In International Conference on Learning Representations, 2018."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2012.6386109"},{"key":"e_1_2_1_22_1","volume-title":"omnetpp. org","author":"Varga A.","year":"2002","unstructured":"Varga, A. Omnet++ http:\/\/www. omnetpp. org. IEEE Network Interactive, 16(4), 2002."},{"key":"e_1_2_1_23_1","volume-title":"Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.09602","author":"Wu H.","year":"2020","unstructured":"Wu, H., Judd, P., Zhang, X., Isaev, M., and Micikevicius, P. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.09602, 2020."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2829988.2787484"}],"container-title":["ACM SIGMETRICS Performance Evaluation Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3512798.3512815","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3512798.3512815","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T22:26:43Z","timestamp":1752532003000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3512798.3512815"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,17]]},"references-count":24,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,1,17]]}},"alternative-id":["10.1145\/3512798.3512815"],"URL":"https:\/\/doi.org\/10.1145\/3512798.3512815","relation":{},"ISSN":["0163-5999"],"issn-type":[{"value":"0163-5999","type":"print"}],"subject":[],"published":{"date-parts":[[2022,1,17]]},"assertion":[{"value":"2022-01-20","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}