{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,2]],"date-time":"2026-01-02T07:11:07Z","timestamp":1767337867963,"version":"3.41.0"},"reference-count":25,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,11,16]],"date-time":"2024-11-16T00:00:00Z","timestamp":1731715200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Auton. Adapt. Syst."],"published-print":{"date-parts":[[2024,12,31]]},"abstract":"<jats:p>The microservices architecture simplifies application development by breaking monolithic applications into manageable microservices. However, this distributed microservice \u201cservice mesh\u201d leads to new challenges due to the more complex application topology. Particularly, each service component scales up and down independently creating load imbalance problems on shared backend services accessed by multiple components. Traditional load balancing algorithms do not port over well to a distributed microservice architecture where load balancers are deployed client-side. In this article, we propose a self-managing load balancing system, BLOC, which provides consistent response times to users without using a centralized metadata store or explicit messaging between nodes. BLOC uses overload control approaches to provide feedback to the load balancers. We show that this performs significantly better in solving the incast problem in microservice architectures. A critical component of BLOC is the dynamic capacity estimation algorithm. We show that a well-tuned capacity estimate can outperform even join-the-shortest-queue, a nearly optimal algorithm, while a reasonable dynamic estimate still outperforms Least Connection, a distributed implementation of join-the-shortest-queue. Evaluating this framework, we found that BLOC improves the response time distribution range, between the 10th and 90th percentiles, by 2 \u20134 times and the tail, 99th percentile, latency by 2 times.<\/jats:p>","DOI":"10.1145\/3676167","type":"journal-article","created":{"date-parts":[[2024,7,5]],"date-time":"2024-07-05T14:23:52Z","timestamp":1720189432000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Dynamically Balancing Load with Overload Control for Microservices"],"prefix":"10.1145","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-1992-3474","authenticated-orcid":false,"given":"Ratnadeep","family":"Bhattacharya","sequence":"first","affiliation":[{"name":"George Washington University, Washington, DC, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-6371-4672","authenticated-orcid":false,"given":"Yuan","family":"Gao","sequence":"additional","affiliation":[{"name":"George Washington University, Washington, DC, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6728-4197","authenticated-orcid":false,"given":"Timothy","family":"Wood","sequence":"additional","affiliation":[{"name":"George Washington University, Washington, DC, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,11,16]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"[n. d.]. Custom Loadtest: Open Loop Poisson Load Generator. Retrieved from https:\/\/github.com\/lyuxiaosu\/loadtest"},{"key":"e_1_3_2_3_2","unstructured":"[n. d.]. The Design and Operation of CloudLab: Flux Research Group. Retrieved from https:\/\/www.flux.utah.edu\/paper\/duplyakin-atc19"},{"key":"e_1_3_2_4_2","unstructured":"[n. d.]. Envoy Proxy - Home. Retrieved from https:\/\/www.envoyproxy.io\/"},{"key":"e_1_3_2_5_2","unstructured":"[n. d.]. Istio. Retrieved from https:\/\/istio.io\/latest\/"},{"key":"e_1_3_2_6_2","unstructured":"[n. d.]. Kubernetes. Retrieved from https:\/\/kubernetes.io\/"},{"key":"e_1_3_2_7_2","unstructured":"[n. d.]. Microservices Communication. Retrieved from https:\/\/github.com\/MSrvComm\/original-date:2021-11-05T02:23:34Z."},{"key":"e_1_3_2_8_2","first-page":"557","volume-title":"Building Secure and Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems","author":"Adkins Heather","year":"2020","unstructured":"Heather Adkins, Betsy Beyer, Paul Blankinship, Piotr Lewandowski, Ana Oprea, and Adam Stubblefield. 2020. Building Secure and Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems. O\u2019Reilly Media, 557."},{"key":"e_1_3_2_9_2","unstructured":"Amazon Web Services. 2015. AWS re:Invent 2015: A Day in the Life of a Netflix Engineer (DVO203). Retrieved from https:\/\/www.youtube.com\/watch?v=-mL3zT1iIKw"},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1007\/978-3-030-31646-4_10","article-title":"A hybrid approach to microservices load balancing","author":"Autili Marco","year":"2020","unstructured":"Marco Autili, Alexander Perucci, and Lorenzo De Lauretis. 2020. A hybrid approach to microservices load balancing. In Microservices: Science and Engineering, 249\u2013269.","journal-title":"Microservices: Science and Engineering"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.5555\/3388242.3388291"},{"key":"e_1_3_2_12_2","first-page":"745","article-title":"Balanced allocations: The heavily loaded case","author":"Berenbrink Petra","year":"2000","unstructured":"Petra Berenbrink, Artur Czumaj, and Angelika Steger. 2000. Balanced allocations: The heavily loaded case. In Proceedings of the 32nd Annual ACM Symposium on Theory of Computing. 745\u2013754","journal-title":"Proceedings of the 32nd Annual ACM Symposium on Theory of Computing"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACSOS55765.2022.00027"},{"key":"e_1_3_2_14_2","unstructured":"Netflix Technology Blog. 2018. Rethinking Netflix\u2019s Edge Load Balancing. Medium. Retrieved from https:\/\/netflixtechblog.com\/netflix-edge-load-balancing-695308b5548c"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/1592681.1592693"},{"key":"e_1_3_2_16_2","unstructured":"Inho Cho Ahmed Saeed Joshua Fried Seo Jin Park Mohammad Alizadeh and Adam Belay. 2020. Overload Control for \u00b5s-scale RPCs with Breakwater. Retrieved from https:\/\/www.usenix.org\/conference\/osdi20\/presentation\/cho"},{"key":"e_1_3_2_17_2","unstructured":"Jaana Dogan. [n. d.]. hey - Open Loop Load Generator. Retrieved from https:\/\/github.com\/rakyll\/heyoriginal-date:2016-09-02T10:24:09Z"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3297858.3304013"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","unstructured":"Anshul Gandhi Xi Zhang and Naman Mittal. 2015. HALO: Heterogeneity-aware load balancing. In Proceedings of the 2015 IEEE 23rd International Symposium on Modeling Analysis and Simulation of Computer and Telecommunication Systems. 242\u2013251. DOI: 10.1109\/MASCOTS.2015.14","DOI":"10.1109\/MASCOTS.2015.14"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11280-010-0101-5"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.peva.2007.06.012"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3281010"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3472883.3487014"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/71.963420"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/263326.263335"},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1007\/3-540-49543-6_13","volume-title":"Proceedings of the 2nd International Workshop on Randomization and Approximation Techniques in Computer Science (RANDOM \u201998)","author":"Raab Martin","year":"1998","unstructured":"Martin Raab and Angelika Steger. 1998. \u201cBalls into Bins\u201d - A simple and tight analysis. In Proceedings of the 2nd International Workshop on Randomization and Approximation Techniques in Computer Science (RANDOM \u201998). Springer, Berlin, 159\u2013170."}],"container-title":["ACM Transactions on Autonomous and Adaptive Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3676167","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3676167","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:05:34Z","timestamp":1750291534000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3676167"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,16]]},"references-count":25,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,12,31]]}},"alternative-id":["10.1145\/3676167"],"URL":"https:\/\/doi.org\/10.1145\/3676167","relation":{},"ISSN":["1556-4665","1556-4703"],"issn-type":[{"type":"print","value":"1556-4665"},{"type":"electronic","value":"1556-4703"}],"subject":[],"published":{"date-parts":[[2024,11,16]]},"assertion":[{"value":"2023-03-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-18","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}