{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T15:04:31Z","timestamp":1775228671684,"version":"3.50.1"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2024,8]]},"abstract":"<jats:p>Alibaba relies on its robust database infrastructure to facilitate realtime data access and ensure business continuity despite regional disruptions. To address these operational imperatives, Alibaba developed the Data Transmission Service (DTS), which has become critical for internal applications and public cloud services alike. This paper presents a comprehensive study of the architectural innovations, resource scheduling mechanisms, and performance optimization strategies that have been implemented within DTS to tackle the significant challenges of cross-network, heterogeneous data transmission in a cost-effective manner. We explore the novel Any-to-Any (A2A) architecture, which simplifies the complexity of data paths between diverse databases and mitigates network connectivity issues, thereby significantly reducing development overhead. Additionally, we examine a dynamic network bandwidth scheduling algorithm that effectively maintains Service-Level Objectives (SLOs), complemented by a serverless mechanism that ensures efficient resource utilization. Furthermore, DTS utilizes advanced strategies such as transaction dependency tracking, hot data consolidation, and batching to enhance synchronization performance and efficiency. DTS has distilled the lessons learned from years of serving our customer base and currently supports nearly 1 million public cloud instances annually. Our evaluation results show that DTS can effectively and efficiently handle real-time data transmission in both experimental and production environments.<\/jats:p>","DOI":"10.14778\/3685800.3685822","type":"journal-article","created":{"date-parts":[[2024,11,8]],"date-time":"2024-11-08T17:25:21Z","timestamp":1731086721000},"page":"4001-4013","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Towards Millions of Database Transmission Services in the Cloud"],"prefix":"10.14778","volume":"17","author":[{"given":"Hua","family":"Fan","sequence":"first","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dachao","family":"Fu","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xu","family":"Wang","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiachi","family":"Zhang","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chaoji","family":"Zuo","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhengyi","family":"Wu","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Miao","family":"Zhang","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kang","family":"Yuan","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xizi","family":"Ni","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guocheng","family":"Huo","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wenchao","family":"Zhou","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Feifei","family":"Li","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jingren","family":"Zhou","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,11,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"[n.d.]. Alibaba Cloud Database Services Empower Your Business. https:\/\/www.alibabacloud.com\/en\/product\/databases Accessed: 2024-04-30."},{"key":"e_1_2_1_2_1","unstructured":"[n.d.]. AWS Database Migration Service. https:\/\/aws.amazon.com\/dms\/ Accessed: 2024-04-30."},{"key":"e_1_2_1_3_1","unstructured":"[n.d.]. Azure Database Migration Service. https:\/\/azure.microsoft.com\/products\/database-migration Accessed: 2024-04-30."},{"key":"e_1_2_1_4_1","unstructured":"[n.d.]. Cloud Enterprise Network (CEN). https:\/\/www.alibabacloud.com\/product\/cen Accessed: 2024-04-30."},{"key":"e_1_2_1_5_1","unstructured":"[n.d.]. Database Gateway. https:\/\/www.alibabacloud.com\/help\/en\/database-gateway Accessed: 2024-04-30."},{"key":"e_1_2_1_6_1","unstructured":"[n.d.]. DB-Engines Ranking. https:\/\/db-engines.com\/en\/ranking Accessed: 2024-04-30."},{"key":"e_1_2_1_7_1","unstructured":"[n.d.]. Express Connect. https:\/\/www.alibabacloud.com\/product\/express-connect Accessed: 2024-04-30."},{"key":"e_1_2_1_8_1","unstructured":"[n.d.]. Fivetran. https:\/\/www.fivetran.com\/ Accessed: 2024-04-30."},{"key":"e_1_2_1_9_1","unstructured":"[n.d.]. Google Cloud Database Migration Service. https:\/\/cloud.google.com\/database-migration Accessed: 2024-04-30."},{"key":"e_1_2_1_10_1","unstructured":"[n.d.]. informatica. https:\/\/www.informatica.com\/ Accessed: 2024-04-30."},{"key":"e_1_2_1_11_1","unstructured":"[n.d.]. Locations of access points. https:\/\/www.alibabacloud.com\/help\/en\/express-connect\/user-guide\/locations-of-access-points Accessed: 2024-04-30."},{"key":"e_1_2_1_12_1","unstructured":"[n.d.]. MySQL Replication. https:\/\/dev.mysql.com\/doc\/mysql-replication-excerpt\/5.7\/en\/ Accessed: 2024-04-30."},{"key":"e_1_2_1_13_1","unstructured":"[n.d.]. Oracle GoldenGate. https:\/\/www.oracle.com\/integration\/goldengate\/ Accessed: 2024-04-30."},{"key":"e_1_2_1_14_1","unstructured":"[n.d.]. Qlik. https:\/\/www.qlik.com\/ Accessed: 2024-04-30."},{"key":"e_1_2_1_15_1","unstructured":"[n.d.]. sysbench. https:\/\/github.com\/akopytov\/sysbench Accessed: 2024-04-30."},{"key":"e_1_2_1_16_1","unstructured":"[n.d.]. VPCs and vSwitches. https:\/\/www.alibabacloud.com\/help\/en\/vpc\/user-guide\/vpcs-and-vswitchs\/ Accessed: 2024-04-30."},{"key":"e_1_2_1_17_1","unstructured":"[n.d.]. What is a VPC NAT gateway. https:\/\/www.alibabacloud.com\/help\/en\/nat-gateway\/user-guide\/what-is-a-vpc-nat-gateway Accessed: 2024-04-30."},{"key":"e_1_2_1_18_1","unstructured":"2024. AWS DMS: Challenges & Solutions Guide. https:\/\/www.integrate.io\/blog\/aws-dms-challenges-solutions-guide\/ Accessed: 2024-04-30."},{"key":"e_1_2_1_19_1","unstructured":"2024. Classic Network. https:\/\/www.alibabacloud.com\/help\/en\/ecs\/user-guide\/classic-network Accessed: 2024-04-30."},{"key":"e_1_2_1_20_1","unstructured":"2024. Data Transmission Service: Data Migration and Synchronization - Alibaba Cloud. https:\/\/www.alibabacloud.com\/en\/product\/data-transmission-service Accessed: 2024-04-30."},{"key":"e_1_2_1_21_1","unstructured":"2024. DTS-Reserve parameter. https:\/\/www.alibabacloud.com\/help\/en\/dts\/developer-reference\/reserve-parameter-description Accessed: 2024-04-30."},{"key":"e_1_2_1_22_1","unstructured":"2024. Parallel Recovery - PostgreSQL wiki. https:\/\/wiki.postgresql.org\/wiki\/Parallel_Recovery Accessed: 2024-04-30."},{"key":"e_1_2_1_23_1","unstructured":"2024. Queuing and Scheduling. https:\/\/www.cisco.com\/c\/en\/us\/td\/docs\/switches\/lan\/catalyst9400\/software\/release\/16-6\/configuration_guide\/qos\/b_166_qos_9400_cg\/b_166_qos_9400_cg_chapter_01.html#concept_whp_jdb_p1b Accessed: 2024-04-30."},{"key":"e_1_2_1_24_1","unstructured":"2024. Virtual Private Cloud (VPC). https:\/\/www.alibabacloud.com\/product\/vpc Accessed: 2024-04-30."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2845915"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation","author":"Al-Fares Mohammad","year":"2010","unstructured":"Mohammad Al-Fares, Sivasankar Radhakrishnan, Barath Raghavan, Nelson Huang, and Amin Vahdat. 2010. Hedera: dynamic flow scheduling for data center networks. In Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation (San Jose, California) (NSDI'10). USENIX Association, USA, 19."},{"key":"e_1_2_1_27_1","volume-title":"Hard Real-Time Scheduling: The Deadline-Monotonic Approach. IFAC Proceedings Volumes 24","author":"Audsley N.C.","year":"1991","unstructured":"N.C. Audsley, A. Burns, M.F. Richardson, and A. J. Wellings. 1991. Hard Real-Time Scheduling: The Deadline-Monotonic Approach. IFAC Proceedings Volumes 24, 2 (1991), 127--132."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415548"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/293927.293930"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.14778\/3561261.3561262"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2013.6544908"},{"key":"e_1_2_1_32_1","unstructured":"IBM. [n.d.]. Change-capture replication and full-refresh copying. https:\/\/www.ibm.com\/docs\/en\/idr\/10.2.1?topic=tables-change-capture-replication-full-refresh-copying Accessed: 2024-04-30."},{"key":"e_1_2_1_33_1","volume-title":"An engineering approach to computer networking: ATM networks, the Internet, and the telephone network","author":"Keshav Srinivasan","unstructured":"Srinivasan Keshav and S Kesahv. 1997. An engineering approach to computer networking: ATM networks, the Internet, and the telephone network. Vol. 116. Addison-Wesley Reading."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.14778\/3352063.3352141"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.14778\/3611540.3611565"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMST.2017.2647981"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/J.JSS.2019.110447"},{"key":"e_1_2_1_38_1","volume-title":"Quadratic programming. Numerical optimization","author":"Nocedal Jorge","year":"2006","unstructured":"Jorge Nocedal and Stephen J Wright. 2006. Quadratic programming. Numerical optimization (2006), 448--492."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.14778\/3514061.3514073"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465298"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477132.3483591"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.14778\/3151106.3151107"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/12.57058"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3603269.3604849"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/UCC.2012.36"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3190650"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457556"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.14778\/3352063.3352124"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3685800.3685822","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,31]],"date-time":"2024-12-31T05:32:45Z","timestamp":1735623165000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3685800.3685822"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8]]},"references-count":48,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2024,8]]}},"alternative-id":["10.14778\/3685800.3685822"],"URL":"https:\/\/doi.org\/10.14778\/3685800.3685822","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2024,8]]},"assertion":[{"value":"2024-11-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}