{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,23]],"date-time":"2026-01-23T18:18:06Z","timestamp":1769192286815,"version":"3.49.0"},"reference-count":37,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2024,8]]},"abstract":"<jats:p>Datastore was one of Google's first cloud databases, launched initially as part of App Engine, and built over Google's internal Megastore database system. Firestore was launched in 2019, both a re-implementation of Datastore over Google's Spanner database system and a new, mobile and web-friendly Firestore API. Spanner was chosen as the storage engine of Firestore in particular for technical reasons---it provides unrestricted transaction capabilities, strong consistency guarantees, and other improvements over Megastore.<\/jats:p>\n          <jats:p>To provide these improvements to all our customers, and simplify our overall system, a non-disruptive, zero-downtime migration was executed of all Datastore databases (stored in Megastore) to Firestore databases (stored in Spanner). This migration took a couple of years to design and plan, and about three to execute. This paper describes both the core engine for migrating databases, and various practical problems that were solved to make this journey successful. As of the writing of this paper, all (over one million) databases have been successfully migrated.<\/jats:p>","DOI":"10.14778\/3685800.3685819","type":"journal-article","created":{"date-parts":[[2024,11,8]],"date-time":"2024-11-08T17:25:21Z","timestamp":1731086721000},"page":"3960-3972","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Transparent Migration from Datastore to Firestore"],"prefix":"10.14778","volume":"17","author":[{"given":"Ed","family":"Davisson","sequence":"first","affiliation":[{"name":"Google, Inc."}]},{"given":"Tilo","family":"Dickopp","sequence":"additional","affiliation":[{"name":"Google, Inc."}]},{"given":"David","family":"Gay","sequence":"additional","affiliation":[{"name":"Google, Inc."}]},{"given":"Eric","family":"Karasuda","sequence":"additional","affiliation":[{"name":"Google, Inc."}]},{"given":"Ram","family":"Kesavan","sequence":"additional","affiliation":[{"name":"Google, Inc."}]},{"given":"Vadim","family":"Yushprakh","sequence":"additional","affiliation":[{"name":"Google, Inc."}]}],"member":"320","published-online":{"date-parts":[[2024,11,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Amazon. 2016. An Overview of AWS Cloud Data Migration Services. https:\/\/docs.aws.amazon.com\/whitepapers\/latest\/overview-aws-cloud-data-migration-services\/overview-aws-cloud-data-migration-services.html. Accessed 2024-07-12."},{"key":"e_1_2_1_2_1","unstructured":"Amazon. 2016. AWS Database Migration Service. https:\/\/aws.amazon.com\/dms\/. Accessed 2024-07-12."},{"key":"e_1_2_1_3_1","unstructured":"Ankita Girish Wagh. 2022. Online Data Migration from HBase to TiDB with Zero Downtime. https:\/\/medium.com\/pinterest-engineering\/online-data-migration-from-hbase-to-tidb-with-zero-downtime-43f0fb474b84. Accessed 2024-07-12."},{"key":"e_1_2_1_4_1","volume-title":"Megastore: Providing Scalable, Highly Available Storage for Interactive Services. In Fifth Biennial Conference on Innovative Data Systems Research, CIDR","author":"Baker Jason","year":"2011","unstructured":"Jason Baker, Chris Bond, James C. Corbett, J. J. Furman, Andrey Khorlin, James Larson, Jean-Michel Leon, Yawei Li, Alexander Lloyd, and Vadim Yushprakh. 2011. Megastore: Providing Scalable, Highly Available Storage for Interactive Services. In Fifth Biennial Conference on Innovative Data Systems Research, CIDR 2011, Asilomar, CA, USA, January 9--12, 2011, Online Proceedings. www.cidrdb.org, 223--234. http:\/\/cidrdb.org\/cidr2011\/Papers\/CIDR11_Paper32.pdf"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/356842.356846"},{"key":"e_1_2_1_6_1","volume-title":"Concurrency Control and Recovery in Database Systems","author":"Bernstein Philip A.","unstructured":"Philip A. Bernstein, Vassco Hadzilacos, and Nathan Goodman. 1987. Concurrency Control and Recovery in Database Systems. Addison-Wesley Longman Publishing Co., Inc., USA."},{"key":"e_1_2_1_7_1","first-page":"363","article-title":"FlumeJava: Easy, Efficient Data-Parallel Pipelines. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). 2 Penn Plaza, Suite 701 New York","volume":"10121","author":"Chambers Craig","year":"2010","unstructured":"Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams, Robert Henry, Robert Bradshaw, and Nathan. 2010. FlumeJava: Easy, Efficient Data-Parallel Pipelines. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). 2 Penn Plaza, Suite 701 New York, NY 10121-0701, 363--375. http:\/\/dl.acm.org\/citation.cfm?id=1806638","journal-title":"NY"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1365815.1365816"},{"key":"e_1_2_1_9_1","unstructured":"Anlek Consulting. 2011. Mongify. https:\/\/github.com\/anlek\/mongify."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491245"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","unstructured":"Ed Davisson Tilo Dickopp David Gay Eric Karasuda Ram Kesavan and Vadim Yushprakh. 2024. Transparent Migration from Datastore to Firestore (Extended Version). https:\/\/research.google.com.","DOI":"10.14778\/3685800.3685819"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/FSKD.2015.7382113"},{"key":"e_1_2_1_13_1","unstructured":"Apache Software Foundation. 2012. Apache Sqoop. https:\/\/sqoop.apache.org\/."},{"key":"e_1_2_1_14_1","unstructured":"Google. [n.d.]. A CIO's Guide to Application Migration. https:\/\/services.google.com\/fh\/files\/misc\/cio_guide_to_application_migraton.pdf. Accessed 2024-07-12."},{"key":"e_1_2_1_15_1","unstructured":"Google. [n.d.]. Automatic Upgrade to Firestore. https:\/\/cloud.google.com\/datastore\/docs\/upgrade-to-firestore. Accessed 2024-07-12."},{"key":"e_1_2_1_16_1","unstructured":"Google. [n.d.]. Cloud Datastore Transactions. https:\/\/cloud.google.com\/datastore\/docs\/concepts\/cloud-datastore-transactions. Accessed 2024-07-12."},{"key":"e_1_2_1_17_1","unstructured":"Google. [n.d.]. Datastore Best Practices. https:\/\/cloud.google.com\/datastore\/docs\/best-practices#indexes. Accessed 2024-07-12."},{"key":"e_1_2_1_18_1","unstructured":"Google. [n.d.]. Firestore API. https:\/\/cloud.google.com\/firestore\/docs\/reference\/rpc. Accessed 2024-07-12."},{"key":"e_1_2_1_19_1","unstructured":"Google. [n.d.]. Firestore: Ramping up traffic. https:\/\/firebase.google.com\/docs\/firestore\/best-practices#ramping_up_traffic. Accessed 2024-07-12."},{"key":"e_1_2_1_20_1","unstructured":"Google. [n.d.]. Overview of Database Migration Service. https:\/\/cloud.google.com\/database-migration\/docs\/overview. Accessed 2024-07-12."},{"key":"e_1_2_1_21_1","unstructured":"Google. [n.d.]. Overview of Key Visualizer. https:\/\/cloud.google.com\/firestore\/docs\/key-visualizer. Accessed 2024-07-12."},{"key":"e_1_2_1_22_1","unstructured":"Google. [n.d.]. Regions and Zones. https:\/\/cloud.google.com\/compute\/docs\/regions-zones. Accessed 2024-07-12."},{"key":"e_1_2_1_23_1","unstructured":"Google. [n.d.]. Transactions. https:\/\/cloud.google.com\/datastore\/docs\/concepts\/transactions. Accessed 2024-07-12."},{"key":"e_1_2_1_24_1","unstructured":"Google. 2019. NoSQL for the serverless age: Announcing Cloud Firestore general availability and updates. https:\/\/cloud.google.com\/blog\/products\/databases\/announcing-cloud-firestore-general-availability-and-updates. Accessed 2024-07-12."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.datak.2014.07.002"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/CICSyN.2015.30"},{"key":"e_1_2_1_27_1","volume-title":"Firestore: The NoSQL Serverless Database for the Application Developer. In 2023 IEEE 39th International Conference on Data Engineering (ICDE). 3367--3379","author":"Kesavan Ram","unstructured":"Ram Kesavan, David Gay, Daniel Thevessen, Jimit Shah, and C. Mohan. 2023. Firestore: The NoSQL Serverless Database for the Application Developer. In 2023 IEEE 39th International Conference on Data Engineering (ICDE). 3367--3379."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/279227.279229"},{"key":"e_1_2_1_29_1","volume-title":"Case Studies in Infrastructure Change Management","author":"Look W.","unstructured":"W. Look, M. Dallman, and an O'Reilly Media Company Safari. 2019. Case Studies in Infrastructure Change Management. O'Reilly Media, Incorporated. https:\/\/books.google.com\/books?id=uUY7zQEACAAJ"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/3PGCIC.2015.64"},{"key":"e_1_2_1_31_1","unstructured":"Microsoft. [n.d.]. Cloud Migration Simplified: A Guide for Migrating Infrastructure Databases and Applications. https:\/\/clouddamcdnprodep.azureedge.net\/gdc\/gdcOB3Q4w\/original. Accessed 2024-07-12."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICACCS51430.2021.9441829"},{"key":"e_1_2_1_33_1","unstructured":"Oracle. [n.d.]. Oracle Cloud Infrastructure Database Migration (DMS). https:\/\/www.oracle.com\/a\/ocom\/docs\/oci-database-migration-onprem-adb.pdf. Accessed 2024-07-12."},{"key":"e_1_2_1_34_1","unstructured":"Charles R. Severance. 2009. Using Google App Engine - start building and running web apps on Google's infrastructure. O'Reilly. http:\/\/www.oreilly.de\/catalog\/9780596800697\/index.html"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICoDSE59534.2023.10291584"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICTSS.2018.8549944"},{"key":"e_1_2_1_37_1","unstructured":"Wikipedia contributors. [n.d.]. Write-ahead Logging. https:\/\/en.wikipedia.org\/wiki\/Write-ahead_logging. Accessed 2024-07-12."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3685800.3685819","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,31]],"date-time":"2024-12-31T05:31:56Z","timestamp":1735623116000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3685800.3685819"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8]]},"references-count":37,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2024,8]]}},"alternative-id":["10.14778\/3685800.3685819"],"URL":"https:\/\/doi.org\/10.14778\/3685800.3685819","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2024,8]]},"assertion":[{"value":"2024-11-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}