{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T22:37:44Z","timestamp":1778279864231,"version":"3.51.4"},"reference-count":32,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2020,8]]},"abstract":"<jats:p>The ongoing and increasing interest in HTAP (Hybrid Transactional and Analytical Processing) systems documents the intense interest from data owners in simultaneously running transactional and analytical workloads over the same data set. Much of the reported work on HTAP has arisen in the context of \"greenfield\" systems, answering the question \"if we could design a system for HTAP from scratch, what would it look like?\" While there is great merit in such an approach, and a lot of valuable technology has been developed with it, we found ourselves facing a different challenge: one in which there is a great deal of transactional data already existing in several transactional systems, heavily queried by an existing federated engine that does not \"own\" the transactional systems, supporting both new and legacy applications that demand transparent fast queries and transactions from this combination. This paper reports on our design and experiences with F1 Lightning, a system we built and deployed to meet this challenge. We describe our design decisions, some details of our implementation, and our experience with the system in production for some of Google's most demanding applications.<\/jats:p>","DOI":"10.14778\/3415478.3415553","type":"journal-article","created":{"date-parts":[[2020,9,14]],"date-time":"2020-09-14T18:46:46Z","timestamp":1600109206000},"page":"3313-3325","source":"Crossref","is-referenced-by-count":38,"title":["F1 lightning"],"prefix":"10.14778","volume":"13","author":[{"given":"Jiacheng","family":"Yang","sequence":"first","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ian","family":"Rae","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jun","family":"Xu","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeff","family":"Shute","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhan","family":"Yuan","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kelvin","family":"Lau","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qiang","family":"Zeng","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xi","family":"Zhao","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jun","family":"Ma","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ziyang","family":"Chen","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuan","family":"Gao","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qilin","family":"Dong","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Junxiong","family":"Zhou","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeremy","family":"Wood","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Goetz","family":"Graefe","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeff","family":"Naughton","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John","family":"Cieslewicz","sequence":"additional","affiliation":[{"name":"Google LLC"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Apache arrow. https:\/\/arrow.apache.org\/."},{"key":"e_1_2_1_2_1","unstructured":"Apache geode. https:\/\/geode.apache.org\/."},{"key":"e_1_2_1_3_1","unstructured":"Oracle database in-memory. https:\/\/www.oracle.com\/a\/tech\/docs\/twp-oracle-database-in-memory-19c.pdf."},{"key":"e_1_2_1_4_1","unstructured":"Parquet. https:\/\/parquet.apache.org\/."},{"key":"e_1_2_1_5_1","unstructured":"Tidb. https:\/\/github.com\/pingcap\/tidb."},{"key":"e_1_2_1_6_1","unstructured":"Tiflash. http:\/\/www.hpts.ws\/papers\/2019\/flash.pdf."},{"key":"e_1_2_1_7_1","unstructured":"Zetasql. https:\/\/github.com\/google\/zetasql\/."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-002-0074-9"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742797"},{"key":"e_1_2_1_10_1","volume-title":"SIGMOD, page 331--343","author":"Bacon D. F.","year":"2017","unstructured":"D. F. Bacon, N. Bales, N. Bruno, B. F. Cooper, A. Dickinson, A. Fikes, C. Fraser, A. Gubarev, M. Joshi, E. Kogan, and et al. Spanner: Becoming a sql system. In SIGMOD, page 331--343, 2017."},{"key":"e_1_2_1_11_1","volume-title":"CIDR","author":"Barber R.","year":"2017","unstructured":"R. Barber, C. Garcia-Arellano, R. Grosman, R. Mueller, V. Raman, R. Sidle, M. Spilchen, A. J. Storm, Y. Tian, P. T\u00f6z\u00fcn, et al. Evolving databases for new-gen big data applications. In CIDR, 2017."},{"key":"e_1_2_1_12_1","first-page":"625","volume-title":"ICDE","author":"Bornea M. A.","year":"2011","unstructured":"M. A. Bornea, O. Hodson, S. Elnikety, and A. Fekete. One-copy serializability with snapshot isolation under the hood. In ICDE, pages 625--636, 2011."},{"key":"e_1_2_1_13_1","volume-title":"SIGMOD, page 739--752","author":"Cecchet E.","year":"2008","unstructured":"E. Cecchet, G. Candea, and A. Ailamaki. Middleware-based database replication: The gaps between theory and practice. In SIGMOD, page 739--752, 2008."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1365815.1365816"},{"issue":"12","key":"e_1_2_1_15_1","first-page":"2022","article-title":"Procella: Unifying serving and analytical data at youtube","volume":"12","author":"Chattopadhyay B.","year":"2019","unstructured":"B. Chattopadhyay, P. Dutta, W. Liu, O. Tinn, A. Mccormick, A. Mokashi, P. Harvey, H. Gonzalez, D. Lomax, S. Mittal, and et al. Procella: Unifying serving and analytical data at youtube. PVLDB, 12(12):2022--2034, 2019.","journal-title":"PVLDB"},{"key":"e_1_2_1_16_1","first-page":"261","volume-title":"OSDI","author":"Corbett J. C.","year":"2012","unstructured":"J. C. Corbett, J. Dean, M. Epstein, A. Fikes, C. Frost, J. J. Furman, S. Ghemawat, A. Gubarev, C. Heiser, P. Hochschild, W. C. Hsieh, S. Kanthak, E. Kogan, H. Li, A. Lloyd, S. Melnik, D. Mwaura, D. Nagle, S. Quinlan, R. Rao, L. Rolig, Y. Saito, M. Szymaniak, C. Taylor, R. Wang, and D. Woodford. Spanner: Google's globally-distributed database. In OSDI, pages 261--264, 2012."},{"key":"e_1_2_1_17_1","first-page":"1","volume-title":"Proceedings of the Third ACM Symposium on Cloud Computing","author":"Das S.","year":"2012","unstructured":"S. Das, C. Botev, K. Surlaker, B. Ghosh, B. Varadarajan, S. Nagaraj, D. Zhang, L. Gao, J. Westerman, P. Ganti, et al. All aboard the databus! linkedin's scalable consistent change data capture platform. In Proceedings of the Third ACM Symposium on Cloud Computing, pages 1--14, 2012."},{"issue":"1","key":"e_1_2_1_18_1","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1145\/1629175.1629198","article-title":"MapReduce: A flexible data processing tool","volume":"53","author":"Dean J.","year":"2010","unstructured":"J. Dean and S. Ghemawat. MapReduce: A flexible data processing tool. CACM, 53(1):72--77, 2010.","journal-title":"CACM"},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1145\/1217935.1217947","volume-title":"Proceedings of the 1st ACM SIGOPS\/EuroSys European Conference on Computer Systems 2006","author":"Elnikety S.","year":"2006","unstructured":"S. Elnikety, S. Dropsho, and F. Pedone. Tashkent: uniting durability with transaction ordering for high-performance scalable database replication. In Proceedings of the 1st ACM SIGOPS\/EuroSys European Conference on Computer Systems 2006, pages 117--130, 2006."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732977.2732999"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137765.3137767"},{"key":"e_1_2_1_22_1","first-page":"1393","volume-title":"SIGMOD","author":"Manoharan G. N. B.","year":"2016","unstructured":"G. N. B. Manoharan, S. Ellner, K. Schnaitter, S. Chegu, A. Estrella-Balderrama, S. Gudmundson, A. Gupta, B. Handy, B. Samwel, C. Whipkey, L. Aharkava, H. Apte, N. Gangahar, J. Xu, S. Venkataraman, D. Agrawal, and J. D. Ullman. Shasta: Interactive reporting at scale. In SIGMOD, pages 1393--1404, 2016."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920886"},{"key":"e_1_2_1_24_1","volume-title":"CIDR","author":"Mozafari B.","year":"2017","unstructured":"B. Mozafari, J. Ramnarayan, S. Menon, Y. Mahajan, S. Chakraborty, H. Bhanawat, and K. Bachhav. Snappydata: A unified cluster for streaming, transactions and interactice analytics. In CIDR, 2017."},{"key":"e_1_2_1_25_1","volume-title":"SIGMOD, page 1771--1775","author":"\u00d6zcan F.","year":"2017","unstructured":"F. \u00d6zcan, Y. Tian, and P. T\u00f6z\u00fcn. Hybrid transactional\/analytical processing: A survey. In SIGMOD, page 1771--1775, 2017."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/s002360050048"},{"key":"e_1_2_1_27_1","first-page":"155","volume-title":"ACM\/IFIP\/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing","author":"Plattner C.","year":"2004","unstructured":"C. Plattner and G. Alonso. Ganymed: Scalable replication for transactional web applications. In ACM\/IFIP\/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing, pages 155--174, 2004."},{"issue":"12","key":"e_1_2_1_28_1","first-page":"1835","article-title":"F1 query: Declarative querying at scale","volume":"11","author":"Samwel B.","year":"2018","unstructured":"B. Samwel, J. Cieslewicz, B. Handy, J. Govig, P. Venetis, C. Yang, K. Peters, J. Shute, D. Tenedorio, H. Apte, and et al. F1 query: Declarative querying at scale. PVLDB, 11(12):1835--1848, 2018.","journal-title":"PVLDB"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536232"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2005.103"},{"key":"e_1_2_1_31_1","volume-title":"PVLDB, page 553--564","author":"Stonebraker M.","year":"2005","unstructured":"M. Stonebraker, D. J. Abadi, A. Batkin, X. Chen, M. Cherniack, M. Ferreira, E. Lau, A. Lin, S. Madden, E. O'Neil, and et al. C-store: A column-oriented dbms. In PVLDB, page 553--564, 2005."},{"key":"e_1_2_1_32_1","first-page":"2","volume-title":"Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation","author":"Zaharia M.","year":"2012","unstructured":"M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, page 2, 2012."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3415478.3415553","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,17]],"date-time":"2025-09-17T02:39:47Z","timestamp":1758076787000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3415478.3415553"}},"subtitle":["HTAP as a service"],"short-title":[],"issued":{"date-parts":[[2020,8]]},"references-count":32,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2020,8]]}},"alternative-id":["10.14778\/3415478.3415553"],"URL":"https:\/\/doi.org\/10.14778\/3415478.3415553","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2020,8]]}}}