{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T06:14:13Z","timestamp":1784182453889,"version":"3.55.0"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:p>After decades of progress, database management systems (DBMSs) are now the backbones of many data applications that we interact with on a daily basis. Yet, with the emergence of new data types and hardware, building and optimizing new data systems remain as difficult as the heyday of relational databases. In this paper, we summarize our work towards automating the building and optimization of data systems. Drawing from our own experience, we further argue that any automation technique must address three aspects: user specification, code generation, and result validation. We conclude by discussing a case study using videos data processing, along with opportunities for future research towards designing data systems that are automatically generated.<\/jats:p>","DOI":"10.14778\/3611540.3611635","type":"journal-article","created":{"date-parts":[[2023,9,15]],"date-time":"2023-09-15T11:32:37Z","timestamp":1694777557000},"page":"4116-4129","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Towards Auto-Generated Data Systems"],"prefix":"10.14778","volume":"16","author":[{"given":"Alvin","family":"Cheung","sequence":"first","affiliation":[{"name":"University of California, Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Maaz Bin Safeer","family":"Ahmad","sequence":"additional","affiliation":[{"name":"Adobe Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brandon","family":"Haynes","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chanwut","family":"Kittivorawong","sequence":"additional","affiliation":[{"name":"University of California, Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shadaj","family":"Laddad","sequence":"additional","affiliation":[{"name":"University of California, Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaoxuan","family":"Liu","sequence":"additional","affiliation":[{"name":"University of California, Berkeley"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chenglong","family":"Wang","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cong","family":"Yan","sequence":"additional","affiliation":[{"name":"Microsoft Research"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,8]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380144"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3056440"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 2018 International Conference on Management of Data, SIGMOD Conference 2018","author":"Safeer Ahmad Maaz Bin","year":"2018","unstructured":"Maaz Bin Safeer Ahmad and Alvin Cheung. 2018. Automatically Leveraging MapReduce Frameworks for Data-Intensive Applications. In Proceedings of the 2018 International Conference on Management of Data, SIGMOD Conference 2018, Houston, TX, USA, June 10--15, 2018. 1205--1220."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064029"},{"key":"e_1_2_1_5_1","volume-title":"Dedalus: Datalog in Time and Space","author":"Alvaro Peter","year":"2011","unstructured":"Peter Alvaro, William R. Marczak, Neil Conway, Joseph M. Hellerstein, David Maier, and Russell Sears. 2011. Dedalus: Datalog in Time and Space. In Datalog Reloaded, Oege de Moor, Georg Gottlob, Tim Furche, and Andrew Sellers (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 262--281."},{"key":"e_1_2_1_6_1","unstructured":"Apache Flink team. [n.d.]. Apache Flink Project. https:\/\/flink.apache.org."},{"key":"e_1_2_1_7_1","unstructured":"Apache Hadoop 2023. http:\/\/hadoop.apache.org. Accessed on: 2023-08-08."},{"key":"e_1_2_1_8_1","unstructured":"Apache Spark 2023. https:\/\/spark.apache.org. Accessed on: 2023-08-08."},{"key":"e_1_2_1_9_1","unstructured":"Fabrice Bellard. [n.d.]. FFmpeg. https:\/\/ffmpeg.org."},{"key":"e_1_2_1_10_1","volume-title":"37th European Conference on Object-Oriented Programming, ECOOP 2023","author":"Bhatia Sahil","year":"2023","unstructured":"Sahil Bhatia, Sumer Kohli, Sanjit A. Seshia, and Alvin Cheung. 2023. Building Code Transpilers for Domain-Specific Languages Using Program Synthesis (Experience Paper). In 37th European Conference on Object-Oriented Programming, ECOOP 2023, July 17--21, 2023, Seattle, Washington, United States (LIPIcs), Vol. 263. Schloss Dagstuhl - Leibniz-Zentrum f\u00fcr Informatik, 38:1--38:30."},{"key":"e_1_2_1_11_1","unstructured":"Mikel Brostr\u00f6m. 2022. Real-time multi-camera multi-object tracker using YOLOv5 and StrongSORT with OSNet. https:\/\/github.com\/mikel-brostrom\/Yolov5_StrongSORT_OSNet."},{"key":"e_1_2_1_12_1","unstructured":"Chip Brown. 2017. Bringing pixels front and center in VR video. https:\/\/www.blog.google\/products\/google-vr\/bringing-pixels-front-and-center-vr-video."},{"key":"e_1_2_1_13_1","volume-title":"The Conference on Innovative Data Systems Research (CIDR), 14","author":"Cheung Alvin","year":"2021","unstructured":"Alvin Cheung, Natacha Crooks, Joseph M Hellerstein, and Mae Milano. 2021. New directions in cloud programming. The Conference on Innovative Data Systems Research (CIDR), 14."},{"key":"e_1_2_1_14_1","doi-asserted-by":"crossref","unstructured":"Alvin Cheung Armando Solar-Lezama and Samuel Madden. 2013. Optimizing database-backed applications with query synthesis. In PLDI. ACM 3--14.","DOI":"10.1145\/2491956.2462180"},{"key":"e_1_2_1_15_1","volume-title":"Demonstration of the Cosette Automated SQL Prover. In SIGMOD Conference. ACM, 1591--1594","author":"Chu Shumo","year":"2017","unstructured":"Shumo Chu, Daniel Li, Chenglong Wang, Alvin Cheung, and Dan Suciu. 2017. Demonstration of the Cosette Automated SQL Prover. In SIGMOD Conference. ACM, 1591--1594."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.14778\/3236187.3236200"},{"key":"e_1_2_1_17_1","volume-title":"Cosette: An Automated Prover for SQL. In CIDR. www.cidrdb.org.","author":"Chu Shumo","year":"2017","unstructured":"Shumo Chu, Chenglong Wang, Konstantin Weitz, and Alvin Cheung. 2017. Cosette: An Automated Prover for SQL. In CIDR. www.cidrdb.org."},{"key":"e_1_2_1_18_1","doi-asserted-by":"crossref","unstructured":"Shumo Chu Konstantin Weitz Alvin Cheung and Dan Suciu. 2017. HoTTSQL: proving query rewrites with univalent SQL semantics. In PLDI. ACM 510--524.","DOI":"10.1145\/3062341.3062348"},{"key":"e_1_2_1_19_1","volume-title":"TASM: A Tile-Based Storage Manager for Video Analytics. In ICDE. 1775--1786.","author":"Daum Maureen","year":"2021","unstructured":"Maureen Daum, Brandon Haynes, Dong He, Amrita Mazumdar, and Magdalena Balazinska. 2021. TASM: A Tile-Based Storage Manager for Video Analytics. In ICDE. 1775--1786."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1327452.1327492"},{"key":"e_1_2_1_21_1","unstructured":"Django Software Foundation. [n.d.]. Django. https:\/\/djangoproject.com"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476311.3476340"},{"key":"e_1_2_1_23_1","unstructured":"Github. [n.d.]. Github Copilot. https:\/\/github.com\/features\/copilot."},{"key":"e_1_2_1_24_1","volume-title":"Michael Firman, and Gabriel Brostow.","author":"Godard Cl\u00e9ment","year":"2019","unstructured":"Cl\u00e9ment Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. 2019. Digging Into Self-Supervised Monocular Depth Estimation. arXiv:1806.01260 [cs.CV]"},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","unstructured":"Todd J. Green Gregory Karvounarakis and Val Tannen. 2007. Provenance semirings. In PODS. 31--40.","DOI":"10.1145\/1265530.1265535"},{"key":"e_1_2_1_26_1","volume-title":"VisualWorldDB: A DBMS for the Visual World. In Conference on Innovative Data Systems Research.","author":"Haynes Brandon","year":"2020","unstructured":"Brandon Haynes, Maureen Daum, Amrita Mazumdar, Magdalena Balazinska, Alvin Cheung, and Luis Ceze. 2020. VisualWorldDB: A DBMS for the Visual World. In Conference on Innovative Data Systems Research."},{"key":"e_1_2_1_27_1","first-page":"1192","article-title":"LightDB: A DBMS for Virtual Reality Video","volume":"11","author":"Haynes Brandon","year":"2018","unstructured":"Brandon Haynes, Amrita Mazumdar, Armin Alaghi, Magdalena Balazinska, Luis Ceze, and Alvin Cheung. 2018. LightDB: A DBMS for Virtual Reality Video. VLDB 11, 10 (2018), 1192--1205.","journal-title":"VLDB"},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Brandon Haynes Artem Minyaylov Magdalena Balazinska Luis Ceze and Alvin Cheung. 2017. VisualCloud Demonstration: A DBMS for Virtual Reality. In SIGMOD. 1615--1618.","DOI":"10.1145\/3035918.3058734"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2908080.2908117"},{"key":"e_1_2_1_30_1","volume-title":"Spatialyze: A Geospatial Video Analytics System with Spatial-Aware Optimizations. arXiv:2308.03276 [cs.DB]","author":"Kittivorawong Chanwut","year":"2023","unstructured":"Chanwut Kittivorawong, Yongming Ge, Yousef Helal, and Alvin Cheung. 2023. Spatialyze: A Geospatial Video Analytics System with Spatial-Aware Optimizations. arXiv:2308.03276 [cs.DB]"},{"key":"e_1_2_1_31_1","volume-title":"Tyler Hou, Alvin Cheung, and Joseph M. Hellerstein.","author":"Laddad Shadaj","year":"2023","unstructured":"Shadaj Laddad, Conor Power, Tyler Hou, Alvin Cheung, and Joseph M. Hellerstein. 2023. Optimizing Stateful Dataflow with Local Rewrites. arXiv:2306.10585 [cs.PL]"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.14778\/3574245.3574268"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3563336"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.14778\/3583140.3583141"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3089269.3089283"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSTSP.2013.2271451"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/0004-3702(82)90040-6"},{"key":"e_1_2_1_38_1","unstructured":"Charles Gregory Nelson. 1980. Techniques for Program Verification. Ph.D. Dissertation. Stanford CA USA. AAI8011683."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-32033-3_33"},{"key":"e_1_2_1_40_1","unstructured":"Ruby on Rails Team. [n.d.]. Active Record Migrations. https:\/\/guides.rubyonrails.org\/active_record_migrations.html."},{"key":"e_1_2_1_41_1","unstructured":"OpenAI. [n.d.]. ChatGPT. https:\/\/chat.openai.com\/."},{"key":"e_1_2_1_42_1","unstructured":"OpenCV. [n.d.]. Open Source Computer Vision Library. https:\/\/opencv.org."},{"key":"e_1_2_1_43_1","unstructured":"Pandas Development Community. [n.d.]. Python Data Analysis Library. http:\/\/pandas.pydata.org."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2008.142"},{"key":"e_1_2_1_45_1","unstructured":"Ruby on Rails Team. [n.d.]. Ruby on Rails. https:\/\/rubyonrails.org\/"},{"key":"e_1_2_1_46_1","volume-title":"The Computer Graphics Manual","author":"Salomon David","unstructured":"David Salomon. 2011. The Computer Graphics Manual. Springer."},{"key":"e_1_2_1_47_1","volume-title":"Demonstration of Chestnut: An In-memory Data Layout Designer for Database Applications. In ACM SIGMOD Conference.","author":"Samuel Mingwei","year":"2020","unstructured":"Mingwei Samuel, Cong Yan, and Alvin Cheung. 2020. Demonstration of Chestnut: An In-memory Data Layout Designer for Database Applications. In ACM SIGMOD Conference."},{"key":"e_1_2_1_48_1","doi-asserted-by":"crossref","unstructured":"P. Griffiths Selinger M. M. Astrahan D. D. Chamberlin R. A. Lorie and T. G. Price. 1979. Access Path Selection in a Relational Database Management System (SIGMOD '79). 23--34.","DOI":"10.1145\/582095.582099"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 1979 ACM SIGMOD International Conference on Management of Data","author":"Selinger P. Griffiths","unstructured":"P. Griffiths Selinger, M. M. Astrahan, D. D. Chamberlin, R. A. Lorie, and T. G. Price. 1979. Access Path Selection in a Relational Database Management System. In Proceedings of the 1979 ACM SIGMOD International Conference on Management of Data (Boston, Massachusetts) (SIGMOD '79). ACM, New York, NY, USA, 23--34."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.5555\/2050613.2050642"},{"key":"e_1_2_1_51_1","volume-title":"Packet Transactions: A Programming Model for Data-Plane Algorithms at Hardware Speed. CoRR abs\/1512.05023","author":"Sivaraman Anirudh","year":"2015","unstructured":"Anirudh Sivaraman, Mihai Budiu, Alvin Cheung, Changhoon Kim, Steve Licking, George Varghese, Hari Balakrishnan, Mohammad Alizadeh, and Nick McKeown. 2015. Packet Transactions: A Programming Model for Data-Plane Algorithms at Hardware Speed. CoRR abs\/1512.05023 (2015)."},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2006","author":"Solar-Lezama Armando","year":"2006","unstructured":"Armando Solar-Lezama, Liviu Tancau, Rastislav Bod\u00edk, Sanjit A. Seshia, and Vijay A. Saraswat. 2006. Combinatorial sketching for finite programs. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2006, San Jose, CA, USA, October 21--25, 2006. 404--415."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/69.50912"},{"key":"e_1_2_1_54_1","unstructured":"SQLite Team. [n.d.]. History Of SQLite Releases. https:\/\/www.sqlite.org\/chronology.html."},{"key":"e_1_2_1_55_1","unstructured":"Iain Thomson. 2011. Google opens BigQuery for cloud analytics. https:\/\/www.theregister.com\/2011\/11\/14\/google_bigquery_cloud_analytics\/."},{"key":"e_1_2_1_56_1","unstructured":"Tidyverse [n.d.]. Tidyverse: R packages for data science. https:\/\/www.tidyverse.org."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2666356.2594340"},{"key":"e_1_2_1_58_1","first-page":"569","article-title":"Impossibility of an algorithm for the decision problem in finite classes","volume":"70","author":"Trakhtenbrot Boris","year":"1950","unstructured":"Boris Trakhtenbrot. 1950. Impossibility of an algorithm for the decision problem in finite classes. D. Akad. Nauk USSR 70, 1 (1950), 569--572.","journal-title":"D. Akad. Nauk USSR"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3058738"},{"key":"e_1_2_1_60_1","doi-asserted-by":"crossref","unstructured":"Chenglong Wang Alvin Cheung and Rastislav Bod\u00edk. 2017. Synthesizing highly expressive SQL queries from input-output examples. In PLDI. 452--466.","DOI":"10.1145\/3062341.3062365"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3371117"},{"key":"e_1_2_1_62_1","volume-title":"Falx: Synthesis-Powered Visualization Authoring. In CHI '21: CHI Conference on Human Factors in Computing Systems, Virtual Event \/ Yokohama, Japan, May 8--13","author":"Wang Chenglong","year":"2021","unstructured":"Chenglong Wang, Yu Feng, Rastislav Bod\u00edk, Isil Dillig, Alvin Cheung, and Amy J. Ko. 2021. Falx: Synthesis-Powered Visualization Authoring. In CHI '21: CHI Conference on Human Factors in Computing Systems, Virtual Event \/ Yokohama, Japan, May 8--13, 2021. ACM, 106:1--106:15."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3434304"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.14778\/3342263.3342630"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132954"},{"key":"e_1_2_1_66_1","doi-asserted-by":"crossref","unstructured":"Cong Yan Alvin Cheung Junwen Yang and Shan Lu. 2020. View-Driven Optimization of Database-Backed Web Applications.. In CIDR.","DOI":"10.1109\/ICSE.2019.00104"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180194"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264589"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00104"},{"key":"e_1_2_1_70_1","volume-title":"Zettlemoyer and Michael Collins","author":"Luke","year":"2005","unstructured":"Luke S. Zettlemoyer and Michael Collins. 2005. Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars. In UAI '05, Proceedings of the 21st Conference in Uncertainty in Artificial Intelligence, Edinburgh, Scotland, July 26--29, 2005. AUAI Press, 658--666."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","unstructured":"Zhengyou Zhang. 2014. Camera Parameters (Intrinsic Extrinsic). Springer US Boston MA 81--85. 10.1007\/978-0-387-31439-6_152","DOI":"10.1007\/978-0-387-31439-6_152"},{"key":"e_1_2_1_72_1","volume-title":"PLDI '22: 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation","author":"Zhou Xiangyu","year":"2017","unstructured":"Xiangyu Zhou, Rastislav Bod\u00edk, Alvin Cheung, and Chenglong Wang. 2017. Synthesizing analytical SQL queries from computation demonstration. In PLDI '22: 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation, San Diego, CA, USA, June 13 - 17, 2022. ACM, 168--182."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3406534"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/1282480.1282482"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3611540.3611635","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T22:36:41Z","timestamp":1757543801000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3611540.3611635"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8]]},"references-count":74,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["10.14778\/3611540.3611635"],"URL":"https:\/\/doi.org\/10.14778\/3611540.3611635","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2023,8]]},"assertion":[{"value":"2023-08-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}