{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,8]],"date-time":"2026-04-08T09:05:11Z","timestamp":1775639111408,"version":"3.50.1"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:p>As the use of online AI inference services rapidly expands in various applications (e.g., fraud detection in banking, product recommendation in e-commerce), real-time feature extraction (RTFE) systems have been developed to compute the requested features from incoming data tuples in ultra-low latency. Similar to relational databases, these RTFE procedures can be expressed using SQL-like languages. However, there is a lack of research on the workload characteristics and specialized benchmarks for RTFE, especially in comparison with existing database workloads and benchmarks (e.g., concurrent transactions in TPC-C). In this paper, we study the RTFE workload characteristics using over one hundred real datasets from open repositories (e.g. Kaggle, Tianchi, UCI ML, KiltHub) and those from 4Paradigm. The study highlights the significant differences between RTFE workloads and existing database benchmarks in terms of application scenarios, operator distributions, and query structures. Based on these findings, we propose to develop a realtime feature extraction benchmark named FEBench based on the four important criteria for a domain-specific benchmark proposed by Jim Gray. FEBench consists of selected representative datasets, query templates, and an online request simulator. We use FEBench to evaluate the effectiveness of feature extraction systems including OpenMLDB and Flink and find that each system exhibits distinct advantages and limitations in terms of overall latency, tail latency, and concurrency performance.<\/jats:p>","DOI":"10.14778\/3611540.3611550","type":"journal-article","created":{"date-parts":[[2023,9,15]],"date-time":"2023-09-15T11:32:37Z","timestamp":1694777557000},"page":"3597-3609","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["FEBench: A Benchmark for Real-Time Relational Data Feature Extraction"],"prefix":"10.14778","volume":"16","author":[{"given":"Xuanhe","family":"Zhou","sequence":"first","affiliation":[{"name":"Tsinghua University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cheng","family":"Chen","sequence":"additional","affiliation":[{"name":"4Paradigm Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kunyi","family":"Li","sequence":"additional","affiliation":[{"name":"Tsinghua University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bingsheng","family":"He","sequence":"additional","affiliation":[{"name":"National Univ. of Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mian","family":"Lu","sequence":"additional","affiliation":[{"name":"4Paradigm Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qiaosheng","family":"Liu","sequence":"additional","affiliation":[{"name":"4Paradigm Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wei","family":"Huang","sequence":"additional","affiliation":[{"name":"4Paradigm Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guoliang","family":"Li","sequence":"additional","affiliation":[{"name":"Tsinghua University, Zhongguancun Laboratory"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhao","family":"Zheng","sequence":"additional","affiliation":[{"name":"4Paradigm Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuqiang","family":"Chen","sequence":"additional","affiliation":[{"name":"4Paradigm Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"https:\/\/archive.ics.uci.edu\/ml\/index.php. Last accessed on 2023-2."},{"key":"e_1_2_1_2_1","unstructured":"https:\/\/github.com\/4paradigm\/openmldb. Last accessed on 2023-2."},{"key":"e_1_2_1_3_1","unstructured":"https:\/\/github.com\/akopytov\/sysbench. Last accessed on 2023-2."},{"key":"e_1_2_1_4_1","unstructured":"https:\/\/github.com\/alibaba\/feathub. Last accessed on 2023-2."},{"key":"e_1_2_1_5_1","unstructured":"https:\/\/github.com\/feathr-ai\/feathr. Last accessed on 2023-2."},{"key":"e_1_2_1_6_1","unstructured":"https:\/\/kilthub.cmu.edu\/. Last accessed on 2023-2."},{"key":"e_1_2_1_7_1","unstructured":"https:\/\/medium.com\/engineering-varo\/feature-store-challenges-and-considerations-d1d59c070634. Last accessed on 2023-2."},{"key":"e_1_2_1_8_1","unstructured":"https:\/\/tianchi.aliyun.com\/. Last accessed on 2023-2."},{"key":"e_1_2_1_9_1","unstructured":"https:\/\/www.irs.gov\/pub\/irs-prior\/p3415--2021.pdf. Last accessed on 2023-2."},{"key":"e_1_2_1_10_1","unstructured":"https:\/\/www.kaggle.com\/competitions. Last accessed on 2023-2."},{"key":"e_1_2_1_11_1","unstructured":"https:\/\/www.tecton.ai\/. Last accessed on 2023-2."},{"key":"e_1_2_1_12_1","unstructured":"https:\/\/www.tpc.org. Last accessed on 2023-2."},{"key":"e_1_2_1_13_1","unstructured":"https:\/\/www.tpc.org\/tpcds\/. Last accessed on 2023-2."},{"key":"e_1_2_1_14_1","unstructured":"https:\/\/www.tpc.org\/tpch\/. Last accessed on 2023-2."},{"key":"e_1_2_1_15_1","volume-title":"Gartner","year":"2018","unstructured":"Forecast: The business value of artificial intelligence. In Gartner, 2018."},{"key":"e_1_2_1_16_1","first-page":"1026","volume-title":"Proc. VLDB Endow.","author":"Ahmed R.","year":"2006","unstructured":"R. Ahmed, A. W. Lee, A. Witkowski, D. Das, H. Su, M. Za\u00eft, and T. Cruanes. Cost-based query transformation in oracle. In Proc. VLDB Endow., pages 1026--1036. ACM, 2006."},{"key":"e_1_2_1_17_1","first-page":"355","volume-title":"Advertising on the internet. The Oxford handbook of the digital economy","author":"Anderson S. P.","year":"2012","unstructured":"S. P. Anderson. Advertising on the internet. The Oxford handbook of the digital economy, pages 355--396, 2012."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465296"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2013.03.012"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1214\/ss\/1042727940"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2017.11.077"},{"key":"e_1_2_1_22_1","volume-title":"Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 36(4)","author":"Carbone P.","year":"2015","unstructured":"P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas. Apache flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 36(4), 2015."},{"key":"e_1_2_1_23_1","unstructured":"S. Charrington. Machine learning platforms."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.14778\/3446095.3446102"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1988842.1988850"},{"key":"e_1_2_1_26_1","first-page":"837","volume-title":"Biometrics","author":"DeLong E. R.","year":"1988","unstructured":"E. R. DeLong, D. M. DeLong, and D. L. Clarke-Pearson. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, pages 837--845, 1988."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732240.2732246"},{"key":"e_1_2_1_28_1","volume-title":"The online advertising industry: Economics, evolution, and privacy. Journal of economic perspectives, 23(3):37--60","author":"Evans D. S.","year":"2009","unstructured":"D. S. Evans. The online advertising industry: Economics, evolution, and privacy. Journal of economic perspectives, 23(3):37--60, 2009."},{"key":"e_1_2_1_29_1","volume-title":"The Benchmark Handbook for Database and Transaction Systems","author":"Gray J.","year":"1991","unstructured":"J. Gray, editor. The Benchmark Handbook for Database and Transaction Systems (1st Edition). Morgan Kaufmann, 1991.","edition":"1"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05318-5_10"},{"key":"e_1_2_1_31_1","unstructured":"M. A. Hall. Correlation-based feature selection for machine learning. PhD thesis The University of Waikato 1999."},{"key":"e_1_2_1_32_1","volume-title":"Practical feature subset selection for machine learning","author":"Hall M. A.","year":"1998","unstructured":"M. A. Hall and L. A. Smith. Practical feature subset selection for machine learning. 1998."},{"issue":"2","key":"e_1_2_1_33_1","first-page":"65","article-title":"A survey on feature store","volume":"36","author":"Hur S.","year":"2021","unstructured":"S. Hur and J. Kim. A survey on feature store. Electronics and Telecommunications Trends, 36(2):65--74, 2021.","journal-title":"Electronics and Telecommunications Trends"},{"key":"e_1_2_1_34_1","volume-title":"Olxpbench: Real-time, semantically consistent, and domain-specific are essential in benchmarking, designing, and implementing htap systems. arXiv preprint arXiv:2203.16095","author":"Kang G.","year":"2022","unstructured":"G. Kang, L. Wang, W. Gao, F. Tang, and J. Zhan. Olxpbench: Real-time, semantically consistent, and domain-specific are essential in benchmarking, designing, and implementing htap systems. arXiv preprint arXiv:2203.16095, 2022."},{"key":"e_1_2_1_35_1","first-page":"232","volume-title":"The fifth international conference on the applications of digital information and web technologies (ICADIWT","author":"Khan K.","year":"2014","unstructured":"K. Khan, S. U. Rehman, K. Aziz, S. Fong, and S. Sarasvady. Dbscan: Past, present and future. In The fifth international conference on the applications of digital information and web technologies (ICADIWT 2014), pages 232--238. IEEE, 2014."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0933-3657(01)00077-X"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s41019-020-00149-7"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.14778\/2850583.2850594"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457542"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476311.3476380"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330679"},{"key":"e_1_2_1_42_1","unstructured":"OpenJDK 2013. https:\/\/openjdk.java.net\/projects\/code-tools\/jmh\/ Last accessed on 2020-11-15."},{"key":"e_1_2_1_43_1","volume-title":"Managing ml pipelines: feature stores and the coming wave of embedding ecosystems. arXiv preprint arXiv:2108.05053","author":"Orr L.","year":"2021","unstructured":"L. Orr, A. Sanyal, X. Ling, K. Goel, and M. Leszczynski. Managing ml pipelines: feature stores and the coming wave of embedding ecosystems. arXiv preprint arXiv:2108.05053, 2021."},{"key":"e_1_2_1_44_1","volume-title":"percentile. Tp-x. https:\/\/support.huaweicloud.com\/intl\/en-us\/productdesc-apm\/apm_06_0002.html","author":"T.","year":"2019","unstructured":"T. percentile. Tp-x. https:\/\/support.huaweicloud.com\/intl\/en-us\/productdesc-apm\/apm_06_0002.html, 2019."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.21437\/ICSLP.1994-538"},{"key":"e_1_2_1_46_1","first-page":"474","volume-title":"EDBT","author":"Sun C.","year":"2020","unstructured":"C. Sun, N. Azari, and C. Turakhia. Gallery: A machine learning model management system at uber. In EDBT, pages 474--485, 2020."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.14778\/3368289.3368296"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.14778\/3594512.3594528"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.14778\/3402755.3402798"},{"key":"e_1_2_1_50_1","unstructured":"T. Tsai. Competitive landscape: Ai startups in china. In Technical Report."},{"issue":"1","key":"e_1_2_1_51_1","first-page":"72","article-title":"A normalizing flow based cardinality estimator","volume":"15","author":"Wang J.","year":"2021","unstructured":"J. Wang, C. Chai, J. Liu, and G. Li. FACE: A normalizing flow based cardinality estimator. VLDB, 15(1):72--84, 2021.","journal-title":"VLDB"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICICTA.2010.831"},{"key":"e_1_2_1_53_1","first-page":"02","year":"2019","unstructured":"Wikipedia. LLVM, 2019. [Online; accessed 02-July-2022].","journal-title":"LLVM"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/s41019-022-00186-4"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISPASS.2014.6844459"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.14778\/3565838.3565846"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.14778\/3303753.3303758"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2017.119"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3300067"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2020.2994641"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.14778\/3485450.3485456"},{"key":"e_1_2_1_62_1","volume-title":"Proc. VLDB Endow., 16(12)","author":"Zhou X.","year":"2023","unstructured":"X. Zhou, G. Li, J. Wu, and et al. A learned query rewrite system. Proc. VLDB Endow., 16(12), 2023."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3611540.3611550","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T22:32:59Z","timestamp":1757543579000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3611540.3611550"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8]]},"references-count":62,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["10.14778\/3611540.3611550"],"URL":"https:\/\/doi.org\/10.14778\/3611540.3611550","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2023,8]]},"assertion":[{"value":"2023-08-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}