{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,13]],"date-time":"2026-03-13T08:53:26Z","timestamp":1773392006759,"version":"3.50.1"},"reference-count":94,"publisher":"Association for Computing Machinery (ACM)","issue":"7","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2022,3]]},"abstract":"<jats:p>To allow performance comparison across different systems, our community has developed multiple benchmarks, such as TPC-C and YCSB, which are widely used. However, despite such effort, interpreting and comparing performance numbers is still a challenging task, because one can tune benchmark parameters, system features, and hardware settings, which can lead to very different system behaviors. Such tuning creates a long-standing question of whether the conclusion of a work can hold under different settings.<\/jats:p>\n          <jats:p>This work tries to shed light on this question by reproducing 11 works evaluated under TPC-C and YCSB, measuring their performance under a wider range of settings, and investigating the reasons for the change of performance numbers. By doing so, this paper tries to motivate the discussion about whether and how we should address this problem. While this paper does not give a complete solution---this is beyond the scope of a single paper, it proposes concrete suggestions we can take to improve the state of the art.<\/jats:p>","DOI":"10.14778\/3523210.3523221","type":"journal-article","created":{"date-parts":[[2022,6,22]],"date-time":"2022-06-22T22:23:21Z","timestamp":1655936601000},"page":"1439-1452","source":"Crossref","is-referenced-by-count":8,"title":["A study of database performance sensitivity to experiment settings"],"prefix":"10.14778","volume":"15","author":[{"given":"Yang","family":"Wang","sequence":"first","affiliation":[{"name":"The Ohio State University"}]},{"given":"Miao","family":"Yu","sequence":"additional","affiliation":[{"name":"The Ohio State University"}]},{"given":"Yujie","family":"Hui","sequence":"additional","affiliation":[{"name":"The Ohio State University"}]},{"given":"Fang","family":"Zhou","sequence":"additional","affiliation":[{"name":"The Ohio State University"}]},{"given":"Yuyang","family":"Huang","sequence":"additional","affiliation":[{"name":"The Ohio State University"}]},{"given":"Rui","family":"Zhu","sequence":"additional","affiliation":[{"name":"The Ohio State University"}]},{"given":"Xueyuan","family":"Ren","sequence":"additional","affiliation":[{"name":"The Ohio State University"}]},{"given":"Tianxi","family":"Li","sequence":"additional","affiliation":[{"name":"The Ohio State University"}]},{"given":"Xiaoyi","family":"Lu","sequence":"additional","affiliation":[{"name":"University of California, Merced"}]}],"member":"320","published-online":{"date-parts":[[2022,6,22]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"ACM SIGMOD Reproducibility. https:\/\/reproducibility.sigmod.org\/.  ACM SIGMOD Reproducibility. https:\/\/reproducibility.sigmod.org\/."},{"key":"e_1_2_1_2_1","unstructured":"EuroSys Call for Artifacts. https:\/\/sysartifacts.github.io\/eurosys2021\/.  EuroSys Call for Artifacts. https:\/\/sysartifacts.github.io\/eurosys2021\/."},{"key":"e_1_2_1_3_1","unstructured":"OSDI Call for Artifacts. https:\/\/www.usenix.org\/conference\/osdi21\/call-for-artifacts.  OSDI Call for Artifacts. https:\/\/www.usenix.org\/conference\/osdi21\/call-for-artifacts."},{"key":"e_1_2_1_4_1","unstructured":"PVLDB Reproducibility. http:\/\/vldb.org\/pvldb\/reproducibility\/.  PVLDB Reproducibility. http:\/\/vldb.org\/pvldb\/reproducibility\/."},{"key":"e_1_2_1_5_1","unstructured":"SortBenchmark. http:\/\/sortbenchmark.org\/.  SortBenchmark. http:\/\/sortbenchmark.org\/."},{"key":"e_1_2_1_6_1","unstructured":"SOSP Call for Artifacts. https:\/\/sysartifacts.github.io\/sosp2021\/call.html.  SOSP Call for Artifacts. https:\/\/sysartifacts.github.io\/sosp2021\/call.html."},{"key":"e_1_2_1_7_1","unstructured":"Alibaba-Cloud. TPC-C Result (Alibaba Cloud Elastic Compute Service Cluster ). http:\/\/www.tpc.org\/tpcc\/results\/tpcc_result_detail.asp?id=120051701.  Alibaba-Cloud. TPC-C Result (Alibaba Cloud Elastic Compute Service Cluster ). http:\/\/www.tpc.org\/tpcc\/results\/tpcc_result_detail.asp?id=120051701."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 24th International Conference on Data Engineering, ICDE 2008, April 7--12, 2008","author":"Alomari Mohammad","year":"2008","unstructured":"Mohammad Alomari , Michael J. Cahill , Alan D. Fekete , and Uwe R\u00f6hm . The Cost of Serializability on Platforms that Use Snapshot Isolation. In Gustavo Alonso, Jos\u00e9 A. Blakeley, and Arbee L. P. Chen, editors , Proceedings of the 24th International Conference on Data Engineering, ICDE 2008, April 7--12, 2008 , Canc\u00fan, Mexico, pages 576--585. IEEE Computer Society , 2008 . Mohammad Alomari, Michael J. Cahill, Alan D. Fekete, and Uwe R\u00f6hm. The Cost of Serializability on Platforms that Use Snapshot Isolation. In Gustavo Alonso, Jos\u00e9 A. Blakeley, and Arbee L. P. Chen, editors, Proceedings of the 24th International Conference on Data Engineering, ICDE 2008, April 7--12, 2008, Canc\u00fan, Mexico, pages 576--585. IEEE Computer Society, 2008."},{"key":"e_1_2_1_9_1","unstructured":"Aria Source Code. https:\/\/github.com\/luyi0619\/aria.  Aria Source Code. https:\/\/github.com\/luyi0619\/aria."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2254756.2254766"},{"key":"e_1_2_1_11_1","volume-title":"Concurrency Control in Distributed Database Systems. ACM Computing Surveys (CSUR), 13(2):185--221","author":"Bernstein Philip A","year":"1981","unstructured":"Philip A Bernstein and Nathan Goodman . Concurrency Control in Distributed Database Systems. ACM Computing Surveys (CSUR), 13(2):185--221 , 1981 . Philip A Bernstein and Nathan Goodman. Concurrency Control in Distributed Database Systems. ACM Computing Surveys (CSUR), 13(2):185--221, 1981."},{"key":"e_1_2_1_12_1","volume-title":"Distributed Systems","author":"Budhijara N.","year":"1993","unstructured":"N. Budhijara , K. Marzullo , F. Schneider , and S. Toueg . The Primary-backup Approach . In S. Mullender, editor, Distributed Systems . Addison-Wesley , 2 nd edition, 1993 . N. Budhijara, K. Marzullo, F. Schneider, and S. Toueg. The Primary-backup Approach. In S. Mullender, editor, Distributed Systems. Addison-Wesley, 2nd edition, 1993.","edition":"2"},{"key":"e_1_2_1_13_1","volume-title":"CDCCA","author":"Budhiraja Navin","year":"1992","unstructured":"Navin Budhiraja , Keith Marzullo , Fred B. Schneider , and Sam Toueg . Primary-Backup Protocols : Lower Bounds and Optimal Implementations . In CDCCA , 1992 . Navin Budhiraja, Keith Marzullo, Fred B. Schneider, and Sam Toueg. Primary-Backup Protocols: Lower Bounds and Optimal Implementations. In CDCCA, 1992."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.14778\/3236187.3236209"},{"key":"e_1_2_1_15_1","volume-title":"Ohio Supercomputer Center","author":"Ohio Supercomputer Center","year":"1987","unstructured":"Ohio Supercomputer Center . Ohio Supercomputer Center , 1987 . Ohio Supercomputer Center. Ohio Supercomputer Center, 1987."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1942776.1942778"},{"key":"e_1_2_1_17_1","unstructured":"Cicada Source Code. https:\/\/github.com\/efficient\/cicada-exp-sigmod2017.  Cicada Source Code. https:\/\/github.com\/efficient\/cicada-exp-sigmod2017."},{"key":"e_1_2_1_18_1","unstructured":"Cloudlab. https:\/\/www.cloudlab.us\/.  Cloudlab. https:\/\/www.cloudlab.us\/."},{"key":"e_1_2_1_19_1","volume-title":"DAWNBench: An End-to-End Deep Learning Benchmark and Competition. https:\/\/cs.stanford.edu\/deepakn\/assets\/papers\/dawnbench-sosp17.pdf","author":"Coleman Cody","year":"2017","unstructured":"Cody Coleman , Deepak Narayanan , Daniel Kang , Tian Zhao , Jian Zhang , Luigi Nardi , Peter Bailis , Kunle Olukotun , Chris Re , and Matei Zaharia . DAWNBench: An End-to-End Deep Learning Benchmark and Competition. https:\/\/cs.stanford.edu\/deepakn\/assets\/papers\/dawnbench-sosp17.pdf , 2017 . Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Re, and Matei Zaharia. DAWNBench: An End-to-End Deep Learning Benchmark and Competition. https:\/\/cs.stanford.edu\/deepakn\/assets\/papers\/dawnbench-sosp17.pdf, 2017."},{"key":"e_1_2_1_20_1","first-page":"143","volume-title":"Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC 2010","author":"Cooper Brian F.","year":"2010","unstructured":"Brian F. Cooper , Adam Silberstein , Erwin Tam , Raghu Ramakrishnan , and Russell Sears . Benchmarking Cloud Serving Systems with YCSB. In Joseph M. Hellerstein, Surajit Chaudhuri, and Mendel Rosenblum, editors , Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC 2010 , Indianapolis, Indiana, USA, June 10--11 , 2010 , pages 143 -- 154 . ACM, 2010. Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. Benchmarking Cloud Serving Systems with YCSB. In Joseph M. Hellerstein, Surajit Chaudhuri, and Mendel Rosenblum, editors, Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC 2010, Indianapolis, Indiana, USA, June 10--11, 2010, pages 143--154. ACM, 2010."},{"key":"e_1_2_1_21_1","unstructured":"Artifacts of this Work. https:\/\/github.com\/sam1016yu\/DB-Exp-Sensitivity.  Artifacts of this Work. https:\/\/github.com\/sam1016yu\/DB-Exp-Sensitivity."},{"key":"e_1_2_1_22_1","unstructured":"Hardware Settings Used in this Work. https:\/\/github.com\/sam1016yu\/DB-Exp-Sensitivity\/blob\/main\/hardware_config.md.  Hardware Settings Used in this Work. https:\/\/github.com\/sam1016yu\/DB-Exp-Sensitivity\/blob\/main\/hardware_config.md."},{"key":"e_1_2_1_23_1","first-page":"223","volume-title":"Workshops Proceedings of the 30th International Conference on Data Engineering Workshops, ICDE 2014","author":"Dey Akon","year":"2014","unstructured":"Akon Dey , Alan D. Fekete , Raghunath Nambiar , and Uwe R\u00f6hm . YCSB+T : Benchmarking Web-scale Transactional Databases . In Workshops Proceedings of the 30th International Conference on Data Engineering Workshops, ICDE 2014 , Chicago, IL, USA, March 31 - April 4, 2014 , pages 223 -- 230 . IEEE Computer Society, 2014. Akon Dey, Alan D. Fekete, Raghunath Nambiar, and Uwe R\u00f6hm. YCSB+T: Benchmarking Web-scale Transactional Databases. In Workshops Proceedings of the 30th International Conference on Data Engineering Workshops, ICDE 2014, Chicago, IL, USA, March 31 - April 4, 2014, pages 223--230. IEEE Computer Society, 2014."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2463710"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732240.2732246"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1177\/1094342015593158"},{"key":"e_1_2_1_27_1","first-page":"401","volume-title":"Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014","author":"Dragojevic Aleksandar","year":"2014","unstructured":"Aleksandar Dragojevic , Dushyanth Narayanan , Miguel Castro , and Orion Hodson . FaRM : Fast Remote Memory. In Ratul Mahajan and Ion Stoica, editors , Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014 , Seattle, WA, USA, April 2--4 , 2014 , pages 401 -- 414 . USENIX Association, 2014. Aleksandar Dragojevic, Dushyanth Narayanan, Miguel Castro, and Orion Hodson. FaRM: Fast Remote Memory. In Ratul Mahajan and Ion Stoica, editors, Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014, Seattle, WA, USA, April 2--4, 2014, pages 401--414. USENIX Association, 2014."},{"key":"e_1_2_1_28_1","unstructured":"DrTM Source Code. https:\/\/github.com\/SJTU-IPADS\/drtm.  DrTM Source Code. https:\/\/github.com\/SJTU-IPADS\/drtm."},{"key":"e_1_2_1_29_1","volume-title":"Proc. VLDB Endow, 8(11)","author":"Jose","year":"2015","unstructured":"Jose M Faleiro and Daniel J Abadi. Rethinking Serializable Multiversion Concurrency Control . Proc. VLDB Endow, 8(11) , 2015 . Jose M Faleiro and Daniel J Abadi. Rethinking Serializable Multiversion Concurrency Control. Proc. VLDB Endow, 8(11), 2015."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055553"},{"key":"e_1_2_1_31_1","first-page":"371","volume-title":"Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2013","author":"Fan Bin","year":"2013","unstructured":"Bin Fan , David G. Andersen , and Michael Kaminsky . MemC3 : Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing. In Nick Feamster and Jeffrey C. Mogul, editors , Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2013 , Lombard, IL, USA, April 2--5 , 2013 , pages 371 -- 384 . USENIX Association, 2013. Bin Fan, David G. Andersen, and Michael Kaminsky. MemC3: Compact and Concurrent MemCache with Dumber Caching and Smarter Hashing. In Nick Feamster and Jeffrey C. Mogul, editors, Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2013, Lombard, IL, USA, April 2--5, 2013, pages 371--384. USENIX Association, 2013."},{"key":"e_1_2_1_32_1","first-page":"1477","volume-title":"Alfons Kemper. Low-Latency Communication for Fast DBMS Using RDMA and Shared Memory. In 36th IEEE International Conference on Data Engineering, ICDE 2020","author":"Fent Philipp","year":"2020","unstructured":"Philipp Fent , Alexander van Renen , Andreas Kipf , Viktor Leis , Thomas Neumann , and Alfons Kemper. Low-Latency Communication for Fast DBMS Using RDMA and Shared Memory. In 36th IEEE International Conference on Data Engineering, ICDE 2020 , Dallas, TX, USA, April 20--24 , 2020 , pages 1477 -- 1488 . IEEE, 2020. Philipp Fent, Alexander van Renen, Andreas Kipf, Viktor Leis, Thomas Neumann, and Alfons Kemper. Low-Latency Communication for Fast DBMS Using RDMA and Shared Memory. In 36th IEEE International Conference on Data Engineering, ICDE 2020, Dallas, TX, USA, April 20--24, 2020, pages 1477--1488. IEEE, 2020."},{"key":"e_1_2_1_33_1","unstructured":"fio - flexible i\/o tester. https:\/\/github.com\/axboe\/fio.  fio - flexible i\/o tester. https:\/\/github.com\/axboe\/fio."},{"key":"e_1_2_1_34_1","unstructured":"Fursin Grigori. Reproducing 150 Research Papers and Testing Them in the Real World: Challenges and Solutions. https:\/\/learning.acm.org\/binaries\/content\/assets\/leaning-center\/webinar-slides\/2021\/grigorifursin_techtalk_slides.pdf 2021.  Fursin Grigori. Reproducing 150 Research Papers and Testing Them in the Real World: Challenges and Solutions. https:\/\/learning.acm.org\/binaries\/content\/assets\/leaning-center\/webinar-slides\/2021\/grigorifursin_techtalk_slides.pdf 2021."},{"key":"e_1_2_1_35_1","unstructured":"GAM Source Code. https:\/\/github.com\/ooibc88\/gam.  GAM Source Code. https:\/\/github.com\/ooibc88\/gam."},{"key":"e_1_2_1_36_1","unstructured":"Graph500 Committee. Graph500 benchmark. http:\/\/graph500.org.  Graph500 Committee. Graph500 benchmark. http:\/\/graph500.org."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-08755-9_9"},{"key":"e_1_2_1_38_1","unstructured":"H2. The H2 Home Page. http:\/\/www.h2database.com.  H2. The H2 Home Page. http:\/\/www.h2database.com."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055548"},{"key":"e_1_2_1_40_1","first-page":"2005","volume-title":"Network Emulation with NetEm. In Linux conf au","volume":"5","author":"Stephen","unstructured":"Stephen Hemminger et al . Network Emulation with NetEm. In Linux conf au , volume 5 , page 2005 . Citeseer, 2005. Stephen Hemminger et al. Network Emulation with NetEm. In Linux conf au, volume 5, page 2005. Citeseer, 2005."},{"key":"e_1_2_1_41_1","unstructured":"HERD Source Code. https:\/\/github.com\/efficient\/HERD.  HERD Source Code. https:\/\/github.com\/efficient\/HERD."},{"key":"e_1_2_1_42_1","unstructured":"Janus Source Code. https:\/\/github.com\/NYU-NEWS\/janus.  Janus Source Code. https:\/\/github.com\/NYU-NEWS\/janus."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2619239.2626299"},{"key":"e_1_2_1_44_1","first-page":"1675","volume-title":"Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016","author":"Kim Kangnyeon","year":"2016","unstructured":"Kangnyeon Kim , Tianzheng Wang , Ryan Johnson , and Ippokratis Pandis . ERMIA : Fast Memory-optimized Database System for Heterogeneous Workloads. In Fatma \u00d6zcan, Georgia Koutrika, and Sam Madden, editors , Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016 , San Francisco, CA, USA, June 26 - July 01, 2016 , pages 1675 -- 1687 . ACM, 2016. Kangnyeon Kim, Tianzheng Wang, Ryan Johnson, and Ippokratis Pandis. ERMIA: Fast Memory-optimized Database System for Heterogeneous Workloads. In Fatma \u00d6zcan, Georgia Koutrika, and Sam Madden, editors, Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016, San Francisco, CA, USA, June 26 - July 01, 2016, pages 1675--1687. ACM, 2016."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2746480"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/279227.279229"},{"key":"e_1_2_1_47_1","volume-title":"December","author":"Lamport Leslie","year":"2001","unstructured":"Leslie Lamport . Paxos Made Simple. ACM SIGACT News (Distributed Computing Column), 32(4):51--58 , December 2001 . Leslie Lamport. Paxos Made Simple. ACM SIGACT News (Distributed Computing Column), 32(4):51--58, December 2001."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.14778\/2095686.2095689"},{"key":"e_1_2_1_49_1","first-page":"429","volume-title":"Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014","author":"Lim Hyeontaek","year":"2014","unstructured":"Hyeontaek Lim , Dongsu Han , David G. Andersen , and Michael Kaminsky . MICA : A Holistic Approach to Fast In-Memory Key-Value Storage. In Ratul Mahajan and Ion Stoica, editors , Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014 , Seattle, WA, USA, April 2--4 , 2014 , pages 429 -- 444 . USENIX Association, 2014. Hyeontaek Lim, Dongsu Han, David G. Andersen, and Michael Kaminsky. MICA: A Holistic Approach to Fast In-Memory Key-Value Storage. In Ratul Mahajan and Ion Stoica, editors, Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014, Seattle, WA, USA, April 2--4, 2014, pages 429--444. USENIX Association, 2014."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064015"},{"key":"e_1_2_1_51_1","first-page":"1659","volume-title":"Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016","author":"Lin Qian","year":"2016","unstructured":"Qian Lin , Pengfei Chang , Gang Chen , Beng Chin Ooi , Kian-Lee Tan , and Zhengkui Wang . Towards a Non-2pc Transaction Management in Distributed Database Systems. In Fatma \u00d6zcan, Georgia Koutrika, and Sam Madden, editors , Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016 , San Francisco, CA, USA, June 26-- July 01, 2016 , pages 1659 -- 1674 . ACM, 2016. Qian Lin, Pengfei Chang, Gang Chen, Beng Chin Ooi, Kian-Lee Tan, and Zhengkui Wang. Towards a Non-2pc Transaction Management in Distributed Database Systems. In Fatma \u00d6zcan, Georgia Koutrika, and Sam Madden, editors, Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016, San Francisco, CA, USA, June 26--July 01, 2016, pages 1659--1674. ACM, 2016."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2751519"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.14778\/3407790.3407808"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.14778\/3342263.3342270"},{"key":"e_1_2_1_55_1","first-page":"183","volume-title":"Robert Tappan Morris. Cache Craftiness for Fast Multicore Key-Value Storage. In European Conference on Computer Systems, Proceedings of the Seventh EuroSys Conference 2012","author":"Mao Yandong","year":"2012","unstructured":"Yandong Mao , Eddie Kohler , and Robert Tappan Morris. Cache Craftiness for Fast Multicore Key-Value Storage. In European Conference on Computer Systems, Proceedings of the Seventh EuroSys Conference 2012 , EuroSys '12, Bern, Switzerland, April 10--13 , 2012 , pages 183 -- 196 . ACM, 2012. Yandong Mao, Eddie Kohler, and Robert Tappan Morris. Cache Craftiness for Fast Multicore Key-Value Storage. In European Conference on Computer Systems, Proceedings of the Seventh EuroSys Conference 2012, EuroSys '12, Bern, Switzerland, April 10--13, 2012, pages 183--196. ACM, 2012."},{"key":"e_1_2_1_56_1","volume-title":"Taylor Robie, Tom St. John, Tsuguchika Tabaru, Carole-Jean Wu, Lingjie Xu, Masafumi Yamazaki, Cliff Young, and Matei Zaharia. MLPerf Training Benchmark","author":"Mattson Peter","year":"2020","unstructured":"Peter Mattson , Christine Cheng , Cody Coleman , Greg Diamos , Paulius Micikevicius , David Patterson , Hanlin Tang , Gu-Yeon Wei , Peter Bailis , Victor Bittorf , David Brooks , Dehao Chen , Debojyoti Dutta , Udit Gupta , Kim Hazelwood , Andrew Hock , Xinyuan Huang , Atsushi Ike , Bill Jia , Daniel Kang , David Kanter , Naveen Kumar , Jeffery Liao , Guokai Ma , Deepak Narayanan , Tayo Oguntebi , Gennady Pekhimenko , Lillian Pentecost , Vijay Janapa Reddi , Taylor Robie, Tom St. John, Tsuguchika Tabaru, Carole-Jean Wu, Lingjie Xu, Masafumi Yamazaki, Cliff Young, and Matei Zaharia. MLPerf Training Benchmark , 2020 . Peter Mattson, Christine Cheng, Cody Coleman, Greg Diamos, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, David Brooks, Dehao Chen, Debojyoti Dutta, Udit Gupta, Kim Hazelwood, Andrew Hock, Xinyuan Huang, Atsushi Ike, Bill Jia, Daniel Kang, David Kanter, Naveen Kumar, Jeffery Liao, Guokai Ma, Deepak Narayanan, Tayo Oguntebi, Gennady Pekhimenko, Lillian Pentecost, Vijay Janapa Reddi, Taylor Robie, Tom St. John, Tsuguchika Tabaru, Carole-Jean Wu, Lingjie Xu, Masafumi Yamazaki, Cliff Young, and Matei Zaharia. MLPerf Training Benchmark, 2020."},{"key":"e_1_2_1_57_1","unstructured":"Memcached. http:\/\/memcached.org.  Memcached. http:\/\/memcached.org."},{"key":"e_1_2_1_58_1","unstructured":"MICA Source Code. https:\/\/github.com\/efficient\/mica.  MICA Source Code. https:\/\/github.com\/efficient\/mica."},{"key":"e_1_2_1_59_1","first-page":"103","volume-title":"CPU-Efficient Key-Value Store","author":"Mitchell Christopher","year":"2013","unstructured":"Christopher Mitchell , Yifeng Geng , and Jinyang Li . Using One-Sided RDMA Reads to Build a Fast , CPU-Efficient Key-Value Store . In Andrew Birrell and Emin G\u00fcn Sirer, editors, 2013 USENIX Annual Technical Conference, San Jose, CA, USA , June 26--28, 2013, pages 103 -- 114 . USENIX Association , 2013. Christopher Mitchell, Yifeng Geng, and Jinyang Li. Using One-Sided RDMA Reads to Build a Fast, CPU-Efficient Key-Value Store. In Andrew Birrell and Emin G\u00fcn Sirer, editors, 2013 USENIX Annual Technical Conference, San Jose, CA, USA, June 26--28, 2013, pages 103--114. USENIX Association, 2013."},{"key":"e_1_2_1_60_1","first-page":"517","volume-title":"12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016","author":"Mu Shuai","year":"2016","unstructured":"Shuai Mu , Lamont Nelson , Wyatt Lloyd , and Jinyang Li . Consolidating Concurrency Control and Consensus for Commits under Conflicts. In Kimberly Keeton and Timothy Roscoe, editors , 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016 , Savannah, GA, USA, November 2--4 , 2016 , pages 517 -- 532 . USENIX Association, 2016. Shuai Mu, Lamont Nelson, Wyatt Lloyd, and Jinyang Li. Consolidating Concurrency Control and Consensus for Commits under Conflicts. In Kimberly Keeton and Timothy Roscoe, editors, 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2--4, 2016, pages 517--532. USENIX Association, 2016."},{"key":"e_1_2_1_61_1","unstructured":"MySQL. http:\/\/www.mysql.com.  MySQL. http:\/\/www.mysql.com."},{"key":"e_1_2_1_62_1","first-page":"385","volume-title":"Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2013","author":"Nishtala Rajesh","year":"2013","unstructured":"Rajesh Nishtala , Hans Fugal , Steven Grimm , Marc Kwiatkowski , Herman Lee , Harry C. Li , Ryan McElroy , Mike Paleczny , Daniel Peek , Paul Saab , David Stafford , Tony Tung , and Venkateshwaran Venkataramani . Scaling Memcache at Facebook . In Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2013 , Lombard, IL, USA, April 2--5 , 2013 , pages 385 -- 398 . USENIX Association, 2013. Rajesh Nishtala, Hans Fugal, Steven Grimm, Marc Kwiatkowski, Herman Lee, Harry C. Li, Ryan McElroy, Mike Paleczny, Daniel Peek, Paul Saab, David Stafford, Tony Tung, and Venkateshwaran Venkataramani. Scaling Memcache at Facebook. In Proceedings of the 10th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2013, Lombard, IL, USA, April 2--5, 2013, pages 385--398. USENIX Association, 2013."},{"key":"e_1_2_1_63_1","unstructured":"OLTPBench. https:\/\/github.com\/oltpbenchmark\/oltpbench.  OLTPBench. https:\/\/github.com\/oltpbenchmark\/oltpbench."},{"key":"e_1_2_1_64_1","unstructured":"Oracle. TPC-C Result (SPARC SuperCluster with T3-4 Servers ). http:\/\/www.tpc.org\/tpcc\/results\/tpcc_result_detail.asp?id=110120201.  Oracle. TPC-C Result (SPARC SuperCluster with T3-4 Servers ). http:\/\/www.tpc.org\/tpcc\/results\/tpcc_result_detail.asp?id=110120201."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1965724.1965751"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3056096"},{"key":"e_1_2_1_67_1","first-page":"251","volume-title":"9th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2010, October 4--6, 2010, Vancouver, BC, Canada, Proceedings","author":"Peng Daniel","year":"2010","unstructured":"Daniel Peng and Frank Dabek . Large-scale Incremental Processing using Distributed Transactions and Notifications . In 9th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2010, October 4--6, 2010, Vancouver, BC, Canada, Proceedings , pages 251 -- 264 . USENIX Association , 2010 . Daniel Peng and Frank Dabek. Large-scale Incremental Processing using Distributed Transactions and Notifications. In 9th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2010, October 4--6, 2010, Vancouver, BC, Canada, Proceedings, pages 251--264. USENIX Association, 2010."},{"key":"e_1_2_1_68_1","unstructured":"perf: Linux Profiling with Performance Counters. https:\/\/perf.wiki.kernel.org.  perf: Linux Profiling with Performance Counters. https:\/\/perf.wiki.kernel.org."},{"key":"e_1_2_1_69_1","unstructured":"Petitet A. and Whaley R. C. and Dongarra Jack and Cleary A. HPL - A Portable Implementation of the High-Performance Linpack Benchmark for Distributed-Memory Computers. https:\/\/www.netlib.org\/benchmark\/hpl\/.  Petitet A. and Whaley R. C. and Dongarra Jack and Cleary A. HPL - A Portable Implementation of the High-Performance Linpack Benchmark for Distributed-Memory Computers. https:\/\/www.netlib.org\/benchmark\/hpl\/."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477132.3483591"},{"key":"e_1_2_1_71_1","first-page":"446","volume-title":"Yuchen Zhou. MLPerf Inference Benchmark. In Proceedings of the ACM\/IEEE 47th Annual International Symposium on Computer Architecture, ISCA '20","author":"Reddi Vijay Janapa","year":"2020","unstructured":"Vijay Janapa Reddi , Christine Cheng , David Kanter , Peter Mattson , Guenther Schmuelling , Carole-Jean Wu , Brian Anderson , Maximilien Breughe , Mark Charlebois , William Chou , Ramesh Chukka , Cody Coleman , Sam Davis , Pan Deng , Greg Diamos , Jared Duke , Dave Fick , J. Scott Gardner , Itay Hubara , Sachin Idgunji , Thomas B. Jablin , Jeff Jiao , Tom St. John , Pankaj Kanwar , David Lee , Jeffery Liao , Anton Lokhmotov , Francisco Massa , Peng Meng , Paulius Micikevicius , Colin Osborne , Gennady Pekhimenko , Arun Tejusve Raghunath Rajan , Dilip Sequeira , Ashish Sirasao , Fei Sun , Hanlin Tang , Michael Thomson , Frank Wei , Ephrem Wu , Lingjie Xu , Koichi Yamada , Bing Yu , George Yuan , Aaron Zhong , Peizhao Zhang , and Yuchen Zhou. MLPerf Inference Benchmark. In Proceedings of the ACM\/IEEE 47th Annual International Symposium on Computer Architecture, ISCA '20 , page 446 -- 459 . IEEE Press , 2020 . Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B. Jablin, Jeff Jiao, Tom St. John, Pankaj Kanwar, David Lee, Jeffery Liao, Anton Lokhmotov, Francisco Massa, Peng Meng, Paulius Micikevicius, Colin Osborne, Gennady Pekhimenko, Arun Tejusve Raghunath Rajan, Dilip Sequeira, Ashish Sirasao, Fei Sun, Hanlin Tang, Michael Thomson, Frank Wei, Ephrem Wu, Lingjie Xu, Koichi Yamada, Bing Yu, George Yuan, Aaron Zhong, Peizhao Zhang, and Yuchen Zhou. MLPerf Inference Benchmark. In Proceedings of the ACM\/IEEE 47th Annual International Symposium on Computer Architecture, ISCA '20, page 446--459. IEEE Press, 2020."},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732951.2732955"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415556"},{"key":"e_1_2_1_74_1","unstructured":"Silo Source Code. https:\/\/github.com\/stephentu\/silo.  Silo Source Code. https:\/\/github.com\/stephentu\/silo."},{"key":"e_1_2_1_75_1","unstructured":"Standard Performance Evaluation Corporation. Standard performance evaluation corporation. https:\/\/www.spec.org\/.  Standard Performance Evaluation Corporation. Standard performance evaluation corporation. https:\/\/www.spec.org\/."},{"key":"e_1_2_1_76_1","unstructured":"Star Source Code. https:\/\/github.com\/luyi0619\/star.  Star Source Code. https:\/\/github.com\/luyi0619\/star."},{"key":"e_1_2_1_77_1","first-page":"1150","volume-title":"Proceedings of the 33rd International Conference on Very Large DataBases, VLDB'07","author":"Stonebraker Michael","year":"2007","unstructured":"Michael Stonebraker , Samuel Madden , Daniel J. Abadi , Stavros Harizopoulos , Nabil Hachem , and Pat Helland . The End of an Architectural Era: (It's Time for a Complete Rewrite) . In Proceedings of the 33rd International Conference on Very Large DataBases, VLDB'07 , pages 1150 -- 1160 . VLDB Endowment , 2007 . Michael Stonebraker, Samuel Madden, Daniel J. Abadi, Stavros Harizopoulos, Nabil Hachem, and Pat Helland. The End of an Architectural Era: (It's Time for a Complete Rewrite). In Proceedings of the 33rd International Conference on Very Large DataBases, VLDB'07, pages 1150--1160. VLDB Endowment, 2007."},{"key":"e_1_2_1_78_1","unstructured":"Alexander Thomson Thaddeus Diamond Shu-Chun Weng Kun Ren Philip Shao and Daniel J. Abadi. Calvin Code Repository. https:\/\/github.com\/yaledb\/calvin.  Alexander Thomson Thaddeus Diamond Shu-Chun Weng Kun Ren Philip Shao and Daniel J. Abadi. Calvin Code Repository. https:\/\/github.com\/yaledb\/calvin."},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213838"},{"key":"e_1_2_1_80_1","first-page":"17","volume-title":"Joint 2013 EDBT\/ICDT Conferences, EDBT '13 Proceedings","author":"T\u00f6z\u00fcn Pinar","year":"2013","unstructured":"Pinar T\u00f6z\u00fcn , Ippokratis Pandis , Cansu Kaynak , Djordje Jevdjic , and Anastasia Ailamaki . From A to E : Analyzing TPC's OLTP Benchmarks: The Obsolete, the Ubiquitous, the Unexplored . In Joint 2013 EDBT\/ICDT Conferences, EDBT '13 Proceedings , Genoa, Italy, March 18--22 , 2013 , pages 17 -- 28 . ACM, 2013. Pinar T\u00f6z\u00fcn, Ippokratis Pandis, Cansu Kaynak, Djordje Jevdjic, and Anastasia Ailamaki. From A to E: Analyzing TPC's OLTP Benchmarks: The Obsolete, the Ubiquitous, the Unexplored. In Joint 2013 EDBT\/ICDT Conferences, EDBT '13 Proceedings, Genoa, Italy, March 18--22, 2013, pages 17--28. ACM, 2013."},{"key":"e_1_2_1_81_1","unstructured":"Transaction Processing Performance Council. The TPC-E home page. http:\/\/www.tpc.org\/tpce\/.  Transaction Processing Performance Council. The TPC-E home page. http:\/\/www.tpc.org\/tpce\/."},{"key":"e_1_2_1_82_1","unstructured":"Transaction Processing Performance Council. TPC Benchmark C Standard Specification Revision 5.11. http:\/\/tpc.org\/tpc_documents_current_versions\/pdf\/tpc-c_v5.11.0.pdf.  Transaction Processing Performance Council. TPC Benchmark C Standard Specification Revision 5.11. http:\/\/tpc.org\/tpc_documents_current_versions\/pdf\/tpc-c_v5.11.0.pdf."},{"key":"e_1_2_1_83_1","unstructured":"Transaction Processing Performance Council. TPC-C - All Results - Sorted by Performance. http:\/\/tpc.org\/tpcc\/results\/tpcc_results5.asp.  Transaction Processing Performance Council. TPC-C - All Results - Sorted by Performance. http:\/\/tpc.org\/tpcc\/results\/tpcc_results5.asp."},{"key":"e_1_2_1_84_1","unstructured":"Transaction Processing Performance Council. The TPC-C home page. http:\/\/www.tpc.org\/tpcc\/.  Transaction Processing Performance Council. The TPC-C home page. http:\/\/www.tpc.org\/tpcc\/."},{"key":"e_1_2_1_85_1","unstructured":"Transaction Processing Performance Council. Tpc-homepage. http:\/\/www.tpc.org.  Transaction Processing Performance Council. Tpc-homepage. http:\/\/www.tpc.org."},{"key":"e_1_2_1_86_1","first-page":"18","volume-title":"Samuel Madden. Speedy Transactions in Multicore In-memory Databases. In ACM SIGOPS 24th Symposium on Operating Systems Principles, SOSP '13","author":"Tu Stephen","year":"2013","unstructured":"Stephen Tu , Wenting Zheng , Eddie Kohler , Barbara Liskov , and Samuel Madden. Speedy Transactions in Multicore In-memory Databases. In ACM SIGOPS 24th Symposium on Operating Systems Principles, SOSP '13 , Farmington, PA, USA, November 3--6 , 2013 , pages 18 -- 32 . ACM, 2013. Stephen Tu, Wenting Zheng, Eddie Kohler, Barbara Liskov, and Samuel Madden. Speedy Transactions in Multicore In-memory Databases. In ACM SIGOPS 24th Symposium on Operating Systems Principles, SOSP '13, Farmington, PA, USA, November 3--6, 2013, pages 18--32. ACM, 2013."},{"key":"e_1_2_1_87_1","first-page":"513","volume-title":"17th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2020","author":"Uta Alexandru","year":"2020","unstructured":"Alexandru Uta , Alexandru Custura , Dmitry Duplyakin , Ivo Jimenez , Jan Rellermeyer , Carlos Maltzahn , Robert Ricci , and Alexandru Iosup . Is Big Data Performance Reproducible in Modern Cloud Networks? . In 17th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2020 , Santa Clara, CA, USA, February 25--27 , 2020 , pages 513 -- 527 . USENIX Association, 2020. Alexandru Uta, Alexandru Custura, Dmitry Duplyakin, Ivo Jimenez, Jan Rellermeyer, Carlos Maltzahn, Robert Ricci, and Alexandru Iosup. Is Big Data Performance Reproducible in Modern Cloud Networks? . In 17th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2020, Santa Clara, CA, USA, February 25--27, 2020, pages 513--527. USENIX Association, 2020."},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.14778\/3015274.3015276"},{"key":"e_1_2_1_89_1","first-page":"87","volume-title":"Haibo Chen. Fast In-memory Transaction Processing Using RDMA and HTM. In Proceedings of the 25th Symposium on Operating Systems Principles, SOSP 2015","author":"Wei Xingda","year":"2015","unstructured":"Xingda Wei , Jiaxin Shi , Yanzhe Chen , Rong Chen , and Haibo Chen. Fast In-memory Transaction Processing Using RDMA and HTM. In Proceedings of the 25th Symposium on Operating Systems Principles, SOSP 2015 , Monterey, CA, USA, October 4--7 , 2015 , pages 87 -- 104 . ACM, 2015. Xingda Wei, Jiaxin Shi, Yanzhe Chen, Rong Chen, and Haibo Chen. Fast In-memory Transaction Processing Using RDMA and HTM. In Proceedings of the 25th Symposium on Operating Systems Principles, SOSP 2015, Monterey, CA, USA, October 4--7, 2015, pages 87--104. ACM, 2015."},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.14778\/3067421.3067427"},{"key":"e_1_2_1_91_1","first-page":"191","volume-title":"14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020","author":"Yang Juncheng","year":"2020","unstructured":"Juncheng Yang , Yao Yue , and K. V. Rashmi . A Large Scale Analysis of Hundreds of In-memory Cache Clusters at Twitter . In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020 , Virtual Event, November 4--6 , 2020 , pages 191 -- 208 . USENIX Association, 2020. Juncheng Yang, Yao Yue, and K. V. Rashmi. A Large Scale Analysis of Hundreds of In-memory Cache Clusters at Twitter. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4--6, 2020, pages 191--208. USENIX Association, 2020."},{"key":"e_1_2_1_92_1","first-page":"1629","volume-title":"Srinivas Devadas. Tictoc: Time Traveling Optimistic Concurrency Control. In Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016","author":"Yu Xiangyao","year":"2016","unstructured":"Xiangyao Yu , Andrew Pavlo , Daniel S\u00e1nchez , and Srinivas Devadas. Tictoc: Time Traveling Optimistic Concurrency Control. In Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016 , San Francisco, CA, USA, June 26 - July 01, 2016 , pages 1629 -- 1642 . ACM, 2016. Xiangyao Yu, Andrew Pavlo, Daniel S\u00e1nchez, and Srinivas Devadas. Tictoc: Time Traveling Optimistic Concurrency Control. In Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016, San Francisco, CA, USA, June 26 - July 01, 2016, pages 1629--1642. ACM, 2016."},{"key":"e_1_2_1_93_1","unstructured":"Irene Zhang Naveen Kr. Sharma Adriana Szekeres Arvind Krishnamurthy and Dan R. K. Ports. Tapir Code Repository. https:\/\/github.com\/UWSysLab\/tapir.  Irene Zhang Naveen Kr. Sharma Adriana Szekeres Arvind Krishnamurthy and Dan R. K. Ports. Tapir Code Repository. https:\/\/github.com\/UWSysLab\/tapir."},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815400.2815404"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3523210.3523221","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:52:03Z","timestamp":1672224723000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3523210.3523221"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3]]},"references-count":94,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2022,3]]}},"alternative-id":["10.14778\/3523210.3523221"],"URL":"https:\/\/doi.org\/10.14778\/3523210.3523221","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2022,3]]}}}