{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T17:59:36Z","timestamp":1774979976906,"version":"3.50.1"},"reference-count":81,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,9,30]],"date-time":"2024-09-30T00:00:00Z","timestamp":1727654400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100006374","name":"NSF","doi-asserted-by":"publisher","award":["2118745"],"award-info":[{"award-number":["2118745"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2024,10,1]]},"abstract":"<jats:p>Benchmark and system parameters often have a significant impact on performance evaluation, which raises a long-lasting question about which settings we should use.<\/jats:p>\n                  <jats:p>This paper studies the feasibility and benefits of extensive evaluation. A full extensive evaluation, which tests all possible settings, is usually too expensive. This work investigates whether it is possible to sample a subset of the settings and, upon them, generate observations that match those from a full extensive evaluation. Towards this goal, we have explored the incremental sampling approach, which starts by measuring a small subset of random settings, builds a prediction model on these samples using the popular ANOVA approach, adds more samples if the model is not accurate enough, and terminates otherwise.<\/jats:p>\n                  <jats:p>To summarize our findings: 1) Enhancing a research prototype to support extensive evaluation mostly involves changing hard-coded configurations, which does not take much effort. 2) Some systems are highly predictable, which means that they can achieve accurate predictions with a low sampling rate, but some systems are less predictable. 3) We have not found a method that can consistently outperform random sampling + ANOVA. Based on these findings, we provide recommendations to improve artifact predictability and strategies for selecting parameter values during evaluation.<\/jats:p>","DOI":"10.1145\/3677137","type":"journal-article","created":{"date-parts":[[2024,9,30]],"date-time":"2024-09-30T17:41:44Z","timestamp":1727718104000},"page":"1-24","source":"Crossref","is-referenced-by-count":1,"title":["On the Feasibility and Benefits of Extensive Evaluation"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-8102-1355","authenticated-orcid":false,"given":"Yujie","family":"Hui","sequence":"first","affiliation":[{"name":"The Ohio State University, Columbus, Ohio, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0539-5692","authenticated-orcid":false,"given":"Miao","family":"Yu","sequence":"additional","affiliation":[{"name":"The Ohio State University, Columbus, Ohio, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-8795-5262","authenticated-orcid":false,"given":"Hao","family":"Qi","sequence":"additional","affiliation":[{"name":"University of California, Merced, Merced, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0732-058X","authenticated-orcid":false,"given":"Yifan","family":"Gan","sequence":"additional","affiliation":[{"name":"The Ohio State University, Columbus, Ohio, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-8059-8861","authenticated-orcid":false,"given":"Tianxi","family":"Li","sequence":"additional","affiliation":[{"name":"The Ohio State University, Columbus, Ohio, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-5806-7228","authenticated-orcid":false,"given":"Yuke","family":"Li","sequence":"additional","affiliation":[{"name":"University of California, Merced, Merced, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-0305-7565","authenticated-orcid":false,"given":"Xueyuan","family":"Ren","sequence":"additional","affiliation":[{"name":"The Ohio State University, Columbus, Ohio, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-2427-739X","authenticated-orcid":false,"given":"Sixiang","family":"Ma","sequence":"additional","affiliation":[{"name":"The Ohio State University, Columbus, Ohio, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7581-8905","authenticated-orcid":false,"given":"Xiaoyi","family":"Lu","sequence":"additional","affiliation":[{"name":"University of California, Merced, Merced, California, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9721-4923","authenticated-orcid":false,"given":"Yang","family":"Wang","sequence":"additional","affiliation":[{"name":"The Ohio State University, Columbus, Ohio, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,9,30]]},"reference":[{"key":"e_1_2_2_1_1","unstructured":"ACM SIGMOD Reproducibility. https:\/\/reproducibility.sigmod.org\/."},{"key":"e_1_2_2_2_1","unstructured":"Calvin Source Code. https:\/\/github.com\/yaledb\/calvin."},{"key":"e_1_2_2_3_1","unstructured":"EuroSys Call for Artifacts. https:\/\/sysartifacts.github.io\/eurosys2021\/."},{"key":"e_1_2_2_4_1","unstructured":"OSDI Call for Artifacts. https:\/\/www.usenix.org\/conference\/osdi21\/call-for-artifacts."},{"key":"e_1_2_2_5_1","unstructured":"PVLDB Reproducibility. http:\/\/vldb.org\/pvldb\/reproducibility\/."},{"key":"e_1_2_2_6_1","unstructured":"SOSP Call for Artifacts. https:\/\/sysartifacts.github.io\/sosp2021\/call.html."},{"key":"e_1_2_2_7_1","unstructured":"Tapir Source Code. https:\/\/github.com\/UWSysLab\/tapir."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2008.4497466"},{"key":"e_1_2_2_9_1","unstructured":"Aria Source Code. https:\/\/github.com\/luyi0619\/aria."},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2254756.2254766"},{"key":"e_1_2_2_11_1","first-page":"1","volume-title":"DaMoN","author":"Bang Tiemo","year":"2020","unstructured":"Tiemo Bang, Norman May, Ilia Petrov, and Carsten Binnig. The Tale of 1000 Cores: An Evaluation of Concurrency Control on Real(ly) Large Multi-socket Hardware. In DaMoN, pages 3:1--3:9. ACM, 2020."},{"key":"e_1_2_2_12_1","volume-title":"The Full Story of 1000 Cores: An Examination of Concurrency Control on Real(ly) Large Multi-socket Hardware. The VLDB Journal, 31(6):1185--1213, apr","author":"Bang Tiemo","year":"2022","unstructured":"Tiemo Bang, Norman May, Ilia Petrov, and Carsten Binnig. The Full Story of 1000 Cores: An Examination of Concurrency Control on Real(ly) Large Multi-socket Hardware. The VLDB Journal, 31(6):1185--1213, apr 2022."},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2015.2449299"},{"key":"e_1_2_2_14_1","first-page":"753","volume-title":"OSDI","author":"Berg Benjamin","year":"2020","unstructured":"Benjamin Berg, Daniel S. Berger, Sara McAllister, Isaac Grosof, Sathya Gunasekar, Jimmy Lu, Michael Uhlar, Jim Carrig, Nathan Beckmann, Mor Harchol-Balter, and Gregory R. Ganger. The CacheLib Caching Engine: Design and Experiences at Scale. In OSDI, pages 753--768. USENIX Association, 2020."},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.14778\/3236187.3236209"},{"key":"e_1_2_2_16_1","first-page":"205","volume-title":"OSDI","author":"Chang Fay","year":"2006","unstructured":"Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Michael Burrows, Tushar Chandra, Andrew Fikes, and Robert Gruber. Bigtable: A Distributed Storage System for Structured Data. In OSDI, pages 205--218. USENIX Association, 2006."},{"key":"e_1_2_2_17_1","unstructured":"Cicada Source Code. https:\/\/github.com\/efficient\/cicada-exp-sigmod2017."},{"key":"e_1_2_2_18_1","unstructured":"Cloudlab. https:\/\/www.cloudlab.us\/."},{"key":"e_1_2_2_19_1","volume-title":"DAWNBench: An End-to-End Deep Learning Benchmark and Competition. https:\/\/cs.stanford.edu\/~deepakn\/assets\/papers\/dawnbench-sosp17.pdf","author":"Coleman Cody","year":"2017","unstructured":"Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Re, and Matei Zaharia. DAWNBench: An End-to-End Deep Learning Benchmark and Competition. https:\/\/cs.stanford.edu\/~deepakn\/assets\/papers\/dawnbench-sosp17.pdf, 2017."},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807152"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1294261.1294281"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556583"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2541940.2541941"},{"key":"e_1_2_2_24_1","first-page":"343","volume-title":"NSDI","author":"Dong Mo","year":"2018","unstructured":"Mo Dong, Tong Meng, Doron Zarchy, Engin Arslan, Yossi Gilad, Brighten Godfrey, and Michael Schapira. PCC Vivace: Online-Learning Congestion Control. In NSDI, pages 343--356. USENIX Association, 2018."},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1177\/1094342015593158"},{"key":"e_1_2_2_26_1","unstructured":"DrTM Source Code. https:\/\/github.com\/SJTU-IPADS\/drtm."},{"key":"e_1_2_2_27_1","unstructured":"fio - Flexible I\/O Tester. https:\/\/github.com\/axboe\/fio."},{"key":"e_1_2_2_28_1","first-page":"763","volume-title":"NSDI","author":"Fu Silvery","year":"2021","unstructured":"Silvery Fu, Saurabh Gupta, Radhika Mittal, and Sylvia Ratnasamy. On the Use of ML for Blackbox System Performance Prediction. In NSDI, pages 763--784. USENIX Association, 2021."},{"key":"e_1_2_2_29_1","unstructured":"Fursin Grigori. Reproducing 150 Research Papers and Testing Them in the RealWorld: Challenges and Solutions. https:\/\/learning.acm.org\/binaries\/content\/assets\/leaning-center\/webinar-slides\/2021\/grigorifursin_techtalk_slides.pdf 2021."},{"key":"e_1_2_2_30_1","unstructured":"GAM Source Code. https:\/\/github.com\/ooibc88\/gam."},{"key":"e_1_2_2_31_1","unstructured":"Graph500 Committee. Graph500 benchmark. http:\/\/graph500.org."},{"key":"e_1_2_2_32_1","volume-title":"Predicting Performance of Software Configurations: There is no Silver Bullet. CoRR, abs\/1911.12643","author":"Grebhahn Alexander","year":"2019","unstructured":"Alexander Grebhahn, Norbert Siegmund, and Sven Apel. Predicting Performance of Software Configurations: There is no Silver Bullet. CoRR, abs\/1911.12643, 2019."},{"key":"e_1_2_2_33_1","first-page":"173","volume-title":"OSDI","author":"Hao Mingzhe","year":"2020","unstructured":"Mingzhe Hao, Levent Toksoz, Nanqinqin Li, Edward Edberg Halim, Henry Hoffmann, and Haryadi S. Gunawi. LinnOS: Predictability on Unpredictable Flash Storage with a Light Neural Network. In OSDI, pages 173--190. USENIX Association, 2020."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055548"},{"key":"e_1_2_2_35_1","unstructured":"Gernot Heiser. Systems Benchmarking Crimes. https:\/\/gernot-heiser.org\/benchmarking-crimes.html."},{"key":"e_1_2_2_36_1","unstructured":"HERD Source Code. https:\/\/github.com\/efficient\/HERD."},{"key":"e_1_2_2_37_1","first-page":"198","volume-title":"SOSP","author":"Hoffmann Henry","year":"2015","unstructured":"Henry Hoffmann. Jouleguard: energy guarantees for approximate applications. In SOSP, pages 198--214. ACM, 2015."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1111\/1467-8659.00681"},{"key":"e_1_2_2_39_1","volume-title":"Measurement, Simulation, and Modeling","author":"Jain Raj","year":"1991","unstructured":"Raj Jain. The Art of Computer Systems Performance Analysis - Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley professional computing. Wiley, 1991."},{"key":"e_1_2_2_40_1","unstructured":"Janus Source Code. https:\/\/github.com\/NYU-NEWS\/janus."},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2619239.2626299"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2020.2987024"},{"key":"e_1_2_2_43_1","first-page":"553","volume-title":"SOSP","author":"Lee Sangmin","year":"2021","unstructured":"Sangmin Lee, Zhenhua Guo, Omer Sunercan, Jun Ying, Thawan Kooburat, Suryadeep Biswal, Jun Chen, Kun Huang, Yatpang Cheung, Yiding Zhou, Kaushik Veeraraghavan, Biren Damani, Pol Mauri Ruiz, Vikas Mehta, and Chunqiang Tang. Shard Manager: A Generic Shard Management Framework for Geo-distributed Applications. In SOSP, pages 553--569. ACM, 2021."},{"key":"e_1_2_2_44_1","first-page":"1081","volume-title":"NSDI","author":"Mike Liang Chieh-Jan","year":"2023","unstructured":"Chieh-Jan Mike Liang, Zilin Fang, Yuqing Xie, Fan Yang, Zhao Lucis Li, Li Lyna Zhang, Mao Yang, and Lidong Zhou. On Modular Learning of Distributed Systems for Predicting End-to-End Latency. In NSDI, pages 1081--1095. USENIX Association, 2023."},{"key":"e_1_2_2_45_1","first-page":"21","volume-title":"Andersen. Cicada: Dependably Fast Multi-Core In-Memory Transactions. In SIGMOD Conference","author":"Lim Hyeontaek","year":"2017","unstructured":"Hyeontaek Lim, Michael Kaminsky, and David G. Andersen. Cicada: Dependably Fast Multi-Core In-Memory Transactions. In SIGMOD Conference, pages 21--35. ACM, 2017."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.14778\/3407790.3407808"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.14778\/3342263.3342270"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3373376.3378525"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341302.3342080"},{"key":"e_1_2_2_50_1","first-page":"409","volume-title":"OSDI","author":"Maricq Aleksander","year":"2018","unstructured":"Aleksander Maricq, Dmitry Duplyakin, Ivo Jimenez, Carlos Maltzahn, Ryan Stutsman, Robert Ricci, and Ana Klimovic. Taming Performance Variability. In OSDI, pages 409--425. USENIX Association, 2018."},{"key":"e_1_2_2_51_1","volume-title":"Taylor Robie, Tom St. John, Carole-Jean Wu, Lingjie Xu, Cliff Young, and Matei Zaharia. MLPerf Training Benchmark. In MLSys. mlsys.org","author":"Mattson Peter","year":"2020","unstructured":"Peter Mattson, Christine Cheng, Gregory F. Diamos, Cody Coleman, Paulius Micikevicius, David A. Patterson, Hanlin Tang, Gu-YeonWei, Peter Bailis, Victor Bittorf, David Brooks, Dehao Chen, Debo Dutta, Udit Gupta, Kim M. Hazelwood, Andy Hock, Xinyuan Huang, Daniel Kang, David Kanter, Naveen Kumar, Jeffery Liao, Deepak Narayanan, Tayo Oguntebi, Gennady Pekhimenko, Lillian Pentecost, Vijay Janapa Reddi, Taylor Robie, Tom St. John, Carole-Jean Wu, Lingjie Xu, Cliff Young, and Matei Zaharia. MLPerf Training Benchmark. In MLSys. mlsys.org, 2020."},{"key":"e_1_2_2_52_1","first-page":"184","volume-title":"ASPLOS","author":"Mishra Nikita","year":"2018","unstructured":"Nikita Mishra, Connor Imes, John D. Lafferty, and Henry Hoffmann. CALOREE: Learning Control for Predictable Latency and Low Energy. In ASPLOS, pages 184--198. ACM, 2018."},{"key":"e_1_2_2_53_1","first-page":"517","volume-title":"OSDI","author":"Mu Shuai","year":"2016","unstructured":"Shuai Mu, Lamont Nelson, Wyatt Lloyd, and Jinyang Li. Consolidating Concurrency Control and Consensus for Commits under Conflicts. In OSDI, pages 517--532. USENIX Association, 2016."},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1002\/0471667196.ess5050"},{"key":"e_1_2_2_55_1","first-page":"385","volume-title":"NSDI","author":"Nishtala Rajesh","year":"2013","unstructured":"Rajesh Nishtala, Hans Fugal, Steven Grimm, Marc Kwiatkowski, Herman Lee, Harry C. Li, Ryan McElroy, Mike Paleczny, Daniel Peek, Paul Saab, David Stafford, Tony Tung, and Venkateshwaran Venkataramani. Scaling Memcache at Facebook. In NSDI, pages 385--398. USENIX Association, 2013."},{"key":"e_1_2_2_56_1","first-page":"1","volume-title":"Wiley StatsRef: Statistics Reference Online","author":"Parsons Van L","year":"2014","unstructured":"Van L Parsons. Stratified Sampling. Wiley StatsRef: Statistics Reference Online, pages 1--11, 2014."},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_2_2_58_1","unstructured":"Petitet A. and Whaley R. C. and Dongarra Jack and Cleary A. HPL - A Portable Implementation of the High-Performance Linpack Benchmark for Distributed-Memory Computers. https:\/\/www.netlib.org\/benchmark\/hpl\/."},{"key":"e_1_2_2_59_1","first-page":"446","volume-title":"ISCA","author":"Reddi Vijay Janapa","year":"2020","unstructured":"Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B. Jablin, Jeff Jiao, Tom St. John, Pankaj Kanwar, David Lee, Jeffery Liao, Anton Lokhmotov, Francisco Massa, Peng Meng, Paulius Micikevicius, Colin Osborne, Gennady Pekhimenko, Arun Tejusve Raghunath Rajan, Dilip Sequeira, Ashish Sirasao, Fei Sun, Hanlin Tang, Michael Thomson, Frank Wei, Ephrem Wu, Lingjie Xu, Koichi Yamada, Bing Yu, George Yuan, Aaron Zhong, Peizhao Zhang, and Yuchen Zhou. MLPerf Inference Benchmark. In ISCA, pages 446--459. IEEE, 2020."},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732951.2732955"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415556"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICTAI.2004.50"},{"key":"e_1_2_2_63_1","first-page":"166","volume-title":"Haystack: Detecting Knee Points in System Behavior. In ICDCS Workshops","author":"Satopaa Ville","year":"2011","unstructured":"Ville Satopaa, Jeannie R. Albrecht, David E. Irwin, and Barath Raghavan. Finding a \"Kneedle\" in a Haystack: Detecting Knee Points in System Behavior. In ICDCS Workshops, pages 166--171. IEEE Computer Society, 2011."},{"key":"e_1_2_2_64_1","volume-title":"The Analysis of Variance","author":"Scheffe Henry","year":"1999","unstructured":"Henry Scheffe. The Analysis of Variance, volume 72. John Wiley & Sons, 1999."},{"key":"e_1_2_2_65_1","unstructured":"Silo Source Code. https:\/\/github.com\/stephentu\/silo."},{"key":"e_1_2_2_66_1","unstructured":"Standard Performance Evaluation Corporation. https:\/\/www.spec.org\/."},{"key":"e_1_2_2_67_1","unstructured":"Star Source Code. https:\/\/github.com\/luyi0619\/star."},{"key":"e_1_2_2_68_1","volume-title":"An Analysis of Concurrency Control Protocols for In-Memory Databases with CCBench. CoRR, abs\/2009.11558","author":"Tanabe Takayuki","year":"2020","unstructured":"Takayuki Tanabe, Takashi Hoshino, Hideyuki Kawashima, and Osamu Tatebe. An Analysis of Concurrency Control Protocols for In-Memory Databases with CCBench. CoRR, abs\/2009.11558, 2020."},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2017.115"},{"key":"e_1_2_2_70_1","first-page":"1","volume-title":"Abadi. Calvin: Fast Distributed Transactions for Partitioned Database Systems. In SIGMOD Conference","author":"Thomson Alexander","year":"2012","unstructured":"Alexander Thomson, Thaddeus Diamond, Shu-Chun Weng, Kun Ren, Philip Shao, and Daniel J. Abadi. Calvin: Fast Distributed Transactions for Partitioned Database Systems. In SIGMOD Conference, pages 1--12. ACM, 2012."},{"key":"e_1_2_2_71_1","unstructured":"Transaction Processing Performance Council. The TPC-C home page. http:\/\/www.tpc.org\/tpcc\/."},{"key":"e_1_2_2_72_1","unstructured":"Transaction Processing Performance Council. The TPC home page. http:\/\/www.tpc.org."},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/2517349.2522713"},{"key":"e_1_2_2_74_1","first-page":"513","volume-title":"NSDI","author":"Uta Alexandru","year":"2020","unstructured":"Alexandru Uta, Alexandru Custura, Dmitry Duplyakin, Ivo Jimenez, Jan S. Rellermeyer, Carlos Maltzahn, Robert Ricci, and Alexandru Iosup. Is Big Data Performance Reproducible in Modern Cloud Networks? In NSDI, pages 513--527. USENIX Association, 2020."},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.14778\/3523210.3523221"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2017.11.004"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815400.2815419"},{"key":"e_1_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.14778\/3067421.3067427"},{"key":"e_1_2_2_79_1","first-page":"191","volume-title":"OSDI","author":"Yang Juncheng","year":"2020","unstructured":"Juncheng Yang, Yao Yue, and K. V. Rashmi. A Large-scale Analysis of Hundreds of In-Memory Cache Clusters at Twitter. In OSDI, pages 191--208. USENIX Association, 2020."},{"key":"e_1_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173162.3173187"},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815400.2815404"}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3677137","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3677137","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,31]],"date-time":"2026-03-31T17:10:42Z","timestamp":1774977042000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3677137"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,30]]},"references-count":81,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,10,1]]}},"alternative-id":["10.1145\/3677137"],"URL":"https:\/\/doi.org\/10.1145\/3677137","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,30]]}}}