{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,12]],"date-time":"2026-10-12T12:56:23Z","timestamp":1791809783629,"version":"4.3.4"},"reference-count":133,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,4,26]],"date-time":"2020-04-26T00:00:00Z","timestamp":1587859200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Finnish Academy Project","award":["310321"],"award-info":[{"award-number":["310321"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2021,3,31]]},"abstract":"<jats:p>Big data processing systems (e.g., Hadoop, Spark, Storm) contain a vast number of configuration parameters controlling parallelism, I\/O behavior, memory settings, and compression. Improper parameter settings can cause significant performance degradation and stability issues. However, regular users and even expert administrators grapple with understanding and tuning them to achieve good performance. We investigate existing approaches on parameter tuning for both batch and stream data processing systems and classify them into six categories: rule-based, cost modeling, simulation-based, experiment-driven, machine learning, and adaptive tuning. We summarize the pros and cons of each approach and raise some open research problems for automatic parameter tuning.<\/jats:p>","DOI":"10.1145\/3381027","type":"journal-article","created":{"date-parts":[[2020,5,4]],"date-time":"2020-05-04T15:58:08Z","timestamp":1588607888000},"page":"1-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":93,"title":["A Survey on Automatic Parameter Tuning for Big Data Processing Systems"],"prefix":"10.1145","volume":"53","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8717-1691","authenticated-orcid":false,"given":"Herodotos","family":"Herodotou","sequence":"first","affiliation":[{"name":"Cyprus University of Technology, Limassol, Cyprus"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6220-2535","authenticated-orcid":false,"given":"Yuxing","family":"Chen","sequence":"additional","affiliation":[{"name":"University of Helsinki, Helsinki, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2067-454X","authenticated-orcid":false,"given":"Jiaheng","family":"Lu","sequence":"additional","affiliation":[{"name":"University of Helsinki, Helsinki, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,4,26]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Storm Applied: Strategies for Real-time Event Processing","author":"Allen Sean T.","year":"2015","unstructured":"Sean T. Allen , Matthew Jankowski , and Peter Pathirana . 2015 . Storm Applied: Strategies for Real-time Event Processing . Manning Publications Co. Sean T. Allen, Matthew Jankowski, and Peter Pathirana. 2015. Storm Applied: Strategies for Real-time Event Processing. Manning Publications Co."},{"key":"e_1_2_1_2_1","unstructured":"AMD Hadoop Tuning. 2012. AMD Hadoop Performance Tuning Guide. Retrieved from https:\/\/developer.amd.com\/wordpress\/media\/2012\/10\/Hadoop_Tuning_Guide-Version5.pdf.  AMD Hadoop Tuning. 2012. AMD Hadoop Performance Tuning Guide. Retrieved from https:\/\/developer.amd.com\/wordpress\/media\/2012\/10\/Hadoop_Tuning_Guide-Version5.pdf."},{"key":"e_1_2_1_3_1","unstructured":"Apache Flink. 2019. Apache Flink. Retrieved from https:\/\/flink.apache.org\/.  Apache Flink. 2019. Apache Flink. Retrieved from https:\/\/flink.apache.org\/."},{"key":"e_1_2_1_4_1","unstructured":"Apache Hadoop. 2019. Apache Hadoop. Retrieved from https:\/\/hadoop.apache.org\/.  Apache Hadoop. 2019. Apache Hadoop. Retrieved from https:\/\/hadoop.apache.org\/."},{"key":"e_1_2_1_5_1","unstructured":"Apache Samza. 2019. Apache Samza. Retrieved from http:\/\/samza.apache.org\/.  Apache Samza. 2019. Apache Samza. Retrieved from http:\/\/samza.apache.org\/."},{"key":"e_1_2_1_6_1","unstructured":"ApacheSpark. 2019. Apache Spark. Retrieved from https:\/\/spark.apache.org\/.  ApacheSpark. 2019. Apache Spark. Retrieved from https:\/\/spark.apache.org\/."},{"key":"e_1_2_1_7_1","unstructured":"Apache Spark Streaming. 2019. Apache Spark Streaming. Retrieved from https:\/\/spark.apache.org\/streaming\/.  Apache Spark Streaming. 2019. Apache Spark Streaming. Retrieved from https:\/\/spark.apache.org\/streaming\/."},{"key":"e_1_2_1_8_1","unstructured":"Apache Spark Tuning. 2017. Apache Spark Tuning - DZone. Retrieved from https:\/\/dzone.com\/articles\/apache-spark-performance-tuning-degree-of-parallel.  Apache Spark Tuning. 2017. Apache Spark Tuning - DZone. Retrieved from https:\/\/dzone.com\/articles\/apache-spark-performance-tuning-degree-of-parallel."},{"key":"e_1_2_1_9_1","unstructured":"Apache Spark Tuning Course. 2018. Apache Spark Tuning and Best Practices. Retrieved from https:\/\/databricks.com\/training-overview\/instructor-led-training\/courses\/apache-spark-tuning-and-best-practices.  Apache Spark Tuning Course. 2018. Apache Spark Tuning and Best Practices. Retrieved from https:\/\/databricks.com\/training-overview\/instructor-led-training\/courses\/apache-spark-tuning-and-best-practices."},{"key":"e_1_2_1_10_1","unstructured":"Apache Spark Tuning Guide. 2019. Apache Spark Tuning Guide. Retrieved from https:\/\/spark.apache.org\/docs\/latest\/tuning.html.  Apache Spark Tuning Guide. 2019. Apache Spark Tuning Guide. Retrieved from https:\/\/spark.apache.org\/docs\/latest\/tuning.html."},{"key":"e_1_2_1_11_1","unstructured":"Apache Storm. 2019. Apache Storm. Retrieved from https:\/\/storm.apache.org\/.  Apache Storm. 2019. Apache Storm. Retrieved from https:\/\/storm.apache.org\/."},{"key":"e_1_2_1_12_1","unstructured":"Apache Storm Performance Tuning. 2019. Apache Storm Performance Tuning. Retrieved from https:\/\/storm.apache.org\/releases\/current\/Performance.html.  Apache Storm Performance Tuning. 2019. Apache Storm Performance Tuning. Retrieved from https:\/\/storm.apache.org\/releases\/current\/Performance.html."},{"key":"e_1_2_1_13_1","unstructured":"Apache Storm Trident. 2019. Apache Storm Trident. Retrieved from http:\/\/storm.apache.org\/releases\/current\/Trident-tutorial.html.  Apache Storm Trident. 2019. Apache Storm Trident. Retrieved from http:\/\/storm.apache.org\/releases\/current\/Trident-tutorial.html."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the ACM International Conference on Management of Data (SIGMOD\u201915)","author":"Armbrust Michael","unstructured":"Michael Armbrust , Reynold S. Xin , Cheng Lian , Yin Huai , Davies Liu , Joseph K. Bradley , Xiangrui Meng et al. 2015. Spark SQL: Relational data processing in Spark . In Proceedings of the ACM International Conference on Management of Data (SIGMOD\u201915) . ACM, 1383--1394. Michael Armbrust, Reynold S. Xin, Cheng Lian, Yin Huai, Davies Liu, Joseph K. Bradley, Xiangrui Meng et al. 2015. Spark SQL: Relational data processing in Spark. In Proceedings of the ACM International Conference on Management of Data (SIGMOD\u201915). ACM, 1383--1394."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807150"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1561\/1900000036"},{"key":"e_1_2_1_17_1","volume-title":"Trevor: Automatic configuration and scaling of stream processing pipelines. CoRR abs\/1812.09442","author":"Bansal Manu","year":"2018","unstructured":"Manu Bansal , Eyal Cidon , Arjun Balasingam , Aditya Gudipati , Christos Kozyrakis , and Sachin Katti . 2018 . Trevor: Automatic configuration and scaling of stream processing pipelines. CoRR abs\/1812.09442 (2018). Manu Bansal, Eyal Cidon, Arjun Balasingam, Aditya Gudipati, Christos Kozyrakis, and Sachin Katti. 2018. Trevor: Automatic configuration and scaling of stream processing pipelines. CoRR abs\/1812.09442 (2018)."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2018.8622018"},{"key":"e_1_2_1_19_1","volume-title":"Real-time Big Data Analytics: Emerging Architecture. O\u2019Reilly Media","author":"Barlow Mike","unstructured":"Mike Barlow . 2013. Real-time Big Data Analytics: Emerging Architecture. O\u2019Reilly Media , Inc . Mike Barlow. 2013. Real-time Big Data Analytics: Emerging Architecture. O\u2019Reilly Media, Inc."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2479871.2479895"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2015.2449299"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3127479.3127492"},{"key":"e_1_2_1_23_1","unstructured":"BTrace. 2018. BTrace: A Dynamic Instrumentation Tool for Java. Retrieved from https:\/\/github.com\/btraceio\/btrace.  BTrace. 2018. BTrace: A Dynamic Instrumentation Tool for Java. Retrieved from https:\/\/github.com\/btraceio\/btrace."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.710"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigDataCongress.2015.64"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2013.297"},{"key":"e_1_2_1_27_1","volume-title":"d-Simplexed: Adaptive Delaunay triangulation for performance modeling and prediction on big data analytics","author":"Chen Yuxing","year":"2019","unstructured":"Yuxing Chen , Peter Goetsch , Mohammad A. Hoque , Jiaheng Lu , and Sasu Tarkoma . 2019. d-Simplexed: Adaptive Delaunay triangulation for performance modeling and prediction on big data analytics . IEEE Trans. Big Data ( 2019 ). https:\/\/ieeexplore.ieee.org\/document\/8878273. Yuxing Chen, Peter Goetsch, Mohammad A. Hoque, Jiaheng Lu, and Sasu Tarkoma. 2019. d-Simplexed: Adaptive Delaunay triangulation for performance modeling and prediction on big data analytics. IEEE Trans. Big Data (2019). https:\/\/ieeexplore.ieee.org\/document\/8878273."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3358090"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2663165.2666089"},{"key":"e_1_2_1_30_1","unstructured":"ClouderaSparkTuning. 2018. Cloudera Performance Management - Tuning Spark Applications. Retrieved from https:\/\/www.cloudera.com\/documentation\/enterprise\/5-9-x\/topics\/admin_spark_tuning.html.  ClouderaSparkTuning. 2018. Cloudera Performance Management - Tuning Spark Applications. Retrieved from https:\/\/www.cloudera.com\/documentation\/enterprise\/5-9-x\/topics\/admin_spark_tuning.html."},{"key":"e_1_2_1_31_1","unstructured":"ClouderaYarnTuning. 2018. Cloudera Performance Management - Tuning YARN. Retrieved from https:\/\/www.cloudera.com\/documentation\/enterprise\/5-8-x\/topics\/cdh_ig_yarn_tuning.html.  ClouderaYarnTuning. 2018. Cloudera Performance Management - Tuning YARN. Retrieved from https:\/\/www.cloudera.com\/documentation\/enterprise\/5-8-x\/topics\/cdh_ig_yarn_tuning.html."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2670979.2670995"},{"key":"e_1_2_1_33_1","unstructured":"Databricks. 2019. Databricks. Retrieved from https:\/\/sparkhub.databricks.com\/.  Databricks. 2019. Databricks. Retrieved from https:\/\/sparkhub.databricks.com\/."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3170432"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1327452.1327492"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 21st International Conference on Parallel and Distributed Systems. IEEE, 831--836","author":"Ding Xiaoan","year":"2015","unstructured":"Xiaoan Ding , Yi Liu , and Depei Qian . 2015 . Jellyfish: Online performance tuning with adaptive configuration and elastic container in Hadoop YARN . In Proceedings of the 21st International Conference on Parallel and Distributed Systems. IEEE, 831--836 . Xiaoan Ding, Yi Liu, and Depei Qian. 2015. Jellyfish: Online performance tuning with adaptive configuration and elastic container in Hadoop YARN. In Proceedings of the 21st International Conference on Parallel and Distributed Systems. IEEE, 831--836."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TBDATA.2017.2723473"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-013-0319-9"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687627.1687767"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the 17th International Conference on Extending Database Technology (EDBT\u201914)","author":"Ead Mostafa","year":"2014","unstructured":"Mostafa Ead , Herodotos Herodotou , Ashraf Aboulnaga , and Shivnath Babu . 2014 . PStorM: Profile storage and matching for feedback-based tuning of MapReduce jobs . In Proceedings of the 17th International Conference on Extending Database Technology (EDBT\u201914) . 1--12. Mostafa Ead, Herodotos Herodotou, Ashraf Aboulnaga, and Shivnath Babu. 2014. PStorM: Profile storage and matching for feedback-based tuning of MapReduce jobs. In Proceedings of the 17th International Conference on Extending Database Technology (EDBT\u201914). 1--12."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTER.2015.13"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137765.3137786"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2015.49"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICECCT.2015.7226035"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the International Conference on High Performance Computing and Communications (HPCC\u201916)","author":"Genkin Mikhail","unstructured":"Mikhail Genkin , Frank Dehne et al. 2016. Automatic, on-line tuning of YARN container memory and CPU parameters . In Proceedings of the International Conference on High Performance Computing and Communications (HPCC\u201916) . IEEE, 317--324. Mikhail Genkin, Frank Dehne et al. 2016. Automatic, on-line tuning of YARN container memory and CPU parameters. In Proceedings of the International Conference on High Performance Computing and Communications (HPCC\u201916). IEEE, 317--324."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2017.2647939"},{"key":"e_1_2_1_47_1","volume-title":"A methodology for Spark parameter tuning. Big Data Res. 11 (Mar","author":"Gounaris Anastasios","year":"2017","unstructured":"Anastasios Gounaris and Jordi Torres . 2017. A methodology for Spark parameter tuning. Big Data Res. 11 (Mar . 2017 ), 22--32. Anastasios Gounaris and Jordi Torres. 2017. A methodology for Spark parameter tuning. Big Data Res. 11 (Mar. 2017), 22--32."},{"key":"e_1_2_1_48_1","unstructured":"HadoopClusterSetup. 2019. Hadoop Cluster Setup. Retrieved from https:\/\/hadoop.apache.org\/docs\/r1.2.1\/cluster_setup.html.  HadoopClusterSetup. 2019. Hadoop Cluster Setup. Retrieved from https:\/\/hadoop.apache.org\/docs\/r1.2.1\/cluster_setup.html."},{"key":"e_1_2_1_49_1","unstructured":"HadoopPerfUI. 2011. Hadoop Perf Monitoring UI. Retrieved from http:\/\/code.google.com\/p\/hadoop-toolkit\/wiki\/HadoopPerformanceMonitoring.  HadoopPerfUI. 2011. Hadoop Perf Monitoring UI. Retrieved from http:\/\/code.google.com\/p\/hadoop-toolkit\/wiki\/HadoopPerformanceMonitoring."},{"key":"e_1_2_1_50_1","unstructured":"HadoopTuning. 2015. Hadoop Performance Tuning Tutorial. Retrieved from http:\/\/hadooptutorial.info\/hadoop-performance-tuning\/.  HadoopTuning. 2015. Hadoop Performance Tuning Tutorial. Retrieved from http:\/\/hadooptutorial.info\/hadoop-performance-tuning\/."},{"key":"e_1_2_1_51_1","unstructured":"HadoopTutorial. 2018. Hadoop MapReduce Tutorial. Retrieved from https:\/\/hadoop.apache.org\/docs\/stable\/hadoop-mapreduce-client\/hadoop-mapreduce-client-core\/MapReduceTutorial.html.  HadoopTutorial. 2018. Hadoop MapReduce Tutorial. Retrieved from https:\/\/hadoop.apache.org\/docs\/stable\/hadoop-mapreduce-client\/hadoop-mapreduce-client-core\/MapReduceTutorial.html."},{"key":"e_1_2_1_52_1","unstructured":"HadoopVaidya. 2011. Hadoop Vaidya. Retrieved from http:\/\/hadoop.apache.org\/mapreduce\/docs\/r0.21.0\/vaidya.html.  HadoopVaidya. 2011. Hadoop Vaidya. Retrieved from http:\/\/hadoop.apache.org\/mapreduce\/docs\/r0.21.0\/vaidya.html."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/FSKD.2010.5569086"},{"key":"e_1_2_1_54_1","unstructured":"Dominique Heger. 2013. Hadoop Performance Tuning\u2014A Pragmatic 8 Iterative Approach. Retrieved from https:\/\/www.cmg.org\/wp-content\/uploads\/2013\/04\/m_97_3.pdf.  Dominique Heger. 2013. Hadoop Performance Tuning\u2014A Pragmatic 8 Iterative Approach. Retrieved from https:\/\/www.cmg.org\/wp-content\/uploads\/2013\/04\/m_97_3.pdf."},{"issue":"2018","key":"e_1_2_1_55_1","first-page":"1076","article-title":"Using machine learning to optimize parallelism in big data applications","volume":"86","author":"Hern\u00e1ndez \u00c1lvaro Brand\u00f3n","year":"2017","unstructured":"\u00c1lvaro Brand\u00f3n Hern\u00e1ndez , Mar\u00eda S. Perez , Smrati Gupta , and Victor Munt\u00e9s-Mulero . 2017 . Using machine learning to optimize parallelism in big data applications . Fut. Gen. Comput. Syst. 86 ( 2018 ), 1076 \u2013 1092 . https:\/\/www.sciencedirect.com\/science\/article\/abs\/pii\/S0167739X17314668?via%3Dihub. \u00c1lvaro Brand\u00f3n Hern\u00e1ndez, Mar\u00eda S. Perez, Smrati Gupta, and Victor Munt\u00e9s-Mulero. 2017. Using machine learning to optimize parallelism in big data applications. Fut. Gen. Comput. Syst. 86 (2018), 1076\u20131092. https:\/\/www.sciencedirect.com\/science\/article\/abs\/pii\/S0167739X17314668?via%3Dihub.","journal-title":"Fut. Gen. Comput. Syst."},{"key":"e_1_2_1_56_1","volume-title":"Hadoop performance models. CoRR abs\/1106.0940","author":"Herodotou Herodotos","year":"2011","unstructured":"Herodotos Herodotou . 2011. Hadoop performance models. CoRR abs\/1106.0940 ( 2011 ). Herodotos Herodotou. 2011. Hadoop performance models. CoRR abs\/1106.0940 (2011)."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.14778\/3402707.3402746"},{"key":"e_1_2_1_59_1","first-page":"5","article-title":"A what-if engine for cost-based MapReduce optimization","volume":"36","author":"Herodotou Herodotos","year":"2013","unstructured":"Herodotos Herodotou and Shivnath Babu . 2013 . A what-if engine for cost-based MapReduce optimization . IEEE Data Eng. Bull. 36 , 1 (2013), 5 -- 14 . Herodotos Herodotou and Shivnath Babu. 2013. A what-if engine for cost-based MapReduce optimization. IEEE Data Eng. Bull. 36, 1 (2013), 5--14.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2038916.2038934"},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the 5th Biennial Conference on Innovative Data Systems Research (CIDR\u201911)","author":"Herodotou Herodotos","year":"2011","unstructured":"Herodotos Herodotou , Harold Lim , Gang Luo , Nedyalko Borisov , Liang Dong , Fatma Bilgen Cetin , and Shivnath Babu . 2011 . Starfish: A self-tuning system for big data analytics . In Proceedings of the 5th Biennial Conference on Innovative Data Systems Research (CIDR\u201911) . 261--272. Herodotos Herodotou, Harold Lim, Gang Luo, Nedyalko Borisov, Liang Dong, Fatma Bilgen Cetin, and Shivnath Babu. 2011. Starfish: A self-tuning system for big data analytics. In Proceedings of the 5th Biennial Conference on Innovative Data Systems Research (CIDR\u201911). 261--272."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2015.10.023"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2528412"},{"key":"e_1_2_1_64_1","first-page":"51","article-title":"SimJava: A discrete event simulation library for Java","volume":"30","author":"Howell Fred","year":"1998","unstructured":"Fred Howell and Ross McNab . 1998 . SimJava: A discrete event simulation library for Java . Simul. Series 30 (1998), 51 -- 56 . Fred Howell and Ross McNab. 1998. SimJava: A discrete event simulation library for Java. Simul. Series 30 (1998), 51--56.","journal-title":"Simul. Series"},{"key":"e_1_2_1_65_1","volume-title":"McCann","author":"Huebscher Markus C.","year":"2008","unstructured":"Markus C. Huebscher and Julie A . McCann . 2008 . A survey of autonomic computing\u2014degrees, models, and applications. Comput. Surv . 40, 3 (2008), 7:1\u20137:28. Markus C. Huebscher and Julie A. McCann. 2008. A survey of autonomic computing\u2014degrees, models, and applications. Comput. Surv. 40, 3 (2008), 7:1\u20137:28."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOTS.2016.17"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2967938.2967957"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920903"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3035933"},{"key":"e_1_2_1_70_1","volume-title":"Proceedings of the 21st International Conference on Computer Communications and Networks (ICCCN\u201912)","author":"Kadirvel Selvi","unstructured":"Selvi Kadirvel and Jos\u00e9 A. B. Fortes . 2012. Grey-box approach for performance prediction in MapReduce based platforms . In Proceedings of the 21st International Conference on Computer Communications and Networks (ICCCN\u201912) . IEEE, 1--9. Selvi Kadirvel and Jos\u00e9 A. B. Fortes. 2012. Grey-box approach for performance prediction in MapReduce based platforms. In Proceedings of the 21st International Conference on Computer Communications and Networks (ICCCN\u201912). IEEE, 1--9."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2019.00204"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCGRID.2010.112"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2015.2405552"},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the IEEE\/ACM International Symposium on Modeling, Analysis, and Simulation on Computer and Telecommunication Systems (MASCOTS\u201917)","author":"Kro\u00df Johannes","year":"2017","unstructured":"Johannes Kro\u00df and Helmut Krcmar . 2017 . Model-based performance evaluation of batch and stream applications for big data . In Proceedings of the IEEE\/ACM International Symposium on Modeling, Analysis, and Simulation on Computer and Telecommunication Systems (MASCOTS\u201917) . IEEE Computer Society, 80--86. Johannes Kro\u00df and Helmut Krcmar. 2017. Model-based performance evaluation of batch and stream applications for big data. In Proceedings of the IEEE\/ACM International Symposium on Modeling, Analysis, and Simulation on Computer and Telecommunication Systems (MASCOTS\u201917). IEEE Computer Society, 80--86."},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742788"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/2371536.2371547"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094114.2094118"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600212.2600229"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1109\/TBDATA.2016.2616148"},{"key":"e_1_2_1_80_1","volume-title":"Willke","author":"Liao Guangdeng","year":"2013","unstructured":"Guangdeng Liao , Kushal Datta , and Theodore L . Willke . 2013 . Gunther : Search-based auto-tuning of MapReduce. In Proceedings of the European Conference on Parallel Processing (Euro-Par\u201913). Springer , 406--419. Guangdeng Liao, Kushal Datta, and Theodore L. Willke. 2013. Gunther: Search-based auto-tuning of MapReduce. In Proceedings of the European Conference on Parallel Processing (Euro-Par\u201913). Springer, 406--419."},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/AINA.2018.00068"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-49665-7_4"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1109\/ClusterW.2012.24"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-27140-8_24"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/2259016.2259022"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132618"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2011.05.007"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.14778\/3352063.3352112"},{"key":"e_1_2_1_89_1","volume-title":"Spark GraphX in Action","author":"Malak Michael","unstructured":"Michael Malak and Robin East . 2016. Spark GraphX in Action . Manning Publications Co. Michael Malak and Robin East. 2016. Spark GraphX in Action. Manning Publications Co."},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.5555\/2946645.2946679"},{"key":"e_1_2_1_91_1","unstructured":"Matt Morgan. 2015. Ensuring the Best Performance from Your Hadoop Clusters Proactively. Retrieved from https:\/\/hortonworks.com\/blog\/ensuring-the-best-performance-from-your-hadoop-clusters-proactively\/.  Matt Morgan. 2015. Ensuring the Best Performance from Your Hadoop Clusters Proactively. Retrieved from https:\/\/hortonworks.com\/blog\/ensuring-the-best-performance-from-your-hadoop-clusters-proactively\/."},{"key":"e_1_2_1_92_1","volume-title":"Mumak: Map-Reduce Simulator.","year":"2010","unstructured":"Mumak. 2010 . Mumak: Map-Reduce Simulator. Retrieved from https:\/\/issues.apache.org\/jira\/browse\/MAPREDUCE-728. Mumak. 2010. Mumak: Map-Reduce Simulator. Retrieved from https:\/\/issues.apache.org\/jira\/browse\/MAPREDUCE-728."},{"key":"e_1_2_1_93_1","volume-title":"Of streams and storms. IBM White Paper","author":"Nabi Zubair","year":"2014","unstructured":"Zubair Nabi , Eric Bouillet , Andrew Bainbridge , and Chris Thomas . 2014. Of streams and storms. IBM White Paper ( 2014 ), 1--31. Zubair Nabi, Eric Bouillet, Andrew Bainbridge, and Chris Thomas. 2014. Of streams and storms. IBM White Paper (2014), 1--31."},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/1559845.1559865"},{"key":"e_1_2_1_95_1","volume-title":"Proceedings of the INNS Conference on Big Data. Springer, 226--237","author":"Petridis Panagiotis","year":"2016","unstructured":"Panagiotis Petridis , Anastasios Gounaris , and Jordi Torres . 2016 . Spark parameter tuning via trial-and-error . In Proceedings of the INNS Conference on Big Data. Springer, 226--237 . Panagiotis Petridis, Anastasios Gounaris, and Jordi Torres. 2016. Spark parameter tuning via trial-and-error. In Proceedings of the INNS Conference on Big Data. Springer, 226--237."},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2018.08.243"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1109\/NOMS.2010.5488494"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1109\/IRI.2017.64"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1145\/3303849"},{"key":"e_1_2_1_100_1","unstructured":"Rumen 2009. Rumen: A Tool to Extract Job Characterization Data from Job Tracker Logs. Retrieved from https:\/\/issues.apache.org\/jira\/browse\/MAPREDUCE-751.  Rumen 2009. Rumen: A Tool to Extract Job Characterization Data from Job Tracker Logs. Retrieved from https:\/\/issues.apache.org\/jira\/browse\/MAPREDUCE-751."},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDEW.2013.6547428"},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.14778\/2733004.2733005"},{"key":"e_1_2_1_103_1","volume-title":"Proceedings of the Technology Conference on Performance Evaluation and Benchmarking (TPCTC\u201917)","author":"Singhal Rekha","year":"2017","unstructured":"Rekha Singhal and Praveen Singh . 2017 . Performance assurance model for applications on SPARK platform . In Proceedings of the Technology Conference on Performance Evaluation and Benchmarking (TPCTC\u201917) . Springer, 131--146. Rekha Singhal and Praveen Singh. 2017. Performance assurance model for applications on SPARK platform. In Proceedings of the Technology Conference on Performance Evaluation and Benchmarking (TPCTC\u201917). Springer, 131--146."},{"key":"e_1_2_1_104_1","unstructured":"SparkCoreParameter 2019. Spark Core Parameters. Retrieved from https:\/\/spark.apache.org\/docs\/latest\/configuration.html.  SparkCoreParameter 2019. Spark Core Parameters. Retrieved from https:\/\/spark.apache.org\/docs\/latest\/configuration.html."},{"key":"e_1_2_1_105_1","volume-title":"Int. J. Parallel Emerg. Distrib. Syst.","author":"Tantalaki Nicoleta","year":"2019","unstructured":"Nicoleta Tantalaki , Stavros Souravlas , and Manos Roumeliotis . 2019. A review on big data real-time stream processing and its scheduling techniques . Int. J. Parallel Emerg. Distrib. Syst. ( 2019 ), 1--31. https:\/\/www.tandfonline.com\/doi\/abs\/10.1080\/17445760.2019.1585848. Nicoleta Tantalaki, Stavros Souravlas, and Manos Roumeliotis. 2019. A review on big data real-time stream processing and its scheduling techniques. Int. J. Parallel Emerg. Distrib. Syst. (2019), 1--31. https:\/\/www.tandfonline.com\/doi\/abs\/10.1080\/17445760.2019.1585848."},{"key":"e_1_2_1_106_1","doi-asserted-by":"publisher","DOI":"10.1109\/MUE.2011.56"},{"key":"e_1_2_1_107_1","unstructured":"TheNS2. 2011. The Network Simulator - ns-2. Retrieved from https:\/\/www.isi.edu\/nsnam\/ns\/.  TheNS2. 2011. The Network Simulator - ns-2. Retrieved from https:\/\/www.isi.edu\/nsnam\/ns\/."},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1109\/FAS-W.2017.144"},{"key":"e_1_2_1_109_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICAC.2019.00025"},{"key":"e_1_2_1_110_1","volume-title":"Vaquero and F\u00e9lix Cuadrado","author":"Luis","year":"2018","unstructured":"Luis M. Vaquero and F\u00e9lix Cuadrado . 2018 . Auto-tuning distributed stream processing systems using reinforcement learning. CoRR abs\/1809.05495 (2018). Luis M. Vaquero and F\u00e9lix Cuadrado. 2018. Auto-tuning distributed stream processing systems using reinforcement learning. CoRR abs\/1809.05495 (2018)."},{"key":"e_1_2_1_111_1","doi-asserted-by":"publisher","DOI":"10.1145\/2523616.2523633"},{"key":"e_1_2_1_112_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132750"},{"key":"e_1_2_1_113_1","volume-title":"Proceedings of the 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201916)","author":"Venkataraman Shivaram","year":"2016","unstructured":"Shivaram Venkataraman , Zongheng Yang , Michael J. Franklin , Benjamin Recht , and Ion Stoica . 2016 . Ernest: Efficient performance prediction for large-scale advanced analytics . In Proceedings of the 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201916) . USENIX Association, 363--378. Shivaram Venkataraman, Zongheng Yang, Michael J. Franklin, Benjamin Recht, and Ion Stoica. 2016. Ernest: Efficient performance prediction for large-scale advanced analytics. In Proceedings of the 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201916). USENIX Association, 363--378."},{"key":"e_1_2_1_114_1","volume-title":"Proceedings of the 8th International Conference on Autonomic Computing (ICAC\u201911)","author":"Verma Abhishek","year":"1998","unstructured":"Abhishek Verma , Ludmila Cherkasova , and Roy H. Campbell . 2011. ARIA: Automatic resource inference and allocation for MapReduce environments . In Proceedings of the 8th International Conference on Autonomic Computing (ICAC\u201911) . ACM, 235--244. DOI:https:\/\/doi.org\/10.1145\/ 1998 582.1998637 10.1145\/1998582.1998637 Abhishek Verma, Ludmila Cherkasova, and Roy H. Campbell. 2011. ARIA: Automatic resource inference and allocation for MapReduce environments. In Proceedings of the 8th International Conference on Autonomic Computing (ICAC\u201911). ACM, 235--244. DOI:https:\/\/doi.org\/10.1145\/1998582.1998637"},{"key":"e_1_2_1_115_1","volume-title":"Campbell","author":"Verma Abhishek","year":"2011","unstructured":"Abhishek Verma , Ludmila Cherkasova , and Roy H . Campbell . 2011 . Play it again, SimMR! In Proceedings of the IEEE International Conference on Cluster Computing (CLUSTER\u201911). IEEE , 253--261. Abhishek Verma, Ludmila Cherkasova, and Roy H. Campbell. 2011. Play it again, SimMR! In Proceedings of the IEEE International Conference on Cluster Computing (CLUSTER\u201911). IEEE, 253--261."},{"key":"e_1_2_1_116_1","volume-title":"Campbell","author":"Verma Abhishek","year":"2011","unstructured":"Abhishek Verma , Ludmila Cherkasova , and Roy H . Campbell . 2011 . Resource provisioning framework for MapReduce jobs with performance goals. In Proceedings of the ACM\/IFIP\/USENIX 12th International Middleware Conference. Springer , 165--186. Abhishek Verma, Ludmila Cherkasova, and Roy H. Campbell. 2011. Resource provisioning framework for MapReduce jobs with performance goals. In Proceedings of the ACM\/IFIP\/USENIX 12th International Middleware Conference. Springer, 165--186."},{"key":"e_1_2_1_117_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2017.2751606"},{"key":"e_1_2_1_118_1","volume-title":"Proceedings of the IEEE\/ACM International Symposium on Modeling, Analysis, and Simulation on Computer and Telecommunication Systems (MASCOTS\u201909)","author":"Wang Guanying","year":"2009","unstructured":"Guanying Wang , Ali R. Butt , Prashant Pandey , and Karan Gupta . 2009 . A simulation approach to evaluating design decisions in MapReduce setups . In Proceedings of the IEEE\/ACM International Symposium on Modeling, Analysis, and Simulation on Computer and Telecommunication Systems (MASCOTS\u201909) . IEEE, 1--11. Guanying Wang, Ali R. Butt, Prashant Pandey, and Karan Gupta. 2009. A simulation approach to evaluating design decisions in MapReduce setups. In Proceedings of the IEEE\/ACM International Symposium on Modeling, Analysis, and Simulation on Computer and Telecommunication Systems (MASCOTS\u201909). IEEE, 1--11."},{"key":"e_1_2_1_119_1","doi-asserted-by":"publisher","DOI":"10.1109\/HPCC-SmartCity-DSS.2016.0088"},{"key":"e_1_2_1_120_1","doi-asserted-by":"publisher","DOI":"10.1109\/HPCC-CSS-ICESS.2015.246"},{"key":"e_1_2_1_121_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2012.6427486"},{"key":"e_1_2_1_122_1","unstructured":"Thomas Weise. 2009. Global Optimization Algorithms\u2014Theory and Application. Self-published. http:\/\/www.it-weise.de\/projects\/book.pdf.  Thomas Weise. 2009. Global Optimization Algorithms\u2014Theory and Application. Self-published. http:\/\/www.it-weise.de\/projects\/book.pdf."},{"key":"e_1_2_1_123_1","volume-title":"Hadoop: The Definitive Guide. O\u2019Reilly Media","author":"White Tom","year":"2012","unstructured":"Tom White . 2012 . Hadoop: The Definitive Guide. O\u2019Reilly Media , Inc . Tom White. 2012. Hadoop: The Definitive Guide. O\u2019Reilly Media, Inc."},{"key":"e_1_2_1_124_1","doi-asserted-by":"publisher","DOI":"10.1109\/HiPC.2013.6799133"},{"key":"e_1_2_1_125_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2014.61"},{"key":"e_1_2_1_126_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPSW.2012.254"},{"key":"e_1_2_1_127_1","doi-asserted-by":"publisher","DOI":"10.1145\/781027.781052"},{"key":"e_1_2_1_128_1","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOTS.2013.9"},{"key":"e_1_2_1_129_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173162.3173187"},{"key":"e_1_2_1_130_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2015.7363758"},{"key":"e_1_2_1_131_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2017.8257950"},{"key":"e_1_2_1_132_1","volume-title":"Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation (NSDI\u201912)","author":"Zaharia Matei","year":"2012","unstructured":"Matei Zaharia , Mosharaf Chowdhury , Tathagata Das , Ankur Dave , Justin Ma , Murphy McCauley , Michael J. Franklin , Scott Shenker , and Ion Stoica . 2012 . Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing . In Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation (NSDI\u201912) . USENIX Association, 2--14. Retrieved from http:\/\/dl.acm.org\/citation.cfm?id=2228298.2228301. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2012. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation (NSDI\u201912). USENIX Association, 2--14. Retrieved from http:\/\/dl.acm.org\/citation.cfm?id=2228298.2228301."},{"key":"e_1_2_1_133_1","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.3229"},{"key":"e_1_2_1_134_1","doi-asserted-by":"publisher","DOI":"10.1145\/3127479.3128605"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3381027","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3381027","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:44:58Z","timestamp":1750203898000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3381027"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,26]]},"references-count":133,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,3,31]]}},"alternative-id":["10.1145\/3381027"],"URL":"https:\/\/doi.org\/10.1145\/3381027","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,26]]},"assertion":[{"value":"2019-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-04-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}