{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T03:55:33Z","timestamp":1760586933867,"version":"3.41.0"},"reference-count":30,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2018,9,30]],"date-time":"2018-09-30T00:00:00Z","timestamp":1538265600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1234983"],"award-info":[{"award-number":["1234983"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100015701","name":"AT&T Foundation","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100015701","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Auton. Adapt. Syst."],"published-print":{"date-parts":[[2018,9,30]]},"abstract":"<jats:p>Many big-data processing jobs use data-analytics frameworks such as Apache Hadoop (currently also known as YARN). Such frameworks have tunable configuration parameters set by experienced system administrators and\/or job developers. However, tuning parameters manually can be hard and time-consuming because it requires domain-specific knowledge and understanding of complex inter-dependencies among parameters. Most of the frameworks seek efficient resource management by assigning resource units to jobs, the maximum number of units allowed in a system being part of the static configuration of the system. This static resource management has limited effectiveness in coping with job diversity and workload dynamics, even in the case of a single job. The work reported in this article seeks to improve performance (e.g., multiple-jobs makespan and job completion time) without modification of either the framework or the applications and avoiding problems of previous self-tuning approaches based on performance models or resource usage. These problems include (1) the need for time-consuming training, typically offline and (2) unsuitability for multi-jobs\/tenant environments. This article proposes a hierarchical self-tuning approach using (1) a fuzzy-logic controller to dynamically adjust the maximum number of concurrent jobs and (2) additional controllers (one for each cluster node) to adjust the maximum number of resource units assigned to jobs on each node. The fuzzy-logic controller uses fuzzy rules based on a concave-downward relationship between aggregate CPU usage and the number of concurrent jobs. The other controllers use a heuristic algorithm to adjust the number of resource units on the basis of both CPU and disk IO usage by jobs. To manage the maximum number of available resource units in each node, the controllers also take resource usage by other processes (e.g., system processes) into account. A prototype of our approach was implemented for Apache Hadoop on a cluster running at CloudLab. The proposed approach was demonstrated and evaluated with workloads composed of jobs with similar resource usage patterns as well as other realistic mixed-pattern workloads synthesized by SWIM, a statistical workload injector for MapReduce. The evaluation shows that the proposed approach yields up to a 48% reduction of the jobs makespan that results from using Hadoop-default settings.<\/jats:p>","DOI":"10.1145\/3309539","type":"journal-article","created":{"date-parts":[[2019,3,15]],"date-time":"2019-03-15T12:13:05Z","timestamp":1552651985000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Improving Data-Analytics Performance Via Autonomic Control of Concurrency and Resource Units"],"prefix":"10.1145","volume":"13","author":[{"given":"Gil Jae","family":"Lee","sequence":"first","affiliation":[{"name":"Electrical and Computer Engineering Department, University of Florida, Gainesville, FL, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jos\u00e9 A. B.","family":"Fortes","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering Department, University of Florida, Gainesville, FL, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,3,15]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOTS.2011.12"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2663165.2666089"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1327452.1327492"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPADS.2015.112"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTER.2012.69"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2013.93"},{"key":"e_1_2_1_7_1","volume-title":"Fatma Bilgen Cetin, and Shivnath Babu","author":"Herodotou Herodotos","year":"2011","unstructured":"Herodotos Herodotou , Harold Lim , Gang Luo , Nedyalko Borisov , Liang Dong , Fatma Bilgen Cetin, and Shivnath Babu . 2011 . Starfish : A self-tuning system for big data analytics. In CIDR. 261--272. Herodotos Herodotou, Harold Lim, Gang Luo, Nedyalko Borisov, Liang Dong, Fatma Bilgen Cetin, and Shivnath Babu. 2011. Starfish: A self-tuning system for big data analytics. In CIDR. 261--272."},{"volume-title":"Retrieved on","year":"2009","key":"e_1_2_1_8_1","unstructured":"Impetus. 2009 . Advanced Hadoop Tuning and Optimizations . Retrieved on January 26, 2019 from http:\/\/www.slideshare.net\/impetusinfo\/ppt-on-advancedhadoop-tuning-n-optimisation. Impetus. 2009. Advanced Hadoop Tuning and Optimizations. Retrieved on January 26, 2019 from http:\/\/www.slideshare.net\/impetusinfo\/ppt-on-advancedhadoop-tuning-n-optimisation."},{"volume-title":"Proceedings of the 2012 21st International Conference on Computer Communications and Networks (ICCCN\u201912)","author":"Kadirvel Selvi","key":"e_1_2_1_9_1","unstructured":"Selvi Kadirvel and Jos\u00e9 A. B. Fortes . 2012. Grey-box approach for performance prediction in map-reduce based platforms . In Proceedings of the 2012 21st International Conference on Computer Communications and Networks (ICCCN\u201912) . IEEE, 1--9. Selvi Kadirvel and Jos\u00e9 A. B. Fortes. 2012. Grey-box approach for performance prediction in map-reduce based platforms. In Proceedings of the 2012 21st International Conference on Computer Communications and Networks (ICCCN\u201912). IEEE, 1--9."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2015.49"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2371536.2371547"},{"volume-title":"Proceedings of the 2016 IEEE International Conference on Autonomic Computing (ICAC\u201916)","author":"Lee Gil Jae","key":"e_1_2_1_12_1","unstructured":"Gil Jae Lee and Jos\u00e9 A. B. Fortes . 2016. Hadoop performance self-tuning using a fuzzy-prediction approach . In Proceedings of the 2016 IEEE International Conference on Autonomic Computing (ICAC\u201916) . IEEE, 55--64. Gil Jae Lee and Jos\u00e9 A. B. Fortes. 2016. Hadoop performance self-tuning using a fuzzy-prediction approach. In Proceedings of the 2016 IEEE International Conference on Autonomic Computing (ICAC\u201916). IEEE, 55--64."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600212.2600229"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-40047-6_42"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/NOMS.2010.5488494"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/HPEC.2016.7761604"},{"key":"e_1_2_1_17_1","volume-title":"Introducing CloudLab: Scientific infrastructure for advancing cloud architectures and applications. USENIX 8 SAGE 39, 6","author":"Ricci Robert","year":"2014","unstructured":"Robert Ricci , Eric Eide , and Cloud Lab Team . 2014. Introducing CloudLab: Scientific infrastructure for advancing cloud architectures and applications. USENIX 8 SAGE 39, 6 ( 2014 ), 36--38. Robert Ricci, Eric Eide, and Cloud Lab Team. 2014. Introducing CloudLab: Scientific infrastructure for advancing cloud architectures and applications. USENIX 8 SAGE 39, 6 (2014), 36--38."},{"key":"e_1_2_1_18_1","unstructured":"Hadoop: Capacity Scheduler. Retrieved from http:\/\/hadoop.apache.org\/docs\/current\/hadoop-yarn\/hadoop-yarn-site\/capacityscheduler.html.  Hadoop: Capacity Scheduler. Retrieved from http:\/\/hadoop.apache.org\/docs\/current\/hadoop-yarn\/hadoop-yarn-site\/capacityscheduler.html."},{"key":"e_1_2_1_19_1","unstructured":"Hadoop: Fair Scheduler. Retrieved from http:\/\/hadoop.apache.org\/docs\/current\/hadoop-yarn\/hadoop-yarn-site\/fairscheduler.html.  Hadoop: Fair Scheduler. Retrieved from http:\/\/hadoop.apache.org\/docs\/current\/hadoop-yarn\/hadoop-yarn-site\/fairscheduler.html."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCC.2014.2329299"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2011.14"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0165-0114(97)00337-0"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2523616.2523633"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1998582.1998637"},{"key":"e_1_2_1_26_1","volume-title":"Hadoop: The Definitive Guide","author":"White Tom","year":"2012","unstructured":"Tom White . 2012 . Hadoop: The Definitive Guide . O'Reilly Media, Inc. Tom White. 2012. Hadoop: The Definitive Guide. O'Reilly Media, Inc."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOTS.2013.9"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2015.2449299"},{"key":"e_1_2_1_29_1","volume-title":"Fuzzy sets. Information and control 8, 3","author":"Zadeh Lotfi A.","year":"1965","unstructured":"Lotfi A. Zadeh . 1965. Fuzzy sets. Information and control 8, 3 ( 1965 ), 338--353. Lotfi A. Zadeh. 1965. Fuzzy sets. Information and control 8, 3 (1965), 338--353."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICAC.2015.54"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCC.2015.2415772"}],"container-title":["ACM Transactions on Autonomous and Adaptive Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3309539","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3309539","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3309539","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T17:49:22Z","timestamp":1750268962000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3309539"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,9,30]]},"references-count":30,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2018,9,30]]}},"alternative-id":["10.1145\/3309539"],"URL":"https:\/\/doi.org\/10.1145\/3309539","relation":{},"ISSN":["1556-4665","1556-4703"],"issn-type":[{"type":"print","value":"1556-4665"},{"type":"electronic","value":"1556-4703"}],"subject":[],"published":{"date-parts":[[2018,9,30]]},"assertion":[{"value":"2018-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-03-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}