{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T01:59:11Z","timestamp":1760234351394,"version":"build-2065373602"},"reference-count":29,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2021,5,5]],"date-time":"2021-05-05T00:00:00Z","timestamp":1620172800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["01091"],"award-info":[{"award-number":["01091"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Workflows are largely used to orchestrate complex sets of operations required to handle and process huge amounts of data. Parallel processing is often vital to reduce execution time when complex data-intensive workflows must be run efficiently, and at the same time, in-memory processing can bring important benefits to accelerate execution. However, optimization techniques are necessary to fully exploit in-memory processing, avoiding performance drops due to memory saturation events. This paper proposed a novel solution, called the Intelligent In-memory Workflow Manager (IIWM), for optimizing the in-memory execution of data-intensive workflows on parallel machines. IIWM is based on two complementary strategies: (1) a machine learning strategy for predicting the memory occupancy and execution time of workflow tasks; (2) a scheduling strategy that allocates tasks to a computing node, taking into account the (predicted) memory occupancy and execution time of each task and the memory available on that node. The effectiveness of the machine learning-based predictor and the scheduling strategy were demonstrated experimentally using as a testbed, Spark, a high-performance Big Data processing framework that exploits in-memory computing to speed up the execution of large-scale applications. In particular, two synthetic workflows were prepared for testing the robustness of the IIWM in scenarios characterized by a high level of parallelism and a limited amount of memory reserved for execution. Furthermore, a real data analysis workflow was used as a case study, for better assessing the benefits of the proposed approach. Thanks to high accuracy in predicting resources used at runtime, the IIWM was able to avoid disk writes caused by memory saturation, outperforming a traditional strategy in which only dependencies among tasks are taken into account. Specifically, the IIWM achieved up to a 31% and a 40% reduction of makespan and a performance improvement up to 1.45\u00d7 and 1.66\u00d7 on the synthetic workflows and the real case study, respectively.<\/jats:p>","DOI":"10.3390\/fi13050121","type":"journal-article","created":{"date-parts":[[2021,5,5]],"date-time":"2021-05-05T11:06:01Z","timestamp":1620212761000},"page":"121","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Exploiting Machine Learning for Improving In-Memory Execution of Data-Intensive Workflows on Parallel Machines"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3053-6132","authenticated-orcid":false,"given":"Riccardo","family":"Cantini","sequence":"first","affiliation":[{"name":"DIMES Department, University of Calabria, 87036 Rende, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7887-1314","authenticated-orcid":false,"given":"Fabrizio","family":"Marozzo","sequence":"additional","affiliation":[{"name":"DIMES Department, University of Calabria, 87036 Rende, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5031-1996","authenticated-orcid":false,"given":"Alessio","family":"Orsino","sequence":"additional","affiliation":[{"name":"DIMES Department, University of Calabria, 87036 Rende, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1910-9236","authenticated-orcid":false,"given":"Domenico","family":"Talia","sequence":"additional","affiliation":[{"name":"DIMES Department, University of Calabria, 87036 Rende, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5076-6544","authenticated-orcid":false,"given":"Paolo","family":"Trunfio","sequence":"additional","affiliation":[{"name":"DIMES Department, University of Calabria, 87036 Rende, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,5,5]]},"reference":[{"key":"ref_1","unstructured":"Talia, D., Trunfio, P., and Marozzo, F. (2015). Data Analysis in the Cloud, Elsevier."},{"key":"ref_2","first-page":"6","article-title":"Exascale machines require new programming paradigms and runtimes","volume":"2","author":"Fahringer","year":"2015","journal-title":"Supercomput. Front. Innov."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Li, M., Tan, J., Wang, Y., Zhang, L., and Salapura, V. (2015, January 18\u201321). SparkBench: A Comprehensive Benchmarking Suite for in Memory Data Analytic Platform Spark. Proceedings of the 12th ACM International Conference on Computing Frontiers, Ischia, Italy.","DOI":"10.1145\/2742854.2747283"},{"key":"ref_4","first-page":"1","article-title":"Data-intensive workflow management: For clouds and data-intensive and scalable computing environments","volume":"14","author":"Liu","year":"2019","journal-title":"Synth. Lect. Data Manag."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Verma, A., Mansuri, A.H., and Jain, N. (2016, January 18\u201319). Big data management processing with Hadoop MapReduce and spark technology: A comparison. Proceedings of the 2016 Symposium on Colossal Data Analysis and Networking (CDAN), Indore, India.","DOI":"10.1109\/CDAN.2016.7570891"},{"key":"ref_6","unstructured":"Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauly, M., Franklin, M.J., Shenker, S., and Stoica, I. (2012, January 25\u201327). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12), San Jose, CA, USA."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"e4367","DOI":"10.1002\/cpe.4367","article-title":"Performance comparison between Hadoop and Spark frameworks using HiBench benchmarks","volume":"30","author":"Samadi","year":"2018","journal-title":"Concurr. Comput. Pract. Exp."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Delimitrou, C., and Kozyrakis, C. (2014, January 1\u20135). Quasar: Resource-Efficient and QoS-Aware Cluster Management. Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems, Salt Lake City, UT, USA.","DOI":"10.1145\/2541940.2541941"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Llull, Q., Fan, S., Zahedi, S.M., and Lee, B.C. (2017, January 4\u20138). Cooper: Task Colocation with Cooperative Games. Proceedings of the 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA), Austin, TX, USA.","DOI":"10.1109\/HPCA.2017.22"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Marco, V.S., Taylor, B., Porter, B., and Wang, Z. (2017, January 11\u201315). Improving Spark Application Throughput via Memory Aware Task Co-Location: A Mixture of Experts Approach. Proceedings of the 18th ACM\/IFIP\/USENIX Middleware Conference, Las Vegas, NV, USA. Middleware\u201917.","DOI":"10.1145\/3135974.3135984"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Maros, A., Murai, F., Couto da Silva, A.P., Almeida, J.M., Lattuada, M., Gianniti, E., Hosseini, M., and Ardagna, D. (2019, January 8\u201313). Machine Learning for Performance Prediction of Spark Cloud Applications. Proceedings of the 2019 IEEE 12th International Conference on Cloud Computing (CLOUD), Milan, Italy.","DOI":"10.1109\/CLOUD.2019.00028"},{"key":"ref_12","first-page":"404525","article-title":"Workflow Systems for Science: Concepts and Tools","volume":"2013","author":"Talia","year":"2013","journal-title":"Int. Sch. Res. Not."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.future.2015.04.019","article-title":"Taxonomies of workflow scheduling problem and techniques in the cloud","volume":"52","author":"Smanchat","year":"2015","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/MCOM.2012.6295710","article-title":"Scheduling in hybrid clouds","volume":"50","author":"Bittencourt","year":"2012","journal-title":"IEEE Commun. Mag."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zhao, Y., Hu, F., and Chen, H. (February, January 31). An adaptive tuning strategy on spark based on in-memory computation characteristics. Proceedings of the 2016 18th International Conference on Advanced Communication Technology (ICACT), Pyeongchang, Korea.","DOI":"10.1109\/ICACT.2016.7423442"},{"key":"ref_16","first-page":"276","article-title":"An adaptive memory tuning strategy with high performance for Spark","volume":"4","author":"Chen","year":"2017","journal-title":"Int. J. Big Data Intell."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Xuan, P., Luo, F., Ge, R., and Srimani, P.K. (2017, January 14\u201317). Dynamic Management of In-Memory Storage for Efficiently Integrating Compute-and Data-Intensive Computing on HPC Systems. Proceedings of the 2017 17th IEEE\/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), Madrid, Spain.","DOI":"10.1109\/CCGRID.2017.66"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.jpdc.2020.03.010","article-title":"Dynamic memory-aware scheduling in spark computing environment","volume":"141","author":"Tang","year":"2020","journal-title":"J. Parallel Distrib. Comput."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Bae, J., Jang, H., Jin, W., Heo, J., Jang, J., Hwang, J., Cho, S., and Lee, J.W. (2017, January 11\u201314). Jointly optimizing task granularity and concurrency for in-memory mapreduce frameworks. Proceedings of the 2017 IEEE International Conference on Big Data (Big Data), Boston, MA, USA.","DOI":"10.1109\/BigData.2017.8257921"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1016\/S0893-6080(05)80023-1","article-title":"Stacked generalization","volume":"5","author":"Wolpert","year":"1992","journal-title":"Neural Netw."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1007\/s10723-015-9329-8","article-title":"A Survey of Data-Intensive Scientific Workflow Management","volume":"13","author":"Liu","year":"2015","journal-title":"J. Grid Comput."},{"key":"ref_22","unstructured":"Raj, P.H., Kumar, P.R., and Jelciana, P. (2018, January 16\u201318). Load Balancing in Mobile Cloud Computing using Bin Packing\u2019s First Fit Decreasing Method. Proceedings of the International Conference on Computational Intelligence in Information System, Brunei."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1016\/j.suscom.2018.05.011","article-title":"Cloud-SEnergy: A bin-packing based multi-cloud service broker for energy efficient composition and execution of data-intensive applications","volume":"19","author":"Baker","year":"2018","journal-title":"Sustain. Comput. Informatics Syst."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"977","DOI":"10.1016\/j.future.2012.03.002","article-title":"Scheduling real-time DAGs in heterogeneous clusters by combining imprecise computations and bin packing techniques for the exploitation of schedule holes","volume":"28","author":"Stavrinides","year":"2012","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1137\/0207001","article-title":"An application of bin-packing to multiprocessor scheduling","volume":"7","author":"Coffman","year":"1978","journal-title":"SIAM J. Comput."},{"key":"ref_26","unstructured":"Darapuneni, Y.J. (2021, April 29). A Survey of Classical and Recent Results in Bin Packing Problem. UNLV Theses, Dissertations, Professional Papers, and Capstones, Available online: https:\/\/digitalscholarship.unlv.edu\/cgi\/viewcontent.cgi?article=2664&context=thesesdissertations."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"e4229","DOI":"10.1002\/cpe.4229","article-title":"A data-aware scheduling strategy for workflow execution in clouds","volume":"29","author":"Marozzo","year":"2017","journal-title":"Concurr. Comput. Pract. Exp."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"480","DOI":"10.1109\/TSC.2016.2589243","article-title":"A Workflow Management System for Scalable Data Mining on Clouds","volume":"11","author":"Marozzo","year":"2018","journal-title":"IEEE Trans. Serv. Comput."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Aseeri, A.O., Zhuang, Y., and Alkatheiri, M.S. (2018, January 2\u20137). A Machine Learning-Based Security Vulnerability Study on XOR PUFs for Resource-Constraint Internet of Things. Proceedings of the 2018 IEEE International Congress on Internet of Things (ICIOT), San Francisco, CA, USA.","DOI":"10.1109\/ICIOT.2018.00014"}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/13\/5\/121\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:57:18Z","timestamp":1760162238000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/13\/5\/121"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,5]]},"references-count":29,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2021,5]]}},"alternative-id":["fi13050121"],"URL":"https:\/\/doi.org\/10.3390\/fi13050121","relation":{},"ISSN":["1999-5903"],"issn-type":[{"type":"electronic","value":"1999-5903"}],"subject":[],"published":{"date-parts":[[2021,5,5]]}}}