{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T05:03:59Z","timestamp":1750309439711,"version":"3.41.0"},"reference-count":27,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2025,2,11]],"date-time":"2025-02-11T00:00:00Z","timestamp":1739232000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Internet Technol."],"published-print":{"date-parts":[[2025,2,28]]},"abstract":"<jats:p>The traditional MapReduce frameworks were originally designed for processing data within a single cluster and are not suitable for handling geo-distributed data. Consequently, alternative approaches such as Hierarchical and Geo-Hadoop have been proposed to address this limitation. However, these approaches still face challenges in efficiently managing inter-cluster data transfer, particularly considering the heterogeneity of clusters and varying bandwidth among them. Moreover, the need to transmit results to a central global reducer for geo-distributed MapReduce operations adds unnecessary complexity. To tackle these issues, we introduce Extended Cross-MapReduce (ECMR), a framework that integrates resource heterogeneity and network links in geo-distributed MapReduce workflows. ECMR optimizes data management and determines the necessary data volume for generating final results. To enhance performance, ECMR leverages the overlap between data transfer and execution time by utilizing multiple global reducers and grouping temporary results that require data transfer over the Internet. In ECMR, we propose a bipartite graph and extend the Gale-Shapley algorithm to determine the optimal number of clusters and select the most suitable locations for global reducers. Through extensive experimental evaluations conducted on a real testbed, we demonstrate the effectiveness of our proposed ECMR method. The results exhibit significant improvements over traditional Hierarchical and Geo-Hadoop approaches, achieving reductions of up to 81% and 85% in overall makespan, respectively.<\/jats:p>","DOI":"10.1145\/3708322","type":"journal-article","created":{"date-parts":[[2024,12,17]],"date-time":"2024-12-17T11:24:01Z","timestamp":1734434641000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Optimizing Geo-Distributed Data Processing with Resource Heterogeneity over the Internet"],"prefix":"10.1145","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-9903-0358","authenticated-orcid":false,"given":"Saeed","family":"Mirpour Marzuni","sequence":"first","affiliation":[{"name":"University of Science and Technology of Mazandaran, Behshahr, Iran (the Islamic Republic of)"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5655-5337","authenticated-orcid":false,"given":"Adel","family":"Toosi","sequence":"additional","affiliation":[{"name":"School of Computing and Information Systems, The University of Melbourne, Melbounre, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9767-1959","authenticated-orcid":false,"given":"Abdorreza","family":"Savadi","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Ferdowsi University of Mashhad, Mashhad, Iran"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5550-5565","authenticated-orcid":false,"given":"Mahmud","family":"Naghibzadeh","sequence":"additional","affiliation":[{"name":"Department of Computer Engineering, Ferdowsi University of Mashhad, Mashhad, Iran"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8862-3960","authenticated-orcid":false,"given":"David","family":"Taniar","sequence":"additional","affiliation":[{"name":"Monash University, Clayton, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,2,11]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"5 tips for building a successful hybrid cloud. Retrieved January 2 2025 from https:\/\/www.computerworld.com\/article\/2834193\/5-tips-for-building-a-successful-hybrid-cloud.html"},{"key":"e_1_3_3_3_2","article-title":"Meta-MapReduce: A technique for reducing communication in MapReduce computations","author":"Afrati Foto","year":"2015","unstructured":"Foto Afrati, Shlomi Dolev, Shantanu Sharma, and Jeffrey D. Ullman. 2015. Meta-MapReduce: A technique for reducing communication in MapReduce computations. arXiv preprint arXiv:1508.01171 (2015).","journal-title":"arXiv preprint arXiv:1508.01171"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","unstructured":"Michael Cardosa Chenyu Wang Anshuman Nangia Abhishek Chandra and Jon Weissman. 2011. Exploring MapReduce efficiency with highly-distributed data. In2nd International Workshop on MapReduce and Its Applications (MapReduce\u201911). Association for Computing Machinery New York NY USA 27\u201334. DOI:10.1145\/1996092.1996100","DOI":"10.1145\/1996092.1996100"},{"key":"e_1_3_3_5_2","first-page":"555","volume-title":"IEEE Symposium on Computers and Communication (ISCC\u201916)","author":"Cavallo Marco","year":"2016","unstructured":"Marco Cavallo, Giuseppe Di Modica, Carmelo Polito, and Orazio Tomarchio. 2016. Application profiling in hierarchical Hadoop for geo-distributed computing environments. In IEEE Symposium on Computers and Communication (ISCC\u201916). IEEE, 555\u2013560."},{"key":"e_1_3_3_6_2","first-page":"27","volume-title":"3rd IEEE\/ACM International Conference on Big Data Computing, Applications and Technologies","author":"Cavallo Marco","year":"2016","unstructured":"Marco Cavallo, Carmelo Polito, Giuseppe Di Modica, and Orazio Tomarchio. 2016. H2F: A hierarchical Hadoop framework for big data processing in geo-distributed environments. In 3rd IEEE\/ACM International Conference on Big Data Computing, Applications and Technologies. 27\u201335."},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEM.2020.2971717"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TBDATA.2017.2723473"},{"issue":"1","key":"e_1_3_3_9_2","doi-asserted-by":"crossref","first-page":"17783","DOI":"10.1038\/s41598-023-44789-x","article-title":"Geographically distributed data management to support large-scale data analysis","volume":"13","author":"Emara Tamer Z.","year":"2023","unstructured":"Tamer Z. Emara, Thanh Trinh, and Joshua Zhexue Huang. 2023. Geographically distributed data management to support large-scale data analysis. Scient. Rep. 13, 1 (2023), 17783.","journal-title":"Scient. Rep."},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1080\/00029890.1962.11989827"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.parco.2003.04.001"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCC.2014.2355225"},{"key":"e_1_3_3_13_2","first-page":"261","volume-title":"13th IEEE\/ACM International Symposium on Cluster, Cloud, and Grid Computing","author":"Iordache Anca","year":"2013","unstructured":"Anca Iordache, Christine Morin, Nikos Parlavantzas, Eugen Feller, and Pierre Riteau. 2013. Resilin: Elastic mapreduce over multiple clouds. In 13th IEEE\/ACM International Symposium on Cluster, Cloud, and Grid Computing. IEEE, 261\u2013268."},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2013.121"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.14778\/2850578.2850582"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3006299.3006312"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2016.2626285"},{"key":"e_1_3_3_18_2","first-page":"769","volume-title":"12th IEEE\/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid\u201912)","author":"Luo Yuan","year":"2012","unstructured":"Yuan Luo and Beth Plale. 2012. Hierarchical MapReduce programming model and scheduling algorithms. In 12th IEEE\/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid\u201912). IEEE, 769\u2013774."},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2020.09.009"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/2699414"},{"key":"e_1_3_3_21_2","first-page":"1","volume-title":"3rd International Conference for Innovation in Technology (INOCON\u201924)","author":"Sharma Anu Priya","year":"2024","unstructured":"Anu Priya Sharma and Jaspreet Singh. 2024. Energy efficient distribution of heterogeneous workloads in cloud data center. In 3rd International Conference for Innovation in Technology (INOCON\u201924). IEEE, 1\u20136."},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2024.103837"},{"key":"e_1_3_3_23_2","first-page":"165","volume-title":"ACM\/IFIP\/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing","author":"Verma Abhishek","year":"2011","unstructured":"Abhishek Verma, Ludmila Cherkasova, and Roy H. Campbell. 2011. Resource provisioning framework for MapReduce jobs with performance goals. In ACM\/IFIP\/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing. Springer, 165\u2013186."},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2735365"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2012.09.001"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10191"},{"issue":"10","key":"e_1_3_3_27_2","first-page":"95","article-title":"Spark: Cluster computing with working sets.","volume":"10","author":"Zaharia Matei","year":"2010","unstructured":"Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster computing with working sets. HotCloud 10, 10-10 (2010), 95.","journal-title":"HotCloud"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2016.03.008"}],"container-title":["ACM Transactions on Internet Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3708322","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3708322","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:09:45Z","timestamp":1750295385000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3708322"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,11]]},"references-count":27,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,2,28]]}},"alternative-id":["10.1145\/3708322"],"URL":"https:\/\/doi.org\/10.1145\/3708322","relation":{},"ISSN":["1533-5399","1557-6051"],"issn-type":[{"type":"print","value":"1533-5399"},{"type":"electronic","value":"1557-6051"}],"subject":[],"published":{"date-parts":[[2025,2,11]]},"assertion":[{"value":"2023-06-28","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}