{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T14:15:26Z","timestamp":1784211326409,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":32,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,8,17]],"date-time":"2020-08-17T00:00:00Z","timestamp":1597622400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,8,17]]},"DOI":"10.1145\/3404397.3404430","type":"proceedings-article","created":{"date-parts":[[2020,8,9]],"date-time":"2020-08-09T03:54:26Z","timestamp":1596945266000},"page":"1-11","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["OPS: Optimized Shuffle Management System for Apache Spark"],"prefix":"10.1145","author":[{"given":"Yuchen","family":"Cheng","sequence":"first","affiliation":[{"name":"Shanghai Jiao Tong University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chunghsuan","family":"Wu","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanqiang","family":"Liu","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rui","family":"Ren","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hong","family":"Xu","sequence":"additional","affiliation":[{"name":"City University of Hong Kong"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bin","family":"Yang","sequence":"additional","affiliation":[{"name":"Intel Corporation"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhengwei","family":"Qi","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,8,17]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Apache. 2020. Apache Hadoop. https:\/\/hadoop.apache.org\/.  Apache. 2020. Apache Hadoop. https:\/\/hadoop.apache.org\/."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2013.297"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2014.2350972"},{"key":"e_1_3_2_1_4_1","volume-title":"2015 IEEE Conference on Computer Communications (INFOCOM). 1688\u20131696","author":"Chen W.","unstructured":"W. Chen , F. Ren , J. Xie , C. Lin , K. Yin , and F. Baker . 2015. Comprehensive understanding of TCP Incast problem . In 2015 IEEE Conference on Computer Communications (INFOCOM). 1688\u20131696 . W. Chen, F. Ren, J. Xie, C. Lin, K. Yin, and F. Baker. 2015. Comprehensive understanding of TCP Incast problem. In 2015 IEEE Conference on Computer Communications (INFOCOM). 1688\u20131696."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1592681.1592693"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2016.2594765"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1327452.1327492"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCC.2016.2552516"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3178487.3178510"},{"key":"e_1_3_2_1_10_1","unstructured":"M.\u00a0R. Garey and David\u00a0S. Johnson. 1978. Computers and Intractability: A Guide to the Theory of NP-Completeness.  M.\u00a0R. Garey and David\u00a0S. Johnson. 1978. Computers and Intractability: A Guide to the Theory of NP-Completeness."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1137\/0117039"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2016.2587645"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCGRID.2010.112"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2015.2405552"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213840"},{"key":"e_1_3_2_1_16_1","volume-title":"Warm-up Your JVM: Understand and Eliminate JVM Warm-up Overhead in Data-Parallel Systems. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)","author":"Lion David","year":"2016","unstructured":"David Lion , Adrian Chiu , Hailong Sun , Xin Zhuang , Nikola Grcevski , and Ding Yuan . 2016 . Don\u2019t Get Caught in the Cold , Warm-up Your JVM: Understand and Eliminate JVM Warm-up Overhead in Data-Parallel Systems. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16) . USENIX Association, Savannah, GA, 383\u2013400. David Lion, Adrian Chiu, Hailong Sun, Xin Zhuang, Nikola Grcevski, and Ding Yuan. 2016. Don\u2019t Get Caught in the Cold, Warm-up Your JVM: Understand and Eliminate JVM Warm-up Overhead in Data-Parallel Systems. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, Savannah, GA, 383\u2013400."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/71.963420"},{"key":"e_1_3_2_1_18_1","volume-title":"Presented as part of the 14th Workshop on Hot Topics in Operating Systems. USENIX","author":"Ousterhout Kay","unstructured":"Kay Ousterhout , Aurojit Panda , Joshua Rosen , Shivaram Venkataraman , Reynold Xin , Sylvia Ratnasamy , Scott Shenker , and Ion Stoica . 2013. The Case for Tiny Tasks in Compute Clusters . In Presented as part of the 14th Workshop on Hot Topics in Operating Systems. USENIX , Santa Ana Pueblo, NM. Kay Ousterhout, Aurojit Panda, Joshua Rosen, Shivaram Venkataraman, Reynold Xin, Sylvia Ratnasamy, Scott Shenker, and Ion Stoica. 2013. The Case for Tiny Tasks in Compute Clusters. In Presented as part of the 14th Workshop on Hot Topics in Operating Systems. USENIX, Santa Ana Pueblo, NM."},{"key":"e_1_3_2_1_19_1","volume-title":"Making Sense of Performance in Data Analytics Frameworks. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15)","author":"Ousterhout Kay","year":"2015","unstructured":"Kay Ousterhout , Ryan Rasti , Sylvia Ratnasamy , Scott Shenker , and Byung-Gon Chun . 2015 . Making Sense of Performance in Data Analytics Frameworks. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15) . USENIX Association, Oakland, CA, 293\u2013307. Kay Ousterhout, Ryan Rasti, Sylvia Ratnasamy, Scott Shenker, and Byung-Gon Chun. 2015. Making Sense of Performance in Data Analytics Frameworks. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15). USENIX Association, Oakland, CA, 293\u2013307."},{"key":"e_1_3_2_1_20_1","volume-title":"Proceedings of the 28th ACM International Conference on Supercomputing","author":"Lu Xiaoyi","unstructured":"Md\u00a0Wasi-ur Rahman, Xiaoyi Lu , Nusrat\u00a0Sharmin Islam , and Dhabaleswar K . \u00a0(DK) Panda. 2014. HOMR: A Hybrid Approach to Exploit Maximum Overlapping in MapReduce over High Performance Interconnects . In Proceedings of the 28th ACM International Conference on Supercomputing ( Munich, Germany) (ICS \u201914). ACM, 33\u201342. Md\u00a0Wasi-ur Rahman, Xiaoyi Lu, Nusrat\u00a0Sharmin Islam, and Dhabaleswar K.\u00a0(DK) Panda. 2014. HOMR: A Hybrid Approach to Exploit Maximum Overlapping in MapReduce over High Performance Interconnects. In Proceedings of the 28th ACM International Conference on Supercomputing (Munich, Germany) (ICS \u201914). ACM, 33\u201342."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2391229.2391233"},{"key":"e_1_3_2_1_22_1","volume-title":"HybridMR: A Hierarchical MapReduce Scheduler for Hybrid Data Centers. In 2013 IEEE 33rd International Conference on Distributed Computing Systems. 102\u2013111","author":"Sharma B.","year":"2013","unstructured":"B. Sharma , T. Wood , and C.\u00a0 R. Das . 2013 . HybridMR: A Hierarchical MapReduce Scheduler for Hybrid Data Centers. In 2013 IEEE 33rd International Conference on Distributed Computing Systems. 102\u2013111 . B. Sharma, T. Wood, and C.\u00a0R. Das. 2013. HybridMR: A Hierarchical MapReduce Scheduler for Hybrid Data Centers. In 2013 IEEE 33rd International Conference on Distributed Computing Systems. 102\u2013111."},{"key":"e_1_3_2_1_23_1","first-page":"38","article-title":"Crail: A High-Performance I\/O Architecture for Distributed Data Processing","volume":"40","author":"Stuedi Patrick","year":"2017","unstructured":"Patrick Stuedi , Animesh Trivedi , Jonas Pfefferle , Radu Stoica , Bernard Metzler , Nikolas Ioannou , and Ioannis Koltsidas . 2017 . Crail: A High-Performance I\/O Architecture for Distributed Data Processing . IEEE Data Eng. Bull. 40 , 1 (2017), 38 \u2013 49 . Patrick Stuedi, Animesh Trivedi, Jonas Pfefferle, Radu Stoica, Bernard Metzler, Nikolas Ioannou, and Ioannis Koltsidas. 2017. Crail: A High-Performance I\/O Architecture for Distributed Data Processing. IEEE Data Eng. Bull. 40, 1 (2017), 38\u201349.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/79173.79181"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1594977.1592604"},{"key":"e_1_3_2_1_26_1","volume-title":"Middleware","author":"Verma Abhishek","year":"2011","unstructured":"Abhishek Verma , Ludmila Cherkasova , and Roy\u00a0 H. Campbell . 2011. Resource Provisioning Framework for MapReduce Jobs with Performance Goals . In Middleware 2011 , Fabio Kon and Anne-Marie Kermarrec (Eds.). Springer Berlin Heidelberg , Berlin, Heidelberg, 165\u2013186. Abhishek Verma, Ludmila Cherkasova, and Roy\u00a0H. Campbell. 2011. Resource Provisioning Framework for MapReduce Jobs with Performance Goals. In Middleware 2011, Fabio Kon and Anne-Marie Kermarrec (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 165\u2013186."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807161"},{"key":"e_1_3_2_1_28_1","volume-title":"2015 IEEE Conference on Computer Communications (INFOCOM). 1328\u20131336","author":"Wang B.","unstructured":"B. Wang , J. Jiang , and G. Yang . 2015. ActCap: Accelerating MapReduce on heterogeneous clusters with capability-aware data placement . In 2015 IEEE Conference on Computer Communications (INFOCOM). 1328\u20131336 . B. Wang, J. Jiang, and G. Yang. 2015. ActCap: Accelerating MapReduce on heterogeneous clusters with capability-aware data placement. In 2015 IEEE Conference on Computer Communications (INFOCOM). 1328\u20131336."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063384.2063461"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPSW.2013.238"},{"key":"e_1_3_2_1_31_1","volume-title":"Presented as part of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12)","author":"Zaharia Matei","unstructured":"Matei Zaharia , Mosharaf Chowdhury , Tathagata Das , Ankur Dave , Justin Ma , Murphy McCauly , Michael\u00a0 J. Franklin , Scott Shenker , and Ion Stoica . 2012. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing . In Presented as part of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12) . USENIX , San Jose, CA , 15\u201328. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauly, Michael\u00a0J. Franklin, Scott Shenker, and Ion Stoica. 2012. Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. In Presented as part of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12). USENIX, San Jose, CA, 15\u201328."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190508.3190534"}],"event":{"name":"ICPP '20: 49th International Conference on Parallel Processing","location":"Edmonton AB Canada","acronym":"ICPP '20"},"container-title":["49th International Conference on Parallel Processing - ICPP"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3404397.3404430","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3404397.3404430","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:31:42Z","timestamp":1750195902000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3404397.3404430"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,17]]},"references-count":32,"alternative-id":["10.1145\/3404397.3404430","10.1145\/3404397"],"URL":"https:\/\/doi.org\/10.1145\/3404397.3404430","relation":{},"subject":[],"published":{"date-parts":[[2020,8,17]]},"assertion":[{"value":"2020-08-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}