{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T15:46:25Z","timestamp":1772725585136,"version":"3.50.1"},"reference-count":76,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,2,16]],"date-time":"2024-02-16T00:00:00Z","timestamp":1708041600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100006374","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2124897"],"award-info":[{"award-number":["2124897"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Meas. Anal. Comput. Syst."],"published-print":{"date-parts":[[2024,2,16]]},"abstract":"<jats:p>This work highlights the significance of I\/O bottlenecks that data-intensive HPC workflows face in serverless environments - an issue that has been largely overlooked by prior works. To address this challenge, we propose a novel framework, StarShip, which effectively addresses I\/O bottlenecks for HPC workflows executing in serverless environments by leveraging different storage options and multi-tier functions, co-optimizing for service time and service cost. StarShip exploits the Levenberg-Marquardt optimization method to find an effective solution in a large, complex search space. StarShip achieves significantly better performance and cost compared to competing techniques, improving service time by 45% and service cost by 37.6% on average over state-of-the-art solutions.<\/jats:p>","DOI":"10.1145\/3639028","type":"journal-article","created":{"date-parts":[[2024,2,21]],"date-time":"2024-02-21T17:01:32Z","timestamp":1708534892000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["StarShip: Mitigating I\/O Bottlenecks in Serverless Computing for Scientific Workflows"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1082-9846","authenticated-orcid":false,"given":"Rohan","family":"Basu Roy","sequence":"first","affiliation":[{"name":"Northeastern University, Boston, MA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7253-2458","authenticated-orcid":false,"given":"Devesh","family":"Tiwari","sequence":"additional","affiliation":[{"name":"Northeastern University, Boston, MA, USA"}]}],"member":"320","published-online":{"date-parts":[[2024,2,21]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"419","volume-title":"17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20)","author":"Agache Alexandru","year":"2020","unstructured":"Alexandru Agache, Marc Brooker, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and DianaMaria Popa. Firecracker: Lightweight virtualization for serverless applications. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20), pages 419--434, 2020."},{"key":"e_1_2_1_2_1","first-page":"923","volume-title":"2018 Usenix Annual Technical Conference (USENIX ATC)","author":"Akkus Istemi Ekin","year":"2018","unstructured":"Istemi Ekin Akkus, Ruichuan Chen, Ivica Rimac, Manuel Stein, Klaus Satzke, Andre Beck, Paarijaat Aditya, and Volker Hilt. Sand: Towards high-performance serverless computing. In 2018 Usenix Annual Technical Conference (USENIX ATC) , pages 923--935, 2018."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC41405.2020.00073"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSDM.2004.1311241"},{"key":"e_1_2_1_5_1","first-page":"10","volume-title":"37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the","author":"Amin Kaizar","unstructured":"Kaizar Amin, Gregor Von Laszewski, Mihael Hategan, Nestor J Zaluzec, Shawn Hampton, and Albert Rossi. Gridant: A client-controllable grid workflow system. In 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the, pages 10--pp. IEEE, 2004."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3361525.3361535"},{"key":"e_1_2_1_7_1","first-page":"1","volume-title":"Characterization of scientific workflows. In 2008 third workshop on workflows in support of large-scale science","author":"Bharathi Shishir","year":"2008","unstructured":"Shishir Bharathi, Ann Chervenak, Ewa Deelman, Gaurang Mehta, Mei-Hui Su, and Karan Vahi. Characterization of scientific workflows. In 2008 third workshop on workflows in support of large-scale science, pages 1--10. IEEE, 2008."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/UrgentHPC54802.2021.00010"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3342195.3392698"},{"key":"e_1_2_1_10_1","first-page":"198","volume-title":"3rd IEEE\/ACM International Symposium on Cluster Computing and the Grid, 2003. Proceedings.","author":"Cao Junwei","year":"2003","unstructured":"Junwei Cao, Stephen A Jarvis, Subhash Saini, and Graham R Nudd. Gridflow: Workflow management for grid computing. In CCGrid 2003. 3rd IEEE\/ACM International Symposium on Cluster Computing and the Grid, 2003. Proceedings., pages 198--205. IEEE, 2003."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357223.3362711"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3392717.3392764"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3419111.3421286"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/PDSW49588.2019.00005"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3369583.3392683"},{"key":"e_1_2_1_16_1","volume-title":"Serverless supercomputing: High performance function as a service for science. arXiv preprint arXiv:1908.04907","author":"Chard Ryan","year":"2019","unstructured":"Ryan Chard, Tyler J Skluzacek, Zhuozhao Li, Yadu Babuji, Anna Woodard, Ben Blaiszik, Steven Tuecke, Ian Foster, and Kyle Chard. Serverless supercomputing: High performance function as a service for science. arXiv preprint arXiv:1908.04907, 2019."},{"key":"e_1_2_1_17_1","volume-title":"A workflow architecture for cloud-based distributed simulation. ACM Transactions on Modeling and Computer Simulation (TOMACS), 32(2):1--26","author":"Chaudhry Nauman Riaz","year":"2022","unstructured":"Nauman Riaz Chaudhry, Anastasia Anagnostou, and Simon JE Taylor. A workflow architecture for cloud-based distributed simulation. ACM Transactions on Modeling and Computer Simulation (TOMACS), 32(2):1--26, 2022."},{"key":"e_1_2_1_18_1","volume-title":"Online","author":"Cochrane Sean","year":"2009","unstructured":"Sean Cochrane, Ken Kutzer, and L McIntosh. Solving the hpc i\/o bottleneck: Sun? lustre? storage system. Sun BluePrints? Online, Sun Microsystems, 2009."},{"key":"e_1_2_1_19_1","volume-title":"rfaas: Rdma-enabled faas platform for serverless high-performance computing. arXiv preprint arXiv:2106.13859","author":"Copik Marcin","year":"2021","unstructured":"Marcin Copik, Konstantin Taranov, Alexandru Calotoiu, and Torsten Hoefler. rfaas: Rdma-enabled faas platform for serverless high-performance computing. arXiv preprint arXiv:2106.13859, 2021."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2017.02.026"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2010.64"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2014.10.008"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2019.2919690"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3373376.3378512"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.2172\/1163236"},{"key":"e_1_2_1_26_1","volume-title":"AR\/VR for European Space Programmes","author":"Gerndt Andreas","year":"2019","unstructured":"Andreas Gerndt. Cosmoscout vr: Interactivity and immersion for space data exploration and mission planning. AR\/VR for European Space Programmes, 2019."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/eScience.2015.39"},{"key":"e_1_2_1_28_1","volume-title":"Serverless computing: One step forward, two steps back. arXiv preprint arXiv:1812.03651","author":"Hellerstein Joseph M","year":"2018","unstructured":"Joseph M Hellerstein, Jose Faleiro, Joseph E Gonzalez, Johann Schleier-Smith, Vikram Sreekanti, Alexey Tumanov, and Chenggang Wu. Serverless computing: One step forward, two steps back. arXiv preprint arXiv:1812.03651, 2018."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.5555\/3027041.3027047"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/3433701.3433744"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472456.3472501"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3445814.3446701"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3127479.3128601"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/ESCIW.2009.5408002"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357223.3362709"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTER51413.2022.00044"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2009.4"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPP.2013.18"},{"key":"e_1_2_1_39_1","first-page":"285","volume-title":"2021 USENIX Annual Technical Conference (USENIXATC 21)","author":"Mahgoub Ashraf","year":"2021","unstructured":"Ashraf Mahgoub, Karthick Shankar, Subrata Mitra, Ana Klimovic, Somali Chaterji, and Saurabh Bagchi. Sonic: Application-aware data passing for chained serverless applications. In 2021 USENIX Annual Technical Conference (USENIXATC 21), pages 285--301, 2021."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2499896.2499900"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2016.2593903"},{"key":"e_1_2_1_42_1","unstructured":"Daniel J Milroy Stephen Herbein and Dong H Ahn. Current and future converged cloud-hpc workflows at llnl."},{"key":"e_1_2_1_43_1","volume-title":"Ampl: a data-driven modeling pipeline for drug discovery. Journal of chemical information and modeling, 60(4):1955--1968","author":"Minnich Amanda J","year":"2020","unstructured":"Amanda J Minnich, Kevin McLoughlin, Margaret Tse, Jason Deng, Andrew Weber, Neha Murad, Benjamin D Madej, Bharath Ramsundar, Tom Rush, Stacie Calad-Thomson, et al. Ampl: a data-driven modeling pipeline for drug discovery. Journal of chemical information and modeling, 60(4):1955--1968, 2020."},{"key":"e_1_2_1_44_1","volume-title":"Lawrence Livermore National Lab.(LLNL)","author":"Minnich Amanda Jean","year":"2019","unstructured":"Amanda Jean Minnich, Kevin S McLoughlin, Jonathan E Allen, Ryan M Forsyth, Claire Weber, Neha Murad, Margaret Tse, Andrew Weber, Jason Deng, and Ben Madej. Atom modeling pipeline. Technical report, Lawrence Livermore National Lab.(LLNL), Livermore, CA (United States), 2019."},{"key":"e_1_2_1_45_1","first-page":"561","volume-title":"13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)","author":"Moritz Philipp","year":"2018","unstructured":"Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging ai applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 561--577, 2018."},{"key":"e_1_2_1_46_1","first-page":"57","volume-title":"2018 USENIX Annual Technical Conference (USENIXATC 18)","author":"Oakes Edward","year":"2018","unstructured":"Edward Oakes, Leon Yang, Dennis Zhou, Kevin Houck, Tyler Harter, Andrea Arpaci-Dusseau, and Remzi ArpaciDusseau. Sock: Rapid task provisioning with serverless-optimized containers. In 2018 USENIX Annual Technical Conference (USENIXATC 18), pages 57--70, 2018."},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the 18th USENIX Conference on File and Storage Technologies, 2020","author":"Patel Tirthak","year":"2020","unstructured":"Tirthak Patel and Rohan Garg. Gift: A coupon based throttle-and-reward mechanism for fair and efficient i\/o bandwidth management on parallel storage systems. In Proceedings of the 18th USENIX Conference on File and Storage Technologies, 2020, 2020."},{"key":"e_1_2_1_48_1","first-page":"1","volume-title":"SC20: International conference for high performance computing, networking, storage and analysis","author":"Patel Tirthak","year":"2020","unstructured":"Tirthak Patel, Zhengchun Liu, Raj Kettimuthu, Paul Rich, William Allcock, and Devesh Tiwari. Job characteristics on large-scale systems: long-term analysis, quantification, and implications. In SC20: International conference for high performance computing, networking, storage and analysis, pages 1--17. IEEE, 2020."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/HiPC50609.2020.00034"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2018.01.022"},{"key":"e_1_2_1_51_1","first-page":"193","volume-title":"16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19)","author":"Pu Qifan","year":"2019","unstructured":"Qifan Pu, Shivaram Venkataraman, and Ion Stoica. Shuffling, fast and slow: Scalable analytics on serverless infrastructure. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19), pages 193--206, 2019."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/MTAGS.2008.4777912"},{"issue":"1","key":"e_1_2_1_53_1","first-page":"101","article-title":"The levenberg-marquardt algorithm","volume":"11","author":"Ranganathan Ananth","year":"2004","unstructured":"Ananth Ranganathan. The levenberg-marquardt algorithm. Tutoral on LM algorithm, 11(1):101--110, 2004.","journal-title":"Tutoral on LM algorithm"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078597.3078604"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3503221.3508407"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/IISWC53511.2021.00018"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC41404.2022.00027"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3503222.3507750"},{"issue":"1","key":"e_1_2_1_59_1","first-page":"243","article-title":"Cancer diagnosis epigenomics scientific workflow scheduling in the cloud computing environment using an improved pso algorithm. Asian Pacific journal of cancer prevention","volume":"19","author":"Sadhasivam N","year":"2018","unstructured":"N Sadhasivam, R Balamurugan, and M Pandi. Cancer diagnosis epigenomics scientific workflow scheduling in the cloud computing environment using an improved pso algorithm. Asian Pacific journal of cancer prevention: APJCP, 19(1):243, 2018.","journal-title":"APJCP"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/AERO53065.2022.9843488"},{"key":"e_1_2_1_61_1","first-page":"205","volume-title":"2020 USENIX Annual Technical Conference (USENIX ATC 20)","author":"Shahrad Mohammad","year":"2020","unstructured":"Mohammad Shahrad, Rodrigo Fonseca, Inigo Goiri, Gohar Chaudhry, Paul Batum, Jason Cooke, Eduardo Laureano, Colby Tresness, Mark Russinovich, and Ricardo Bianchini. Serverless in the wild: Characterizing and optimizing the serverless workload at a large cloud provider. In 2020 USENIX Annual Technical Conference (USENIX ATC 20), pages 205--218, 2020."},{"key":"e_1_2_1_62_1","volume-title":"Numpywren: Serverless linear algebra. arXiv preprint arXiv:1810.09679","author":"Shankar Vaishaal","year":"2018","unstructured":"Vaishaal Shankar, Karl Krauth, Qifan Pu, Eric Jonas, Shivaram Venkataraman, Ion Stoica, Benjamin Recht, and Jonathan Ragan-Kelley. Numpywren: Serverless linear algebra. arXiv preprint arXiv:1810.09679, 2018."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3419111.3421287"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13062-015-0071-8"},{"key":"e_1_2_1_65_1","volume-title":"Optimizing prediction serving on low-latency serverless dataflow. arXiv preprint arXiv:2007.05832","author":"Sreekanti Vikram","year":"2020","unstructured":"Vikram Sreekanti, Harikaran Subbaraj, Chenggang Wu, Joseph E Gonzalez, and Joseph M Hellerstein. Optimizing prediction serving on low-latency serverless dataflow. arXiv preprint arXiv:2007.05832, 2020."},{"key":"e_1_2_1_66_1","volume-title":"Proceedings of the VLDB Endowment, 13(11)","author":"Sreekanti Vikram","unstructured":"Vikram Sreekanti, Chenggang Wu, Xiayue Charles Lin, Johann Schleier-Smith, Joseph E Gonzalez, Joseph M Hellerstein, and Alexey Tumanov. Cloudburst: Stateful functions-as-a-service. Proceedings of the VLDB Endowment, 13(11)."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCGrid.2016.31"},{"key":"e_1_2_1_68_1","volume-title":"Distributed computing in practice: the condor experience. Concurrency and computation: practice and experience, 17(2--4):323--356","author":"Thain Douglas","year":"2005","unstructured":"Douglas Thain, Todd Tannenbaum, and Miron Livny. Distributed computing in practice: the condor experience. Concurrency and computation: practice and experience, 17(2--4):323--356, 2005."},{"key":"e_1_2_1_69_1","first-page":"133","volume-title":"2018 USENIX Annual Technical Conference (USENIXATC 18)","author":"Wang Liang","year":"2018","unstructured":"Liang Wang, Mengyuan Li, Yinqian Zhang, Thomas Ristenpart, and Michael Swift. Peeking behind the curtains of serverless platforms. In 2018 USENIX Annual Technical Conference (USENIXATC 18), pages 133--146, 2018."},{"key":"e_1_2_1_70_1","volume-title":"Amazon web services in action","author":"Wittig Michael","year":"2018","unstructured":"Michael Wittig and Andreas Wittig. Amazon web services in action. Simon and Schuster, 2018."},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2008.92"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.5555\/2388996.2389007"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2010.5470453"},{"key":"e_1_2_1_74_1","first-page":"1049","volume-title":"2019 USENIX Annual Technical Conference (USENIXATC 19)","author":"Zhang Chengliang","year":"2019","unstructured":"Chengliang Zhang, Minchen Yu, Wei Wang, and Feng Yan. Mark: Exploiting cloud services for cost-effective, slo-aware machine learning inference serving. In 2019 USENIX Annual Technical Conference (USENIXATC 19), pages 1049--1062, 2019."},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357223.3362723"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2014.2341235"}],"container-title":["Proceedings of the ACM on Measurement and Analysis of Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639028","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3639028","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,23]],"date-time":"2025-08-23T01:39:22Z","timestamp":1755913162000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3639028"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,16]]},"references-count":76,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,2,16]]}},"alternative-id":["10.1145\/3639028"],"URL":"https:\/\/doi.org\/10.1145\/3639028","relation":{},"ISSN":["2476-1249"],"issn-type":[{"value":"2476-1249","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,16]]},"assertion":[{"value":"2024-02-21","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}