{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T11:23:16Z","timestamp":1790767396147,"version":"4.1.0"},"reference-count":35,"publisher":"SAGE Publications","issue":"6","license":[{"start":{"date-parts":[[2019,4,10]],"date-time":"2019-04-10T00:00:00Z","timestamp":1554854400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["The International Journal of High Performance Computing Applications"],"published-print":{"date-parts":[[2019,11]]},"abstract":"<jats:p>Scientific insights in the coming decade will clearly depend on the effective processing of large data sets generated by dynamic heterogeneous applications typical of workflows in large data centers or of emerging fields like neuroscience. In this article, we show how these big data workflows have a unique set of characteristics that pose challenges for leveraging HPC methodologies, particularly in scheduling. Our findings indicate that execution times for these workflows are highly unpredictable and are not correlated with the size of the data set involved or the precise functions used in the analysis. We characterize this inherent variability and sketch the need for new scheduling approaches by quantifying significant gaps in achievable performance. Through simulations, we show how on-the-fly scheduling approaches can deliver benefits in both system-level and user-level performance measures. On average, we find improvements of up to 35% in system utilization and up to 45% in average stretch of the applications, illustrating the potential of increasing performance through new scheduling approaches.<\/jats:p>","DOI":"10.1177\/1094342019841681","type":"journal-article","created":{"date-parts":[[2019,4,11]],"date-time":"2019-04-11T02:13:35Z","timestamp":1554948815000},"page":"1140-1158","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":9,"title":["On-the-fly scheduling versus reservation-based scheduling for unpredictable workflows"],"prefix":"10.1177","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1375-9468","authenticated-orcid":false,"given":"Ana","family":"Gainaru","sequence":"first","affiliation":[{"name":"Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hongyang","family":"Sun","sequence":"additional","affiliation":[{"name":"Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guillaume","family":"Aupy","sequence":"additional","affiliation":[{"name":"Inria & Labri, University of Bordeaux, Talence, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuankai","family":"Huo","sequence":"additional","affiliation":[{"name":"Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bennett A","family":"Landman","sequence":"additional","affiliation":[{"name":"Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Padma","family":"Raghavan","sequence":"additional","affiliation":[{"name":"Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2019,4,10]]},"reference":[{"key":"bibr1-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2014.06.005"},{"key":"bibr2-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2019.00027"},{"key":"bibr3-1094342019841681","volume-title":"Cray User Group Conference","author":"Ayyalasomayajula H","year":"2016"},{"key":"bibr4-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1145\/1008731.1008732"},{"key":"bibr5-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-04550-3"},{"key":"bibr6-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1145\/322234.322242"},{"key":"bibr7-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1109\/CCGRID.2005.1558641"},{"key":"bibr8-1094342019841681","first-page":"609","volume-title":"SODA","author":"Chekuri C","year":"1997"},{"key":"bibr9-1094342019841681","volume-title":"Theory of Scheduling","author":"Conway R","year":"1967"},{"key":"bibr10-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1145\/1327452.1327492"},{"key":"bibr11-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1007\/11407522_1"},{"key":"bibr12-1094342019841681","volume-title":"Computers and Intractability: A Guide to the Theory of NP-Completeness","author":"Garey MR","year":"1990"},{"key":"bibr13-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1109\/SFFCS.1999.814632"},{"key":"bibr14-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1966.tb01709.x"},{"key":"bibr15-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2015.05.021"},{"key":"bibr16-1094342019841681","first-page":"295","volume-title":"Proceedings of the 8th USENIX conference on networked systems design and implementation","author":"Hindman B","year":"2011"},{"key":"bibr17-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1145\/1272996.1273005"},{"key":"bibr18-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1145\/258533.258661"},{"key":"bibr19-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0061737"},{"key":"bibr20-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1145\/331524.331530"},{"key":"bibr21-1094342019841681","first-page":"422","volume-title":"SODA","author":"Motwani R","year":"1993"},{"key":"bibr22-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1109\/COMSWA.2007.382430"},{"key":"bibr23-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-74759-0_665"},{"key":"bibr24-1094342019841681","volume-title":"Scheduling: Theory, Algorithms, and Systems","author":"Pinedo ML","year":"2008","edition":"3"},{"key":"bibr25-1094342019841681","volume-title":"Online scheduling. In handbook of scheduling: Algorithms, models, and performance analysis","author":"Pruhs K","year":"2004"},{"key":"bibr26-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1109\/HPCS.2018.00100"},{"key":"bibr27-1094342019841681","first-page":"131","volume-title":"FOCS","author":"Shmoys DB","year":"1991"},{"key":"bibr28-1094342019841681","unstructured":"Simakov NA, White JP, DeLeon RL, et al. (2018) A workload analysis of NSF\u2019s innovative HPC resources using XDMoD.\n                      CoRR\n                      abs\/1801.04306."},{"key":"bibr29-1094342019841681","doi-asserted-by":"crossref","unstructured":"Thusoo A, Sen Sarma J, Jain N, et al. (2010) Hive - a petabyte scale data warehouse using hadoop, Long Beach, California, USA, 1\u20136 March 2010, pp. 996\u20131005.","DOI":"10.1109\/ICDE.2010.5447738"},{"key":"bibr30-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1109\/MCSE.2014.80"},{"key":"bibr31-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1145\/2523616.2523633"},{"key":"bibr32-1094342019841681","doi-asserted-by":"publisher","DOI":"10.3389\/fninf.2013.00027"},{"key":"bibr33-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1145\/2912152.2912155"},{"key":"bibr34-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1007\/10968987_3"},{"key":"bibr35-1094342019841681","doi-asserted-by":"publisher","DOI":"10.1145\/2934664"}],"container-title":["The International Journal of High Performance Computing Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342019841681","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/1094342019841681","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/1094342019841681","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T08:15:49Z","timestamp":1777450549000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/1094342019841681"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,4,10]]},"references-count":35,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2019,11]]}},"alternative-id":["10.1177\/1094342019841681"],"URL":"https:\/\/doi.org\/10.1177\/1094342019841681","relation":{},"ISSN":["1094-3420","1741-2846"],"issn-type":[{"value":"1094-3420","type":"print"},{"value":"1741-2846","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,4,10]]}}}