{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,7]],"date-time":"2025-10-07T08:26:16Z","timestamp":1759825576609},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,2,13]],"date-time":"2020-02-13T00:00:00Z","timestamp":1581552000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,2,13]],"date-time":"2020-02-13T00:00:00Z","timestamp":1581552000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Comput Softw Big Sci"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>We describe a high-throughput computing system for running jobs on public and private computing clouds using the HTCondor job scheduler and the cloudscheduler VM provisioning service. The distributed cloud computing system is designed to simultaneously use dedicated and opportunistic cloud resources at local and remote locations. It has been used for large-scale production particle physics workloads for many years using thousands of cores on three continents. A decade after its initial design and implementation, cloudscheduler has been modernized to take advantage of new software designs, improved operating system capabilities and support packages. The updated cloudscheduler is more resilient and scalable, with expanded capabilities. We present an overview of the original design and then describe the new version of the distributed compute cloud system. We conclude with a review of the current status and future plans.<\/jats:p>","DOI":"10.1007\/s41781-020-0036-1","type":"journal-article","created":{"date-parts":[[2020,2,13]],"date-time":"2020-02-13T11:02:52Z","timestamp":1581591772000},"update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["High-Throughput Cloud Computing with the Cloudscheduler VM Provisioning Service"],"prefix":"10.1007","volume":"4","author":[{"given":"F.","family":"Berghaus","sequence":"first","affiliation":[]},{"given":"K.","family":"Casteels","sequence":"additional","affiliation":[]},{"given":"C.","family":"Driemel","sequence":"additional","affiliation":[]},{"given":"M.","family":"Ebert","sequence":"additional","affiliation":[]},{"given":"F. F.","family":"Galindo","sequence":"additional","affiliation":[]},{"given":"C.","family":"Leavett-Brown","sequence":"additional","affiliation":[]},{"given":"D.","family":"MacDonell","sequence":"additional","affiliation":[]},{"given":"M.","family":"Paterson","sequence":"additional","affiliation":[]},{"given":"R.","family":"Seuster","sequence":"additional","affiliation":[]},{"given":"R. J.","family":"Sobie","sequence":"additional","affiliation":[]},{"given":"S.","family":"Tolkamp","sequence":"additional","affiliation":[]},{"given":"J.","family":"Weldon","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,2,13]]},"reference":[{"key":"36_CR1","unstructured":"Elsen E The end of computing\u2019s steam age. http:\/\/cds.cern.ch\/record\/2212298"},{"key":"36_CR2","unstructured":"Armstrong P et al (2010) Cloud Scheduler: a resource manager for distributed compute clouds. arXiv preprint arXiv:1007.0050"},{"issue":"5","key":"36_CR3","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1109\/MIC.2009.94","volume":"13","author":"K Keahey","year":"2009","unstructured":"Keahey K, Tsugawa M, Matsunaga A, Fortes J (2009) Sky computing. Internet Comput IEEE 13(5):43\u201351. https:\/\/doi.org\/10.1109\/MIC.2009.94","journal-title":"Internet Comput IEEE"},{"key":"36_CR4","doi-asserted-by":"publisher","first-page":"052008","DOI":"10.1088\/1742-6596\/898\/5\/052008","volume":"898","author":"RP Taylor","year":"2017","unstructured":"Taylor RP et al (2017) Consolidation of cloud computing in ATLAS. J Phys Conf Ser 898:052008. https:\/\/doi.org\/10.1088\/1742-6596\/898\/5\/052008","journal-title":"J Phys Conf Ser"},{"key":"36_CR5","doi-asserted-by":"publisher","first-page":"022037","DOI":"10.1088\/1742-6596\/664\/2\/022037","volume":"664","author":"RJ Sobie","year":"2015","unstructured":"Sobie RJ et al (2015) Utilizing clouds for Belle II. J Phys Conf Ser 664:022037. https:\/\/doi.org\/10.1088\/1742-6596\/664\/2\/022037","journal-title":"J Phys Conf Ser"},{"key":"36_CR6","doi-asserted-by":"publisher","first-page":"022031","DOI":"10.1088\/1742-6596\/664\/2\/022031","volume":"664","author":"A McNab","year":"2015","unstructured":"McNab A et al (2015) Managing virtual machines with Vac and Vcycle. J Phys Conf Ser 664:022031. https:\/\/doi.org\/10.1088\/1742-6596\/664\/2\/022031","journal-title":"J Phys Conf Ser"},{"key":"36_CR7","doi-asserted-by":"publisher","first-page":"032096","DOI":"10.1088\/1742-6596\/513\/3\/032096","volume":"513","author":"A Tsaregorodtsev","year":"2014","unstructured":"Tsaregorodtsev A et al (2014) DIRAC distributed computing services. J Phys Conf Ser 513:032096. https:\/\/doi.org\/10.1088\/1742-6596\/513\/3\/032096","journal-title":"J Phys Conf Ser"},{"key":"36_CR8","doi-asserted-by":"publisher","first-page":"022021","DOI":"10.1088\/1742-6596\/664\/2\/022021","volume":"664","author":"R Grzymkowski","year":"2015","unstructured":"Grzymkowski R et al (2015) Belle II public and private cloud management in VMDIRAC. J Phys Conf Ser 664:022021. https:\/\/doi.org\/10.1088\/1742-6596\/664\/2\/022021","journal-title":"J Phys Conf Ser"},{"key":"36_CR9","doi-asserted-by":"publisher","first-page":"082031","DOI":"10.1088\/1742-6596\/898\/8\/082031","volume":"898","author":"A Amoroso","year":"2017","unstructured":"Amoroso A et al (2017) A modular (almost) automatic set-up for elastic multi-tenants cloud (micro)infrastructures. J Phys Conf Ser 898:082031. https:\/\/doi.org\/10.1088\/1742-6596\/898\/8\/082031","journal-title":"J Phys Conf Ser"},{"key":"36_CR10","doi-asserted-by":"publisher","first-page":"092021","DOI":"10.1088\/1742-6596\/898\/9\/092021","volume":"898","author":"B Bockelman","year":"2017","unstructured":"Bockelman B et al (2017) Interfacing HTCondor-CE with OpenStack. J Phys Conf Ser 898:092021. https:\/\/doi.org\/10.1088\/1742-6596\/898\/9\/092021","journal-title":"J Phys Conf Ser"},{"key":"36_CR11","doi-asserted-by":"publisher","first-page":"012003","DOI":"10.1088\/1742-6596\/256\/1\/012003","volume":"256","author":"K Fransham","year":"2010","unstructured":"Fransham K et al (2010) Research computing in a distributed cloud environment. J Phys Conf Ser 256:012003. https:\/\/doi.org\/10.1088\/1742-6596\/256\/1\/012003","journal-title":"J Phys Conf Ser"},{"key":"36_CR12","doi-asserted-by":"publisher","first-page":"062110","DOI":"10.1088\/1742-6596\/331\/6\/062010","volume":"331","author":"I Gable","year":"2011","unstructured":"Gable I et al (2011) A batch system for HEP applications in a distributed IaaS cloud. J Phys Conf Ser 331:062110. https:\/\/doi.org\/10.1088\/1742-6596\/331\/6\/062010","journal-title":"J Phys Conf Ser"},{"key":"36_CR13","doi-asserted-by":"publisher","first-page":"323","DOI":"10.1002\/cpe.938","volume":"17","author":"D Thain","year":"2005","unstructured":"Thain D, Tannenbaum T, Livny M (2005) Distributed computing in practice: the Condor experience. Concurr Comput Pract Exp 17:323. https:\/\/doi.org\/10.1002\/cpe.938","journal-title":"Concurr Comput Pract Exp"},{"key":"36_CR14","unstructured":"The Worldwide LHC Computing Grid. http:\/\/wlcg.web.cern.ch"},{"key":"36_CR15","doi-asserted-by":"publisher","first-page":"052002","DOI":"10.1088\/1742-6596\/898\/5\/052002","volume":"898","author":"FH Barreiro Megino","year":"2017","unstructured":"Barreiro Megino FH et al (2017) PanDA for ATLAS distributed computing in the next decade. J Phys Conf Ser 898:052002. https:\/\/doi.org\/10.1088\/1742-6596\/898\/5\/052002","journal-title":"J Phys Conf Ser"},{"key":"36_CR16","unstructured":"MariaDB open-source relational database. https:\/\/mariadb.org"},{"key":"36_CR17","doi-asserted-by":"publisher","unstructured":"Keahey K, Freeman T (2009) Contextualization: providing one-click virtual Clusters. IEEE Xplore. https:\/\/doi.org\/10.1109\/eScience.2008.82","DOI":"10.1109\/eScience.2008.82"},{"key":"36_CR18","unstructured":"Cloud\\_init: the standard for customizing cloud instances. https:\/\/cloud-init.io"},{"key":"36_CR19","doi-asserted-by":"publisher","first-page":"032009","DOI":"10.1088\/1742-6596\/513\/3\/032009","volume":"513","author":"J Blomer","year":"2014","unstructured":"Blomer J et al (2014) Micro-CernVM: slashing the cost of building and deploying virtual machines. J Phys Conf Ser 513:032009. https:\/\/doi.org\/10.1088\/1742-6596\/513\/3\/032009","journal-title":"J Phys Conf Ser"},{"key":"36_CR20","doi-asserted-by":"publisher","first-page":"062031","DOI":"10.1088\/1742-6596\/898\/6\/062031","volume":"898","author":"J Blomer","year":"2017","unstructured":"Blomer J et al (2017) New directions in the CernVM file system. J Phys Conf Ser 898:062031. https:\/\/doi.org\/10.1088\/1742-6596\/898\/6\/062031","journal-title":"J Phys Conf Ser"},{"key":"36_CR21","doi-asserted-by":"publisher","first-page":"032035","DOI":"10.1088\/1742-6596\/513\/3\/032035","volume":"513","author":"I Gable","year":"2014","unstructured":"Gable I et al (2014) Dynamic web cache publishing for IaaS clouds using Shoal. J Phys Conf Ser 513:032035. https:\/\/doi.org\/10.1088\/1742-6596\/513\/3\/032035","journal-title":"J Phys Conf Ser"},{"key":"36_CR22","unstructured":"Squid is a caching proxy for the Web. http:\/\/www.squid-cache.org"},{"key":"36_CR23","unstructured":"APEL is an accounting tool that collects accounting data from sites participating in the EGI and WLCG infrastructures. https:\/\/wiki.egi.eu\/wiki\/APEL"},{"key":"36_CR24","doi-asserted-by":"publisher","first-page":"052009","DOI":"10.1088\/1742-6596\/219\/5\/052009","volume":"219","author":"M Michelotto","year":"2010","unstructured":"Michelotto M et al (2010) A Comparison of HEP code with SPEC1 benchmarks on multi-core worker nodes. J Phys Conf Ser 219:052009. https:\/\/doi.org\/10.1088\/1742-6596\/219\/5\/052009","journal-title":"J Phys Conf Ser"},{"key":"36_CR25","doi-asserted-by":"publisher","first-page":"082011","DOI":"10.1088\/1742-6596\/898\/8\/082011","volume":"898","author":"P Charpentier","year":"2017","unstructured":"Charpentier P (2017) Benchmarking worker nodes using LHCb productions and comparing with HEPSpec06. J Phys Conf Ser 898:082011. https:\/\/doi.org\/10.1088\/1742-6596\/898\/8\/082011","journal-title":"J Phys Conf Ser"},{"key":"36_CR26","unstructured":"Valassi A et al (2019) Benchmarking WLCG resources using HEP experiment workloads. In: The proceedings of the computing in high energy physics conference, Adelaide (to be published)"},{"key":"36_CR27","unstructured":"Dynafed-The Dynamic Federation project. http:\/\/lcgdm.web.cern.ch\/dynafed-dynamic-federation-project"},{"key":"36_CR28","doi-asserted-by":"publisher","first-page":"629","DOI":"10.1134\/S1547477116050186","volume":"13","author":"F Furano","year":"2016","unstructured":"Furano F, Keeble O, Field L (2016) Dynamic federation of grid and cloud storage. Phys Part Nucl Lett 13:629. https:\/\/doi.org\/10.1134\/S1547477116050186","journal-title":"Phys Part Nucl Lett"},{"key":"36_CR29","doi-asserted-by":"publisher","first-page":"032027","DOI":"10.1088\/1742-6596\/1085\/3\/032027","volume":"1085","author":"F Berghaus","year":"2018","unstructured":"Berghaus F et al (2018) Federating distributed storage for clouds in ATLAS. J Phys Conf Ser 1085:032027. https:\/\/doi.org\/10.1088\/1742-6596\/1085\/3\/032027","journal-title":"J Phys Conf Ser"},{"key":"36_CR30","doi-asserted-by":"publisher","first-page":"04026","DOI":"10.1051\/epjconf\/201921404026","volume":"214","author":"M Ebert","year":"2019","unstructured":"Ebert M et al (2019) Using a dynamic data federation for running Belle-II simulation applications in a distributed cloud environment. EPJ Web Conf 214:04026. https:\/\/doi.org\/10.1051\/epjconf\/201921404026","journal-title":"EPJ Web Conf"},{"key":"36_CR31","unstructured":"Berghaus F et al (2019) The Dynafed data federator as grid site storage element. In: The Proceedings of the Computing in High Energy Physics Conference, Adelaide (to be published)"}],"container-title":["Computing and Software for Big Science"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s41781-020-0036-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s41781-020-0036-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s41781-020-0036-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,2,12]],"date-time":"2021-02-12T00:24:28Z","timestamp":1613089468000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s41781-020-0036-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,13]]},"references-count":31,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["36"],"URL":"https:\/\/doi.org\/10.1007\/s41781-020-0036-1","relation":{},"ISSN":["2510-2036","2510-2044"],"issn-type":[{"value":"2510-2036","type":"print"},{"value":"2510-2044","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,13]]},"assertion":[{"value":"25 September 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 January 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 February 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with Ethical Standards"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"4"}}