{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T06:37:01Z","timestamp":1790145421119,"version":"4.0.1"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2013,12,1]],"date-time":"2013-12-01T00:00:00Z","timestamp":1385856000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100005492","name":"Stanford University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100005492","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100006785","name":"Google","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006785","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Syst."],"published-print":{"date-parts":[[2013,12]]},"abstract":"<jats:p>Large-scale datacenters (DCs) host tens of thousands of diverse applications each day. However, interference between colocated workloads and the difficulty of matching applications to one of the many hardware platforms available can degrade performance, violating the quality of service (QoS) guarantees that many cloud workloads require. While previous work has identified the impact of heterogeneity and interference, existing solutions are computationally intensive, cannot be applied online, and do not scale beyond a few applications.<\/jats:p>\n          <jats:p>We present Paragon, an online and scalable DC scheduler that is heterogeneity- and interference-aware. Paragon is derived from robust analytical methods, and instead of profiling each application in detail, it leverages information the system already has about applications it has previously seen. It uses collaborative filtering techniques to quickly and accurately classify an unknown incoming workload with respect to heterogeneity and interference in multiple shared resources. It does so by identifying similarities to previously scheduled applications. The classification allows Paragon to greedily schedule applications in a manner that minimizes interference and maximizes server utilization. After the initial application placement, Paragon monitors application behavior and adjusts the scheduling decisions at runtime to avoid performance degradations. Additionally, we design ARQ, a multiclass admission control protocol that constrains application waiting time. ARQ queues applications in separate classes based on the type of resources they need and avoids long queueing delays for easy-to-satisfy workloads in highly-loaded scenarios. Paragon scales to tens of thousands of servers and applications with marginal scheduling overheads in terms of time or state.<\/jats:p>\n          <jats:p>We evaluate Paragon with a wide range of workload scenarios, on both small and large-scale systems, including 1,000 servers on EC2. For a 2,500-workload scenario, Paragon enforces performance guarantees for 91% of applications, while significantly improving utilization. In comparison, heterogeneity-oblivious, interference-oblivious, and least-loaded schedulers only provide similar guarantees for 14%, 11%, and 3% of workloads. The differences are more striking in oversubscribed scenarios where resource efficiency is more critical.<\/jats:p>","DOI":"10.1145\/2556583","type":"journal-article","created":{"date-parts":[[2013,12,20]],"date-time":"2013-12-20T19:50:46Z","timestamp":1387569046000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":116,"title":["QoS-Aware scheduling in heterogeneous datacenters with paragon"],"prefix":"10.1145","volume":"31","author":[{"given":"Christina","family":"Delimitrou","sequence":"first","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christos","family":"Kozyrakis","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2013,12,20]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/MM.2006.73"},{"key":"e_1_2_1_2_1","unstructured":"Amazon EC2. http:\/\/aws.amazon.com\/ec2\/.  Amazon EC2. http:\/\/aws.amazon.com\/ec2\/."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the Third Symposium on Operating Systems Design and Implementation (OSDI).","author":"Banga G.","unstructured":"Banga , G. , Druschel , P. , and Mogul , J. C . 1999. Resource containers: A new facility for resource management in server systems . In Proceedings of the Third Symposium on Operating Systems Design and Implementation (OSDI). Banga, G., Druschel, P., and Mogul, J. C. 1999. Resource containers: A new facility for resource management in server systems. In Proceedings of the Third Symposium on Operating Systems Design and Implementation (OSDI)."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2024723.2019527"},{"key":"e_1_2_1_5_1","volume-title":"Computer: An Introduction to the Design of Warehouse-Scale Machines. Morgan and Claypool.","author":"Barroso L.","year":"2009","unstructured":"Barroso , L. and Hoelzle , U . 2009 . The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines. Morgan and Claypool. Barroso, L. and Hoelzle, U. 2009. The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines. Morgan and Claypool."},{"key":"e_1_2_1_6_1","unstructured":"Bell R. M. Koren Y. and Volinsky C. 2007. The BellKor 2008 solution to the Netflix Prize. Tech. rep. AT&T Labs.  Bell R. M. Koren Y. and Volinsky C. 2007. The BellKor 2008 solution to the Netflix Prize. Tech. rep. AT&T Labs."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoap\/1015345407"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1454115.1454128"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-7908-2604-3_16"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043556.2043571"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/502034.502045"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the International Symposium on Computer Architecture (ISCA).","author":"Craeynest K. V.","unstructured":"Craeynest , K. V. , Jaleel , A. , Eeckhout , L. , Narvaez , P. , and Emer , J . 2012. Scheduling heterogeneous multi-cores through performance impact estimation (PIE) . In Proceedings of the International Symposium on Computer Architecture (ISCA). Craeynest, K. V., Jaleel, A., Eeckhout, L., Narvaez, P., and Emer, J. 2012. Scheduling heterogeneous multi-cores through performance impact estimation (PIE). In Proceedings of the International Symposium on Computer Architecture (ISCA)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoap\/1177004828"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoap\/1034968225"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the IEEE International Symposium on Workload Characterization (IISWC).","author":"Delimitrou C.","unstructured":"Delimitrou , C. and Kozyrakis , C . 2013a. iBench: Quantifying interference for datacenter applications . In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC). Delimitrou, C. and Kozyrakis, C. 2013a. iBench: Quantifying interference for datacenter applications. In Proceedings of the IEEE International Symposium on Workload Characterization (IISWC)."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2451116.2451125"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/L-CA.2012.10"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/1299042.1299108"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 11th Annual ACM-SIAM Symposium on Discrete Algorithms. 467--476","author":"Gamarnik D.","year":"2000","unstructured":"Gamarnik , D. 2000 . On deciding stability of scheduling policies in queuing systems . In Proceedings of the 11th Annual ACM-SIAM Symposium on Discrete Algorithms. 467--476 . Gamarnik, D. 2000. On deciding stability of scheduling policies in queuing systems. In Proceedings of the 11th Annual ACM-SIAM Symposium on Discrete Algorithms. 467--476."},{"key":"e_1_2_1_20_1","unstructured":"Google Compute Engine GCE. http:\/\/cloud.google.com\/products\/compute-engine.html.  Google Compute Engine GCE. http:\/\/cloud.google.com\/products\/compute-engine.html."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation (NSDI).","author":"Ghodsi A.","unstructured":"Ghodsi , A. , Zaharia , M. , Hindman , B. , Konwinski , A. , Shenker , S. , and Stoica , I . 2011. Dominant resource fairness: Fair allocation of multiple resource types . In Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation (NSDI). Ghodsi, A., Zaharia, M., Hindman, B., Konwinski, A., Shenker, S., and Stoica, I. 2011. Dominant resource fairness: Fair allocation of multiple resource types. In Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation (NSDI)."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/IISWC.2007.4362193"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2038916.2038938"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1555754.1555756"},{"key":"e_1_2_1_25_1","unstructured":"Hamilton J. 2010. Cost of power in large-scale data centers. http:\/\/perspectives.mvdirona.com.  Hamilton J. 2010. Cost of power in large-scale data centers. http:\/\/perspectives.mvdirona.com."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI).","author":"Hindman B.","unstructured":"Hindman , B. , Konwinski , A. , Zaharia , M. , Ghodsi , A. , Joseph , A. D. , Katz , R. , Shenker , S. , and Stoica , I . 2011. Mesos: A platform for fine-grained resource sharing in the data center . In Proceedings of the 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI). Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A. D., Katz, R., Shenker, S., and Stoica, I. 2011. Mesos: A platform for fine-grained resource sharing in the data center. In Proceedings of the 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI)."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 12th International Symposium on High-Performance Computer Architecture (HPCA-12)","author":"Jaleel A.","unstructured":"Jaleel , A. , Mattina , M. , and Jacob , B. L . 2006. Last level cache (LLC) performance of data mining workloads on a CMP\u2014A case study of parallel bioinformatics workloads . In Proceedings of the 12th International Symposium on High-Performance Computer Architecture (HPCA-12) . Jaleel, A., Mattina, M., and Jacob, B. L. 2006. Last level cache (LLC) performance of data mining workloads on a CMP\u2014A case study of parallel bioinformatics workloads. In Proceedings of the 12th International Symposium on High-Performance Computer Architecture (HPCA-12)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Katz J. and Lindell Y. 2007. Introduction to Modern Cryptography. Chapman & Hall\/CRC Press.   Katz J. and Lindell Y. 2007. Introduction to Modern Cryptography. Chapman & Hall\/CRC Press.","DOI":"10.1201\/9781420010756"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/PL00011414"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/MM.2010.73"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1740390.1740405"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213958"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2485922.2485975"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/L-CA.2011.14"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2000064.2000103"},{"key":"e_1_2_1_37_1","first-page":"234","article-title":"A queuing reward system with several customer classes. Manage","volume":"16","author":"Miller B. L.","year":"1969","unstructured":"Miller , B. L. 1969 . A queuing reward system with several customer classes. Manage . Sci. 16 , 3, 234 -- 245 . Miller, B. L. 1969. A queuing reward system with several customer classes. Manage. Sci. 16, 3, 234--245.","journal-title":"Sci."},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 9th IEEE International Symposium on Workload Characterization (IISWC).","author":"Narayanan R.","unstructured":"Narayanan , R. , Ozisikyilmaz , B. , Zambreno , J. , Memik , G. , and Choudhary , A. N . 2006. MineBench: A benchmark suite for data mining workloads . In Proceedings of the 9th IEEE International Symposium on Workload Characterization (IISWC). Narayanan, R., Ozisikyilmaz, B., Zambreno, J., Memik, G., and Choudhary, A. N. 2006. MineBench: A benchmark suite for data mining workloads. In Proceedings of the 9th IEEE International Symposium on Workload Characterization (IISWC)."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICAC.2007.16"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1755913.1755938"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the USENIX Annual Technical Conference (ATC).","author":"Novakovi","unstructured":"Novakovi &cgrave;, D., Vasi &cgrave;, N., Novakovi &cgrave;, S., Kosti &cgrave;, D., and Bianchini , R . 2013. DeepDive: Transparently identifying and managing performance interference in virtualized environments . In Proceedings of the USENIX Annual Technical Conference (ATC). Novakovi&cgrave;, D., Vasi&cgrave;, N., Novakovi&cgrave;, S., Kosti&cgrave;, D., and Bianchini, R. 2013. DeepDive: Transparently identifying and managing performance interference in virtualized environments. In Proceedings of the USENIX Annual Technical Conference (ATC)."},{"key":"e_1_2_1_42_1","unstructured":"Rackspace. Open Cloud. http:\/\/www.rackspace.com\/.  Rackspace. Open Cloud. http:\/\/www.rackspace.com\/."},{"key":"e_1_2_1_43_1","doi-asserted-by":"crossref","unstructured":"Rajaraman A. and Ullman J. 2011. Textbook on Mining of Massive Datasets. Rightscale. https:\/\/aws.amazon.com\/solution-providers\/isv\/rightscale.  Rajaraman A. and Ullman J. 2011. Textbook on Mining of Massive Datasets. Rightscale. https:\/\/aws.amazon.com\/solution-providers\/isv\/rightscale.","DOI":"10.1017\/CBO9781139058452"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2000064.2000073"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564421"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2465351.2465386"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1531793.1531804"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2038916.2038921"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1002\/sam.102"},{"key":"e_1_2_1_50_1","volume-title":"Modern Operating Systems","author":"Tanenbaum A. S.","unstructured":"Tanenbaum , A. S. 2007. Modern Operating Systems . 3 rd Ed. Peason Education, Inc. Tanenbaum, A. S. 2007. Modern Operating Systems. 3rd Ed. Peason Education, Inc.","edition":"3"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2150976.2151021"},{"key":"e_1_2_1_52_1","unstructured":"vMotion. Migrate VMs with Zero Downtime. http\/\/www.vmware.com\/products\/vmotion.  vMotion. Migrate VMs with Zero Downtime. http\/\/www.vmware.com\/products\/vmotion."},{"key":"e_1_2_1_53_1","first-page":"1","article-title":"Distributed resource scheduler: design, implementation and lessons learned","volume":"1","author":"Ware DRS","year":"2012","unstructured":"VM Ware - DRS . 2012 . Distributed resource scheduler: design, implementation and lessons learned . VMware Tech. J. 1 , 1 . VMWare-DRS. 2012. Distributed resource scheduler: design, implementation and lessons learned. VMware Tech. J. 1, 1.","journal-title":"VMware Tech. J."},{"key":"e_1_2_1_54_1","unstructured":"VMWare vSphere. http:\/\/www.vmware.com\/products\/vsphere\/.  VMWare vSphere. http:\/\/www.vmware.com\/products\/vsphere\/."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICTAI.2008.97"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/MM.2006.79"},{"key":"e_1_2_1_57_1","unstructured":"Windows Azure. http:\/\/www.windowsazure.com\/.  Windows Azure. http:\/\/www.windowsazure.com\/."},{"key":"e_1_2_1_58_1","volume-title":"Data Mining: Practical Machine Learning Tools and Techniques","author":"Witten I. H.","year":"2011","unstructured":"Witten , I. H. , Frank , E. , and Holmes , G . 2011 . Data Mining: Practical Machine Learning Tools and Techniques . 3 rd Ed. Witten, I. H., Frank, E., and Holmes, G. 2011. Data Mining: Practical Machine Learning Tools and Techniques. 3rd Ed.","edition":"3"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/223982.223990"},{"key":"e_1_2_1_60_1","unstructured":"Xenserver. 6.1. http:\/\/www.citrix.com\/xenserver\/.  Xenserver. 6.1. http:\/\/www.citrix.com\/xenserver\/."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2485922.2485974"},{"key":"e_1_2_1_62_1","volume-title":"Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI).","author":"Zaharia M.","unstructured":"Zaharia , M. , Chowdhury , M. , Das , T. , Dave , A. , Ma , J. , McCauley , M. , Franklin , M. J. , Shenker , S. , and Stoica , I . 2012. Spark: Cluster computing with working sets . In Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI). Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M. J., Shenker, S., and Stoica, I. 2012. Spark: Cluster computing with working sets. In Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI)."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2465351.2465388"},{"key":"e_1_2_1_64_1","doi-asserted-by":"crossref","unstructured":"Zhang Z.-K. Liu C. Zhang Y.-C. and Zhou T. 2010. Solving the cold-start problem in recommender systems with social tags. arXiv:1004.3732v2.  Zhang Z.-K. Liu C. Zhang Y.-C. and Zhou T. 2010. Solving the cold-start problem in recommender systems with social tags. arXiv:1004.3732v2.","DOI":"10.1209\/0295-5075\/92\/28002"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10586-008-0067-6"}],"container-title":["ACM Transactions on Computer Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2556583","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2556583","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T08:10:05Z","timestamp":1750234205000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2556583"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,12]]},"references-count":64,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2013,12]]}},"alternative-id":["10.1145\/2556583"],"URL":"https:\/\/doi.org\/10.1145\/2556583","relation":{},"ISSN":["0734-2071","1557-7333"],"issn-type":[{"value":"0734-2071","type":"print"},{"value":"1557-7333","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,12]]},"assertion":[{"value":"2013-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2013-12-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}