{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,7]],"date-time":"2025-11-07T13:33:12Z","timestamp":1762522392585},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"6","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2018,2]]},"abstract":"<jats:p>\n            In this paper, we focus on general-purpose\n            <jats:italic>Distributed Stream Data Processing Systems (DSDPSs)<\/jats:italic>\n            , which deal with processing of unbounded streams of continuous data at scale distributedly in real or near-real time. A fundamental problem in a DSDPS is the scheduling problem (i.e., assigning workload to workers\/machines) with the objective of minimizing average end-to-end tuple processing time. A widely-used solution is to distribute workload evenly over machines in the cluster in a round-robin manner, which is obviously not efficient due to lack of consideration for communication delay. Model-based approaches (such as queueing theory) do not work well either due to the high complexity of the system environment.\n          <\/jats:p>\n          <jats:p>We aim to develop a novel model-free approach that can learn to well control a DSDPS from its experience rather than accurate and mathematically solvable system models, just as a human learns a skill (such as cooking, driving, swimming, etc). Specifically, we, for the first time, propose to leverage emerging Deep Reinforcement Learning (DRL) for enabling model-free control in DSDPSs; and present design, implementation and evaluation of a novel and highly effective DRL-based control framework, which minimizes average end-to-end tuple processing time by jointly learning the system environment via collecting very limited runtime statistics data and making decisions under the guidance of powerful Deep Neural Networks (DNNs). To validate and evaluate the proposed framework, we implemented it based on a widely-used DSDPS, Apache Storm, and tested it with three representative applications: continuous queries, log stream processing and word count (stream version). Extensive experimental results show 1) Compared to Storm's default scheduler and the state-of-the-art model-based method, the proposed framework reduces average tuple processing by 33.5% and 14.0% respectively on average. 2) The proposed framework can quickly reach a good scheduling solution during online learning, which justifies its practicability for online control in DSDPSs.<\/jats:p>","DOI":"10.14778\/3184470.3184474","type":"journal-article","created":{"date-parts":[[2020,2,16]],"date-time":"2020-02-16T19:50:53Z","timestamp":1581882653000},"page":"705-718","source":"Crossref","is-referenced-by-count":35,"title":["Model-free control for distributed stream data processing using deep reinforcement learning"],"prefix":"10.14778","volume":"11","author":[{"given":"Teng","family":"Li","sequence":"first","affiliation":[{"name":"Syracuse University"}]},{"given":"Zhiyuan","family":"Xu","sequence":"additional","affiliation":[{"name":"Syracuse University"}]},{"given":"Jian","family":"Tang","sequence":"additional","affiliation":[{"name":"Syracuse University"}]},{"given":"Yanzhi","family":"Wang","sequence":"additional","affiliation":[{"name":"Syracuse University"}]}],"member":"320","published-online":{"date-parts":[[2018,10,5]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Alice's Adventures in Wonderland http:\/\/www.gutenberg.org\/files\/11\/11-pdf.pdf  Alice's Adventures in Wonderland http:\/\/www.gutenberg.org\/files\/11\/11-pdf.pdf"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2012.120"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536229"},{"key":"e_1_2_1_4_1","volume-title":"Storm real-time processing cookbook","author":"Anderson Q.","year":"2013","unstructured":"Q. Anderson , Storm real-time processing cookbook , PACKT Publishing , 2013 . Q. Anderson, Storm real-time processing cookbook, PACKT Publishing, 2013."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488222.2488267"},{"key":"e_1_2_1_6_1","volume-title":"Deep reinforcement learning in large discrete action spaces, arXiv: 1512.07679","author":"Arnold G. D.","year":"2016","unstructured":"G. D. Arnold , R. Evans , H. v. Hasselt , P. Sunehag , T. Lillicrap , J. Hunt , T. Mann , T. Weber , T. Degris and B. Coppin , Deep reinforcement learning in large discrete action spaces, arXiv: 1512.07679 , 2016 . G. D. Arnold, R. Evans, H. v. Hasselt, P. Sunehag, T. Lillicrap, J. Hunt, T. Mann, T. Weber, T. Degris and B. Coppin, Deep reinforcement learning in large discrete action spaces, arXiv: 1512.07679, 2016."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2287016.2287018"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1735688.1735706"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2479871.2479895"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/UCC.2014.46"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511804441","volume-title":"Convex Optimization Cambridge University Press","author":"Boyd S.","year":"2004","unstructured":"S. Boyd and L. Vandenberghe , Convex Optimization Cambridge University Press , 2004 . S. Boyd and L. Vandenberghe, Convex Optimization Cambridge University Press, 2004."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1806596.1806638"},{"key":"e_1_2_1_13_1","first-page":"1143","volume-title":"Proceedings of IEEE Infocom'2012","author":"Chen F.","unstructured":"F. Chen , M. Kodialam and T. V. Lakshman , Joint scheduling of processing and shuffle phases in MapReduce systems , Proceedings of IEEE Infocom'2012 , pp. 1143 -- 1151 . F. Chen, M. Kodialam and T. V. Lakshman, Joint scheduling of processing and shuffle phases in MapReduce systems, Proceedings of IEEE Infocom'2012, pp. 1143--1151."},{"key":"e_1_2_1_14_1","first-page":"155","volume-title":"Proceedings of NIPS'1996","author":"Drucker H.","unstructured":"H. Drucker , C. Burges , L. Kaufman , A. Smola and V. Vapnik , Support vector regression machines , Proceedings of NIPS'1996 , pp. 155 -- 161 . H. Drucker, C. Burges, L. Kaufman, A. Smola and V. Vapnik, Support vector regression machines, Proceedings of NIPS'1996, pp. 155--161."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of ICML'2016","author":"Duan Y.","unstructured":"Y. Duan , X. Chen , R. Houthooft , J. Schulman and P. Abbeel , Benchmarking deep reinforcement learning for continuous control , Proceedings of ICML'2016 . Y. Duan, X. Chen, R. Houthooft, J. Schulman and P. Abbeel, Benchmarking deep reinforcement learning for continuous control, Proceedings of ICML'2016."},{"key":"e_1_2_1_16_1","unstructured":"Apache Flink https:\/\/flink.apache.org\/  Apache Flink https:\/\/flink.apache.org\/"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of NIPS'2016","author":"Foerster J. N.","unstructured":"J. N. Foerster , Y. M. Assael , N. d. Freitas and S. Whiteson , Learning to communicate with deep multi-agent reinforcement learning , Proceedings of NIPS'2016 . J. N. Foerster, Y. M. Assael, N. d. Freitas and S. Whiteson, Learning to communicate with deep multi-agent reinforcement learning, Proceedings of NIPS'2016."},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of ICML'2016","author":"Gu S.","unstructured":"S. Gu , T. Lillicrap , I. Sutskever and S. Levine , Continuous deep Q-Learning with model-based acceleration , Proceedings of ICML'2016 . S. Gu, T. Lillicrap, I. Sutskever and S. Levine, Continuous deep Q-Learning with model-based acceleration, Proceedings of ICML'2016."},{"key":"e_1_2_1_19_1","unstructured":"Gurobi Optimizer http:\/\/www.gurobi.com\/  Gurobi Optimizer http:\/\/www.gurobi.com\/"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/78.492552"},{"key":"e_1_2_1_21_1","volume-title":"Data Mining: Concepts and Techniques","author":"Han J.","year":"2011","unstructured":"J. Han , M. Kamber and J. Pei , Data Mining: Concepts and Techniques ( 3 rd Edition), Morgan Kaufmann , 2011 . J. Han, M. Kamber and J. Pei, Data Mining: Concepts and Techniques (3rd Edition), Morgan Kaufmann, 2011.","edition":"3"},{"key":"e_1_2_1_22_1","unstructured":"Apache Hadoop http:\/\/hadoop.apache.org\/  Apache Hadoop http:\/\/hadoop.apache.org\/"},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of AAAI'2016","author":"v. Hasselt H.","unstructured":"H. v. Hasselt , A. Guez , and D. Silver , Deep reinforcement learning with double Q-learning , Proceedings of AAAI'2016 . H. v. Hasselt, A. Guez, and D. Silver, Deep reinforcement learning with double Q-learning, Proceedings of AAAI'2016."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.14778\/2367502.2367520"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TBDATA.2016.2616148"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of ICLR'2016","author":"Lillicrap T. P.","unstructured":"T. P. Lillicrap , J. J. Hunt , A. Pritzel , N. Heess , T. Erez , Y. Tassa , D. Silver and D. Wierstra , Continuous control with deep reinforcement learning , Proceedings of ICLR'2016 . T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver and D. Wierstra, Continuous control with deep reinforcement learning, Proceedings of ICLR'2016."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of ICDCS'2017","author":"Liu N.","unstructured":"N. Liu , Z. Li , J. Xu , Z. Xu , S. Lin , Q. Qiu , J. Tang and Y. Wang , A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning , Proceedings of ICDCS'2017 . N. Liu, Z. Li, J. Xu, Z. Xu, S. Lin, Q. Qiu, J. Tang and Y. Wang, A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning, Proceedings of ICDCS'2017."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2320765.2320789"},{"key":"e_1_2_1_29_1","unstructured":"Logstash - Open Source Log Management http:\/\/logstash.net\/  Logstash - Open Source Log Management http:\/\/logstash.net\/"},{"key":"e_1_2_1_31_1","first-page":"654","volume-title":"Proceedings of ACM STOC'1997","author":"Khoshkbarforoushha A.","unstructured":"A. Khoshkbarforoushha , R. Ranjan , R. Gaire , P. P. Jayaraman , J. Hosking and E. Abbasnejad , Resource usage estimation of data stream processing workloads in datacenter clouds, 2015, http:\/\/arxiv.org\/abs\/1501.07020 . Proceedings of ACM STOC'1997 , pp. 654 -- 663 . A. Khoshkbarforoushha, R. Ranjan, R. Gaire, P. P. Jayaraman, J. Hosking and E. Abbasnejad, Resource usage estimation of data stream processing workloads in datacenter clouds, 2015, http:\/\/arxiv.org\/abs\/1501.07020. Proceedings of ACM STOC'1997, pp. 654--663."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1739041.1739120"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/69.868912"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2015.04.080"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1001"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1504\/IJSN.2017.084342"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2465351.2465353"},{"key":"e_1_2_1_39_1","unstructured":"Redis http:\/\/redis.io  Redis http:\/\/redis.io"},{"key":"e_1_2_1_40_1","first-page":"346","volume-title":"Proceedings of IEEE DSN'2008","author":"Repantis T.","unstructured":"T. Repantis and V. Kalogeraki , Hot-spot prediction and alleviation in distributed stream processing applications , Proceedings of IEEE DSN'2008 , pp. 346 -- 355 . T. Repantis and V. Kalogeraki, Hot-spot prediction and alleviation in distributed stream processing applications, Proceedings of IEEE DSN'2008, pp. 346--355."},{"key":"e_1_2_1_41_1","volume-title":"Reinforcement learning - exploration vs exploitation","author":"Restelli M.","year":"2015","unstructured":"M. Restelli , Reinforcement learning - exploration vs exploitation , 2015 , http:\/\/home.deib.polimi.it\/restelli\/MyWebSite\/pdf\/rl5.pdf M. Restelli, Reinforcement learning - exploration vs exploitation, 2015, http:\/\/home.deib.polimi.it\/restelli\/MyWebSite\/pdf\/rl5.pdf"},{"key":"e_1_2_1_42_1","unstructured":"Apache S4 http:\/\/incubator.apache.org\/s4\/  Apache S4 http:\/\/incubator.apache.org\/s4\/"},{"key":"e_1_2_1_43_1","unstructured":"Apache Samza http:\/\/samza.apache.org\/  Apache Samza http:\/\/samza.apache.org\/"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of ICML'2014","author":"Silver D.","unstructured":"D. Silver , G. Lever , N. Heess , T. Degris , D. Wierstra and M. Riedmiller , Deterministic policy gradient algorithms , Proceedings of ICML'2014 . D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra and M. Riedmiller, Deterministic policy gradient algorithms, Proceedings of ICML'2014."},{"key":"e_1_2_1_45_1","volume-title":"Mastering the game of Go with deep neural networks and tree search Nature, 529: 484--489","author":"Silver D.","year":"2016","unstructured":"D. Silver , , Mastering the game of Go with deep neural networks and tree search Nature, 529: 484--489 , 2016 . D. Silver, et al., Mastering the game of Go with deep neural networks and tree search Nature, 529: 484--489, 2016."},{"key":"e_1_2_1_46_1","unstructured":"Apache Spark http:\/\/spark.apache.org\/  Apache Spark http:\/\/spark.apache.org\/"},{"key":"e_1_2_1_47_1","unstructured":"Spark Streaming --- Apache Spark http:\/\/spark.apache.org\/streaming\/  Spark Streaming --- Apache Spark http:\/\/spark.apache.org\/streaming\/"},{"key":"e_1_2_1_48_1","unstructured":"Apache Storm http:\/\/storm.apache.org\/  Apache Storm http:\/\/storm.apache.org\/"},{"key":"e_1_2_1_49_1","volume-title":"Reinforcement learning: an introduction","author":"Sutton R.","year":"1998","unstructured":"R. Sutton and A. Barto , Reinforcement learning: an introduction , MIT press Cambridge , 1998 . R. Sutton and A. Barto, Reinforcement learning: an introduction, MIT press Cambridge, 1998."},{"key":"e_1_2_1_50_1","unstructured":"TensorFlow https:\/\/www.tensorflow.org\/  TensorFlow https:\/\/www.tensorflow.org\/"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/RTSS.2006.34"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2014.61"},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of OSDI'2008","author":"Zaharia M.","unstructured":"M. Zaharia , A. Konwinski , A. D. Joseph , R. Katz , I. Stoica, Improving MapReduce performance in heterogeneous environments , Proceedings of OSDI'2008 . M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, I. Stoica, Improving MapReduce performance in heterogeneous environments, Proceedings of OSDI'2008."},{"key":"e_1_2_1_54_1","unstructured":"Apache Zookeeper https:\/\/zookeeper.apache.org\/  Apache Zookeeper https:\/\/zookeeper.apache.org\/"},{"key":"e_1_2_1_55_1","first-page":"2166","volume-title":"Proceedings of IEEE Infocom'2014","author":"Zhu Y.","unstructured":"Y. Zhu , , Minimizing makespan and total completion time in MapReduce-like systems , Proceedings of IEEE Infocom'2014 , pp. 2166 -- 2174 . Y. Zhu, et al., Minimizing makespan and total completion time in MapReduce-like systems, Proceedings of IEEE Infocom'2014, pp. 2166--2174."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3184470.3184474","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:57:44Z","timestamp":1672225064000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3184470.3184474"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,2]]},"references-count":54,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2018,2]]}},"alternative-id":["10.14778\/3184470.3184474"],"URL":"https:\/\/doi.org\/10.14778\/3184470.3184474","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2018,2]]}}}