{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,27]],"date-time":"2026-06-27T16:27:15Z","timestamp":1782577635291,"version":"3.54.5"},"reference-count":35,"publisher":"PeerJ","license":[{"start":{"date-parts":[[2025,4,25]],"date-time":"2025-04-25T00:00:00Z","timestamp":1745539200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"<jats:p>Stream processing engines (SPEs) allow applications to process a large amount of data in real-time. However, to schedule big data applications; the SPEs create several challenges regarding resource utilisation, dynamic configurations, heterogeneous environment, resource awareness, load balancing, <jats:italic>etc.<\/jats:italic> As the volume of data increases over time, it also poses a challenge to predict the resource and application requirements for processing. All these factors play an important role, they can cause problems in achieving maximum throughput due to inefficiency in any of them. Most SPEs ignore the topology\u2019s structure, which may minimise throughput during scheduling and may increase network latency. In this article, a topology-aware and resource-aware scheduler (named WG-Storm) is proposed based on a directed acyclic graph (DAG) that enhances resource usage and overall throughput using efficient task assignment. WG-Storm is built on Apache Storm. Results are generated using the two linear topologies and compared with the five state-of-art schedulers including A3-Storm, Default, Isolation, Multi-tenant, and Resource-aware. The experimental results show up to 30% increased throughput using the least required computing resources in a heterogeneous cluster.<\/jats:p>","DOI":"10.7717\/peerj-cs.2767","type":"journal-article","created":{"date-parts":[[2025,4,25]],"date-time":"2025-04-25T08:07:06Z","timestamp":1745568426000},"page":"e2767","source":"Crossref","is-referenced-by-count":6,"title":["WG-Storm: a resource-aware scheduler for distributed stream processing engines"],"prefix":"10.7717","volume":"11","author":[{"given":"Rizwan","family":"Ali","sequence":"first","affiliation":[{"name":"Department of Software Engineering, National University of Computer and Emerging Sciences, Islamabad, Islamabad, Punjab, Pakistan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Asif","family":"Muhammad","sequence":"additional","affiliation":[{"name":"Department of Software Engineering, National University of Computer and Emerging Sciences, Islamabad, Islamabad, Punjab, Pakistan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Muhammad","family":"Aleem","sequence":"additional","affiliation":[{"name":"School of Science, Coventry University, Coventry, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Omair","family":"Shafiq","sequence":"additional","affiliation":[{"name":"Carleton University, Ottawa, Ontario, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"4443","published-online":{"date-parts":[[2025,4,25]]},"reference":[{"key":"10.7717\/peerj-cs.2767\/ref-1","article-title":"Example storm topologies","author":"Apache","year":"2024"},{"key":"10.7717\/peerj-cs.2767\/ref-2","article-title":"Storm\u2019s scheduler","author":"Apache Storm","year":"2024"},{"key":"10.7717\/peerj-cs.2767\/ref-3","article-title":"Word count topology","author":"Apache Storm Community","year":"2022"},{"key":"10.7717\/peerj-cs.2767\/ref-4","first-page":"303","article-title":"On the usability of Hadoop MapReduce, Apache Spark & Apache flink for data science","author":"Akil","year":"2017"},{"key":"10.7717\/peerj-cs.2767\/ref-5","article-title":"Maximizing the processing rate for streaming applications in apache storm","author":"Al-Sinayyid","year":"2018"},{"issue":"12","key":"10.7717\/peerj-cs.2767\/ref-6","doi-asserted-by":"publisher","first-page":"9609","DOI":"10.1007\/s11227-020-03223-z","article-title":"Job scheduler for streaming applications in heterogeneous distributed processing systems","volume":"76","author":"Al-Sinayyid","year":"2020","journal-title":"The Journal of Supercomputing"},{"key":"10.7717\/peerj-cs.2767\/ref-7","first-page":"207","article-title":"Adaptive online scheduling in storm","author":"Aniello","year":"2013"},{"key":"10.7717\/peerj-cs.2767\/ref-8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10479-023-05745-0","article-title":"Cloud computing load prediction by decomposition reinforced attention long short-term memory network optimized by modified particle swarm optimization algorithm","author":"Bacanin","year":"2023","journal-title":"Annals of Operations Research"},{"key":"10.7717\/peerj-cs.2767\/ref-9","first-page":"189","article-title":"Towards automatic parameter tuning of stream processing systems","author":"Bilal","year":"2017"},{"key":"10.7717\/peerj-cs.2767\/ref-10","doi-asserted-by":"publisher","first-page":"617","DOI":"10.1016\/j.future.2018.07.011","article-title":"T3-Scheduler: a topology and traffic aware two-level scheduler for stream processing systems in a heterogeneous cluster","volume":"89","author":"Eskandari","year":"2018","journal-title":"Future Generation Computer Systems"},{"key":"10.7717\/peerj-cs.2767\/ref-11","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1016\/j.future.2020.11.011","article-title":"I-Scheduler: iterative scheduling for distributed stream processing systems","volume":"117","author":"Eskandari","year":"2021","journal-title":"Future Generation Computer Systems"},{"key":"10.7717\/peerj-cs.2767\/ref-12","doi-asserted-by":"publisher","first-page":"975","DOI":"10.1016\/j.future.2017.08.051","article-title":"Aging-related performance anomalies in the apache storm stream processing system","volume":"86","author":"Ficco","year":"2018","journal-title":"Future Generation Computer Systems"},{"key":"10.7717\/peerj-cs.2767\/ref-13","article-title":"Clojure","author":"Hickey","year":"2010"},{"key":"10.7717\/peerj-cs.2767\/ref-14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/ICICET.2018.8533771","article-title":"A study of apache kafka in big data stream processing","author":"Hiraman","year":"2018"},{"issue":"1","key":"10.7717\/peerj-cs.2767\/ref-15","doi-asserted-by":"publisher","first-page":"9","DOI":"10.14445\/22312803\/IJCTT-V19P103","article-title":"Big data analysis: apache storm perspective","volume":"19","author":"Iqbal","year":"2015","journal-title":"International Journal of Computer Trends and Technology"},{"issue":"2","key":"10.7717\/peerj-cs.2767\/ref-16","doi-asserted-by":"publisher","first-page":"e7469","DOI":"10.1002\/cpe.7469","article-title":"Experimental performance analysis of cloud resource allocation framework using spider monkey optimization algorithm","volume":"35","author":"Kumar","year":"2023","journal-title":"Concurrency and Computation: Practice and Experience"},{"key":"10.7717\/peerj-cs.2767\/ref-17","doi-asserted-by":"publisher","first-page":"18285","DOI":"10.1007\/s00521-020-04955-y","article-title":"Autonomic cloud resource provisioning and scheduling using meta-heuristic algorithm","volume":"32","author":"Kumar","year":"2020","journal-title":"Neural Computing and Applications"},{"key":"10.7717\/peerj-cs.2767\/ref-18","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1007\/978-1-4302-4807-1_11","article-title":"Batch processing","author":"Layka","year":"2013","journal-title":"Beginning Groovy, Grails and Griffon"},{"key":"10.7717\/peerj-cs.2767\/ref-19","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1016\/j.jnca.2017.03.007","article-title":"Real-time scheduling based on optimized topology and communication traffic in distributed real-time computation platform of storm","volume":"87","author":"Li","year":"2017","journal-title":"Journal of Network and Computer Applications"},{"key":"10.7717\/peerj-cs.2767\/ref-20","first-page":"485","article-title":"D-storm: dynamic resource-efficient scheduling of stream processing applications","author":"Liu","year":"2017"},{"issue":"4","key":"10.7717\/peerj-cs.2767\/ref-21","doi-asserted-by":"publisher","first-page":"1373","DOI":"10.1109\/TNET.2019.2918341","article-title":"An adaptive online scheme for scheduling and resource enforcement in Storm","volume":"27","author":"Liu","year":"2019","journal-title":"IEEE\/ACM Transactions on Networking"},{"key":"10.7717\/peerj-cs.2767\/ref-22","doi-asserted-by":"publisher","first-page":"8359","DOI":"10.1007\/s11042-023-16008-2","article-title":"DRLBTSA: deep reinforcement learning based task-scheduling algorithm in cloud computing","volume":"83","author":"Mangalampalli","year":"2023","journal-title":"Multimedia Tools and Applications"},{"issue":"1","key":"10.7717\/peerj-cs.2767\/ref-23","doi-asserted-by":"publisher","first-page":"77","DOI":"10.3233\/IDA-173691","article-title":"Efficient resource scheduling for the analysis of big data streams","volume":"23","author":"Mortazavi-Dehkordi","year":"2019","journal-title":"Intelligent Data Analysis"},{"issue":"2","key":"10.7717\/peerj-cs.2767\/ref-24","doi-asserted-by":"publisher","first-page":"1059","DOI":"10.1007\/s11227-020-03289-9","article-title":"A3-Storm: topology-, traffic-, and resource-aware storm scheduler for heterogeneous clusters","volume":"77","author":"Muhammad","year":"2021","journal-title":"The Journal of Supercomputing"},{"issue":"1","key":"10.7717\/peerj-cs.2767\/ref-25","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1007\/s10586-020-03117-y","article-title":"TOP-Storm: a topology-based resource-aware scheduler for Stream Processing Engine","volume":"24","author":"Muhammad","year":"2021","journal-title":"Cluster Computing"},{"key":"10.7717\/peerj-cs.2767\/ref-26","doi-asserted-by":"publisher","first-page":"e1077","DOI":"10.7717\/peerj-cs.1077","article-title":"MF-Storm: a maximum flow-based job scheduler for stream processing engines on computational clusters to increase throughput","volume":"8","author":"Muhammad","year":"2022","journal-title":"PeerJ Computer Science"},{"key":"10.7717\/peerj-cs.2767\/ref-27","article-title":"A scheduling algorithm to maximize storm throughput in heterogeneous cluster","author":"Nasiri","year":"2020"},{"issue":"2","key":"10.7717\/peerj-cs.2767\/ref-28","doi-asserted-by":"publisher","first-page":"2249","DOI":"10.1007\/s40747-023-01265-3","article-title":"Cloud-load forecasting via decomposition-aided attention recurrent neural network tuned by modified particle swarm optimization","volume":"10","author":"Predi\u0107","year":"2024","journal-title":"Complex & Intelligent Systems"},{"key":"10.7717\/peerj-cs.2767\/ref-29","first-page":"1","article-title":"A traffic and resource aware online storm scheduler","author":"Qi","year":"2021"},{"issue":"6","key":"10.7717\/peerj-cs.2767\/ref-30","doi-asserted-by":"publisher","first-page":"2951","DOI":"10.1007\/s41870-022-01045-1","article-title":"An optimistic approach for task scheduling in cloud computing","volume":"14","author":"Sharma","year":"2022","journal-title":"International Journal of Information Technology"},{"issue":"14","key":"10.7717\/peerj-cs.2767\/ref-31","doi-asserted-by":"publisher","first-page":"4796","DOI":"10.3390\/app10144796","article-title":"Pipelined dynamic scheduling of big data streams","volume":"10","author":"Souravlas","year":"2020","journal-title":"Applied Sciences"},{"key":"10.7717\/peerj-cs.2767\/ref-32","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1016\/j.future.2020.08.003","article-title":"Lr-Stream: using latency and resource aware scheduling to improve latency and throughput for streaming applications","volume":"114","author":"Sun","year":"2021","journal-title":"Future Generation Computer Systems"},{"key":"10.7717\/peerj-cs.2767\/ref-33","first-page":"269","article-title":"Adaptive topology decomposition for storm","author":"Xie","year":"2017"},{"key":"10.7717\/peerj-cs.2767\/ref-34","first-page":"587","article-title":"TOSS: a topology-based scheduler for storm c1usters","author":"Zhou","year":"2020"},{"key":"10.7717\/peerj-cs.2767\/ref-35","first-page":"267","article-title":"Effective multi-stream joining in apache samza framework","author":"Zhuang","year":"2016"}],"container-title":["PeerJ Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/peerj.com\/articles\/cs-2767.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-2767.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-2767.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-2767.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,25]],"date-time":"2025-04-25T08:07:16Z","timestamp":1745568436000},"score":1,"resource":{"primary":{"URL":"https:\/\/peerj.com\/articles\/cs-2767"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4,25]]},"references-count":35,"alternative-id":["10.7717\/peerj-cs.2767"],"URL":"https:\/\/doi.org\/10.7717\/peerj-cs.2767","archive":["CLOCKSS","LOCKSS","Portico"],"relation":{},"ISSN":["2376-5992"],"issn-type":[{"value":"2376-5992","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4,25]]},"article-number":"e2767"}}