{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,27]],"date-time":"2026-06-27T16:27:13Z","timestamp":1782577633852,"version":"3.54.5"},"reference-count":44,"publisher":"PeerJ","license":[{"start":{"date-parts":[[2022,9,26]],"date-time":"2022-09-26T00:00:00Z","timestamp":1664150400000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"abstract":"<jats:sec>\n<jats:title>Background<\/jats:title>\n<jats:p>A scheduling algorithm tries to schedule multiple computational tasks on a cluster of multiple computing nodes to maximize throughput with optimal utilization of computational and communicational resources. A Stream Processing Engine (SPE) is deployed to run streaming applications (computational tasks) on a computational cluster which helps execution and coordination of these applications. It is observed that there is a gap in the optimal mapping of a computational and communicational load of a streaming application on the underlying computational and communication power of the resources (cluster). Frequently communicated tasks are scheduled at different processing nodes with relatively slow communicating links. This increases network latency with a decrease in resource utilization. Hence, reduces the achieved throughput of the cluster significantly.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title>Methods<\/jats:title>\n<jats:p>MF-Storm, a max-flow min-cut based job scheduler is presented to achieve a near-optimum schedule to maximize throughput. It schedules a streaming application by considering the processing, communication demands, available computational and communicational resources in a heterogeneous cluster, dynamically with minimized scheduling cost. To keep the scheduling cost minimum, the scheduler is built in a pipeline with two major stages: in the first stage, the application\u2019s tasks graph is partitioned using the max-flow min-cut algorithm to minimize inter-partition traffic, and in the second stage, these partitions are assigned to computing nodes according to the computational power of the cluster\u2019s nodes.<\/jats:p>\n<\/jats:sec>\n<jats:sec>\n<jats:title>Results<\/jats:title>\n<jats:p>Extensive experiments were done to evaluate the performance of MF-Storm using different topologies with multiple scenarios on a physical cluster implementation. Results showed on average 148% improvement in throughput with 30% less computational resources as compared to different state-of-the-art schedulers.<\/jats:p>\n<\/jats:sec>","DOI":"10.7717\/peerj-cs.1077","type":"journal-article","created":{"date-parts":[[2022,9,26]],"date-time":"2022-09-26T08:11:55Z","timestamp":1664179915000},"page":"e1077","source":"Crossref","is-referenced-by-count":5,"title":["MF-Storm: a maximum flow-based job scheduler for stream processing engines on computational clusters to increase throughput"],"prefix":"10.7717","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9215-4100","authenticated-orcid":true,"given":"Asif","family":"Muhammad","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Muhammad","family":"Abdul Qadir","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"4443","published-online":{"date-parts":[[2022,9,26]]},"reference":[{"issue":"12","key":"10.7717\/peerj-cs.1077\/ref-1","doi-asserted-by":"publisher","first-page":"9609","DOI":"10.1007\/s11227-020-03223-z","article-title":"Job scheduler for streaming applications in heterogeneous distributed processing systems","volume":"76","author":"Al-Sinayyid","year":"2020","journal-title":"Journal of Supercomputing"},{"key":"10.7717\/peerj-cs.1077\/ref-2","first-page":"207","article-title":"Adaptive online scheduling in storm","author":"Aniello","year":"2013"},{"issue":"9","key":"10.7717\/peerj-cs.1077\/ref-3","doi-asserted-by":"publisher","first-page":"e4334","DOI":"10.1002\/cpe.4334","article-title":"Optimal operator deployment and replication for elastic distributed data stream processing","volume":"30","author":"Cardellini","year":"2018","journal-title":"Concurrency Computation"},{"issue":"9","key":"10.7717\/peerj-cs.1077\/ref-4","doi-asserted-by":"publisher","first-page":"e932","DOI":"10.7717\/peerj-cs.932","article-title":"An evolutionary algorithm for multi-objective optimization of freshwater consumption in textile dyeing industry","volume":"8","author":"Elahi","year":"2022","journal-title":"PeerJ Computer Science"},{"key":"10.7717\/peerj-cs.1077\/ref-5","first-page":"1","article-title":"P-scheduler: adaptive hierarchical scheduling in Apache Storm","author":"Eskandari","year":"2016"},{"key":"10.7717\/peerj-cs.1077\/ref-6","first-page":"234","article-title":"Poster: iterative scheduling for distributed stream processing systems","author":"Eskandari","year":"2018a"},{"issue":"8","key":"10.7717\/peerj-cs.1077\/ref-7","doi-asserted-by":"publisher","first-page":"617","DOI":"10.1016\/j.future.2018.07.011","article-title":"T3-Scheduler: a topology and Traffic aware two-level Scheduler for stream processing systems in a heterogeneous cluster","volume":"89","author":"Eskandari","year":"2018b","journal-title":"Future Generation Computer Systems"},{"key":"10.7717\/peerj-cs.1077\/ref-8","article-title":"Task scheduling in data stream processing","author":"Falt","year":"2011"},{"key":"10.7717\/peerj-cs.1077\/ref-9","first-page":"309","article-title":"Adaptive task scheduling in storm","author":"Fan","year":"2016"},{"issue":"1","key":"10.7717\/peerj-cs.1077\/ref-10","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1007\/s10922-021-09632-6","article-title":"A novel cooperative micro-caching algorithm based on fuzzy inference through NFV in ultra-dense IoT networks","volume":"30","author":"Farooq","year":"2022","journal-title":"Journal of Network and Systems Management"},{"key":"10.7717\/peerj-cs.1077\/ref-11","first-page":"124","article-title":"Workload scheduling in distributed stream processors using graph partitioning","volume":"2015","author":"Fischer","year":"2015"},{"key":"10.7717\/peerj-cs.1077\/ref-12","article-title":"Apache Spark\u2122 \u2013 Unified analytics engine for big data","author":"Foundation AS","year":"2018"},{"key":"10.7717\/peerj-cs.1077\/ref-13","article-title":"Apache Flink: stateful computations over data streams","author":"Foundation AS, Apache Software Foundation","year":"2015"},{"issue":"4","key":"10.7717\/peerj-cs.1077\/ref-14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2904080","article-title":"Scheduling storms and streams in the cloud","volume":"1","author":"Ghaderi","year":"2016","journal-title":"ACM Transactions on Modeling and Performance Evaluation of Computing Systems"},{"key":"10.7717\/peerj-cs.1077\/ref-15","article-title":"Storm isolation scheduler","author":"GitHub","year":"2014"},{"key":"10.7717\/peerj-cs.1077\/ref-16","article-title":"Exclamation topology","author":"GitHub","year":"2019"},{"issue":"1","key":"10.7717\/peerj-cs.1077\/ref-17","doi-asserted-by":"publisher","first-page":"708","DOI":"10.1007\/s11227-019-03060-9","article-title":"Use case-based evaluation of workflow optimization strategy in real-time computation system","volume":"76","author":"Gulzar Ahmad","year":"2020","journal-title":"Journal of Supercomputing"},{"key":"10.7717\/peerj-cs.1077\/ref-18","volume-title":"Survey of distributed stream processing","author":"Kamburugamuve","year":"2016"},{"key":"10.7717\/peerj-cs.1077\/ref-19","article-title":"Mastering hadoop","author":"Karanth","year":"2014"},{"issue":"10","key":"10.7717\/peerj-cs.1077\/ref-20","doi-asserted-by":"publisher","first-page":"5399","DOI":"10.1007\/s11227-018-2435-1","article-title":"E-OSched: a load balancing scheduler for heterogeneous multicores","volume":"74","author":"Khalid","year":"2018","journal-title":"Journal of Supercomputing"},{"issue":"6","key":"10.7717\/peerj-cs.1077\/ref-21","doi-asserted-by":"publisher","first-page":"705","DOI":"10.14778\/3184470.3184474","article-title":"Model-free control for distributed stream data processing using deep reinforcement learning","volume":"11","author":"Li","year":"2018","journal-title":"Proceedings of the VLDB Endowment"},{"issue":"12","key":"10.7717\/peerj-cs.1077\/ref-22","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1016\/j.jnca.2017.03.007","article-title":"Real-time scheduling based on optimized topology and communication traffic in distributed real-time computation platform of storm","volume":"87","author":"Li","year":"2017","journal-title":"Journal of Network and Computer Applications"},{"key":"10.7717\/peerj-cs.1077\/ref-23","article-title":"Energy usage proling and topology-based scheduling for clusters","author":"Liu","year":"2017"},{"key":"10.7717\/peerj-cs.1077\/ref-24","first-page":"485","article-title":"D-Storm: dynamic resource-efficient scheduling of stream processing applications","author":"Liu","year":"2018"},{"issue":"3","key":"10.7717\/peerj-cs.1077\/ref-25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3355399","article-title":"Resource management and scheduling in distributed stream processing systems: a taxonomy, review, and future directions","volume":"53","author":"Liu","year":"2020","journal-title":"ACM Computing Surveys"},{"key":"10.7717\/peerj-cs.1077\/ref-26","first-page":"571","article-title":"E-Storm: replication-based state management in distributed stream processing systems","author":"Liu","year":"2017"},{"key":"10.7717\/peerj-cs.1077\/ref-27","first-page":"13","article-title":"Dynamic resource management in a massively parallel stream processing engine","author":"Madsen","year":"2015"},{"issue":"2","key":"10.7717\/peerj-cs.1077\/ref-28","doi-asserted-by":"publisher","first-page":"1059","DOI":"10.1007\/s11227-020-03289-9","article-title":"A3-Storm: topology-, traffic-, and resource-aware storm scheduler for heterogeneous clusters","volume":"77","author":"Muhammad","year":"2021","journal-title":"Journal of Supercomputing"},{"issue":"1","key":"10.7717\/peerj-cs.1077\/ref-29","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1007\/s10586-020-03117-y","article-title":"TOP-Storm: a topology-based resource-aware scheduler for stream processing engine","volume":"24","author":"Muhammad","year":"2021","journal-title":"Cluster Computing"},{"key":"10.7717\/peerj-cs.1077\/ref-30","article-title":"A scheduling algorithm to maximize storm throughput in heterogeneous cluster","author":"Nasiri","year":"2020"},{"key":"10.7717\/peerj-cs.1077\/ref-31","first-page":"149","article-title":"R-storm: resource-aware scheduling in storm","author":"Peng","year":"2015"},{"key":"10.7717\/peerj-cs.1077\/ref-32","first-page":"623","article-title":"S-Storm: a slot-aware scheduling strategy for even scheduler in storm","volume":"2016","author":"Qian","year":"2017"},{"key":"10.7717\/peerj-cs.1077\/ref-33","doi-asserted-by":"publisher","first-page":"2240","DOI":"10.1016\/j.procs.2017.05.249","article-title":"Performance-aware scheduling of streaming applications using genetic algorithm","volume":"108","author":"Smirnov","year":"2017","journal-title":"Procedia Computer Science"},{"key":"10.7717\/peerj-cs.1077\/ref-34","article-title":"Storm documentation","author":"Storm A","year":"2014"},{"issue":"1","key":"10.7717\/peerj-cs.1077\/ref-35","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1016\/j.ins.2015.03.027","article-title":"Re-Stream: real-time and energy-efficient resource scheduling in big data stream computing environments","volume":"319","author":"Sun","year":"2015","journal-title":"Information Sciences"},{"key":"10.7717\/peerj-cs.1077\/ref-36","first-page":"160","article-title":"MCD: mutually connected community detection using clustering coefficient approach in social networks","volume":"2019","author":"Tahir","year":"2019"},{"issue":"5","key":"10.7717\/peerj-cs.1077\/ref-37","doi-asserted-by":"publisher","first-page":"571","DOI":"10.1080\/17445760.2019.1585848","article-title":"A review on big data real-time stream processing and its scheduling techniques","volume":"35","author":"Tantalaki","year":"2020","journal-title":"International Journal of Parallel, Emergent and Distributed Systems"},{"key":"10.7717\/peerj-cs.1077\/ref-38","article-title":"Apache Heron","author":"Twitter","year":"2014"},{"key":"10.7717\/peerj-cs.1077\/ref-39","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2930652","article-title":"Energy-efficient harvested-aware clustering and cooperative routing protocol for WBAN (E-HARP)","volume":"7","author":"Ullah","year":"2019","journal-title":"IEEE Access"},{"key":"10.7717\/peerj-cs.1077\/ref-40","first-page":"154","article-title":"Dynamically scaling apache storm for the analysis of streaming data","author":"van der Veen","year":"2015"},{"key":"10.7717\/peerj-cs.1077\/ref-41","first-page":"1363","article-title":"AdaStorm: Resource efficient storm with adaptive configuration","author":"Weng","year":"2017"},{"key":"10.7717\/peerj-cs.1077\/ref-42","first-page":"535","article-title":"T-storm: traffic-aware online scheduling in storm","author":"Xu","year":"2014"},{"key":"10.7717\/peerj-cs.1077\/ref-43","first-page":"154","article-title":"When computing meets heterogeneous cluster: workload assignment in graph computation","volume":"2015","author":"Xue","year":"2015"},{"key":"10.7717\/peerj-cs.1077\/ref-44","first-page":"372","article-title":"The real-time scheduling strategy based on traffic and load balancing in storm","author":"Zhang","year":"2017"}],"container-title":["PeerJ Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/peerj.com\/articles\/cs-1077.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-1077.xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-1077.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/peerj.com\/articles\/cs-1077.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,26]],"date-time":"2022-09-26T08:12:05Z","timestamp":1664179925000},"score":1,"resource":{"primary":{"URL":"https:\/\/peerj.com\/articles\/cs-1077"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,26]]},"references-count":44,"alternative-id":["10.7717\/peerj-cs.1077"],"URL":"https:\/\/doi.org\/10.7717\/peerj-cs.1077","archive":["CLOCKSS","LOCKSS","Portico"],"relation":{},"ISSN":["2376-5992"],"issn-type":[{"value":"2376-5992","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,26]]},"article-number":"e1077"}}