{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,9]],"date-time":"2026-01-09T18:26:50Z","timestamp":1767983210917,"version":"3.49.0"},"reference-count":39,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2022,7,7]],"date-time":"2022-07-07T00:00:00Z","timestamp":1657152000000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Korean Ministry of Trade, Industry and Energy, Republic of Korea"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,7,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Multi-agent scheduling algorithm is a useful method for the flexible job shop scheduling problem (FJSP). Also, the variability of the target system has to be considered in the scheduling problem that includes the machine failure, the setup change, etc. This study proposes the scheduling method that combines the independent learners with the implicit quantile network by modeling of the FJSP with high variability to the form of the multi-agent. The proposed method demonstrates superior performance compared to the several known heuristic dispatching rules. In addition, the trained model exhibits superior performance compared to the reinforcement learning algorithms such as proximal policy optimization and deep Q-network.<\/jats:p>","DOI":"10.1093\/jcde\/qwac044","type":"journal-article","created":{"date-parts":[[2022,5,11]],"date-time":"2022-05-11T11:43:27Z","timestamp":1652269407000},"page":"1157-1174","source":"Crossref","is-referenced-by-count":30,"title":["Distributional reinforcement learning with the independent learners for flexible job shop scheduling problem with high variability"],"prefix":"10.1093","volume":"9","author":[{"given":"Seung Heon","family":"Oh","sequence":"first","affiliation":[{"name":"Department of Naval Architecture and Ocean Engineering, Seoul National University , Seoul 08826, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Young In","family":"Cho","sequence":"additional","affiliation":[{"name":"Department of Naval Architecture and Ocean Engineering, Seoul National University , Seoul 08826, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jong Hun","family":"Woo","sequence":"additional","affiliation":[{"name":"Department of Naval Architecture and Ocean Engineering, Seoul National University , Seoul 08826, Republic of Korea"},{"name":"Research Institute of Marine Systems Engineering, Seoul National University , Seoul 08826, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2022,7,6]]},"reference":[{"key":"2022083110581990600_bib6","doi-asserted-by":"crossref","first-page":"805","DOI":"10.1023\/B:JIMS.0000042665.10086.cf","article-title":"A simulated annealing algorithm for multi-agent systems: A job-shop scheduling application","volume":"15","author":"Aydin","year":"2004","journal-title":"Journal of Intelligent Manufacturing"},{"key":"2022083110581990600_bib8","doi-asserted-by":"crossref","first-page":"761","DOI":"10.1093\/jcde\/qwaa055","article-title":"Production planning and scheduling problem of continuous parallel lines with demand uncertainty and different production capacities","volume":"7","author":"Bhosale","year":"2020","journal-title":"Journal of Computational Design and Engineering"},{"key":"2022083110581990600_bib4","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1016\/S0925-2312(00)00344-1","article-title":"Competitive neural network to solve scheduling problems","volume":"37","author":"Chen","year":"2001","journal-title":"Neurocomputing"},{"key":"2022083110581990600_bib10","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1093\/jcde\/qwab068","article-title":"Minimize makespan of permutation flowshop using pointer network","volume":"9","author":"Cho","year":"2022","journal-title":"Journal of Computational Design and Engineering"},{"key":"2022083110581990600_bib29","first-page":"746","article-title":"The dynamics of reinforcement learning in cooperative multiagent systems","volume-title":"Proceedings of the Fifteenth National\/Tenth Conference on Artificial Intelligence\/Innovative Applications of Artificial Intelligence","author":"Claus","year":"1998"},{"key":"2022083110581990600_bib38","article-title":"Quantifying generalization in reinforcement learning","volume-title":"Proceedings of the 36th International Conference on Machine Learning","author":"Cobbe","year":"2019"},{"key":"2022083110581990600_bib32","article-title":"Implicit quantile networks for distributional reinforcement learning","volume-title":"Proceedings of the 35th International Conference on Machine Learning","author":"Dabney","year":"2018"},{"key":"2022083110581990600_bib27","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1016\/j.cirp.2020.04.005","article-title":"Cooperative multi-agent system for production control using reinforcement learning","volume":"69","author":"Dittrich","year":"2020","journal-title":"CIRP Annals"},{"key":"2022083110581990600_bib23","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1007\/s10458-008-9031-3","article-title":"New local diversification techniques for flexible job shop scheduling problem with a multi-agent approach","volume":"17","author":"Ennigrou","year":"2008","journal-title":"Autonomous Agents and Multi-Agent Systems"},{"key":"2022083110581990600_bib2","first-page":"1333","article-title":"Reinforcement learning for DEC-MDPs with changing action sets and partially ordered dependencies","volume-title":"AAMAS","author":"Gabel","year":"2008"},{"key":"2022083110581990600_bib3","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1080\/00207543.2011.571443","article-title":"Distributed policy search reinforcement learning for job-shop scheduling tasks","volume":"50","author":"Gabel","year":"2012","journal-title":"International Journal of Production Research"},{"key":"2022083110581990600_bib5","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1016\/S0278-6125(97)85674-9","article-title":"A tabu search approach to scheduling an automated wet etch station","volume":"16","author":"Geiger","year":"1997","journal-title":"Journal of Manufacturing Systems"},{"key":"2022083110581990600_bib31","first-page":"66","article-title":"Cooperative multi-agent control using deep reinforcement learning","volume-title":"Autonomous Agents and Multiagent Systems. AAMAS 2017. Lecture Notes in Computer Science","author":"Gupta","year":"2017"},{"key":"2022083110581990600_bib24","first-page":"385","article-title":"Particle swarm optimization combined with tabu search in a multi-agent model for flexible job shop problem","volume-title":"Advances in Swarm Intelligence. ICSI 2013. Lecture Notes in Computer Science","author":"Henchiri","year":"2013"},{"key":"2022083110581990600_bib39","article-title":"A closer look at invalid action masking in policy gradient algorithms","author":"Huang","year":"2020"},{"key":"2022083110581990600_bib36","article-title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","volume-title":"Proceedings of the 32nd International Conference on Machine Learning","author":"Ioffe","year":"2015"},{"key":"2022083110581990600_bib21","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1080\/00207543.2010.539276","article-title":"Multi-agent job shop scheduling system based on co-operative approach of idle time minimisation","volume":"50","author":"Kouider","year":"2012","journal-title":"International Journal of Production Research"},{"key":"2022083110581990600_bib35","article-title":"Continuous control with deep reinforcement learning","author":"Lillicrap","year":"2015"},{"key":"2022083110581990600_bib15","doi-asserted-by":"crossref","first-page":"4276","DOI":"10.1109\/TII.2019.2908210","article-title":"Smart manufacturing scheduling with edge computing using multiclass deep q network","volume":"15","author":"Lin","year":"2019","journal-title":"IEEE Transactions on Industrial Informatics"},{"key":"2022083110581990600_bib16","doi-asserted-by":"crossref","first-page":"71752","DOI":"10.1109\/ACCESS.2020.2987820","article-title":"Actor-critic deep reinforcement learning for solving job shop scheduling problems","volume":"8","author":"Liu","year":"2020","journal-title":"IEEE Access"},{"key":"2022083110581990600_bib22","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1007\/s00170-011-3482-4","article-title":"Multi-agent-based proactive\u2013reactive scheduling for a job shop","volume":"59","author":"Lou","year":"2012","journal-title":"The International Journal of Advanced Manufacturing Technology"},{"key":"2022083110581990600_bib19","doi-asserted-by":"crossref","first-page":"106208","DOI":"10.1016\/j.asoc.2020.106208","article-title":"Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning","volume":"91","author":"Luo","year":"2020","journal-title":"Applied Soft Computing"},{"key":"2022083110581990600_bib37","article-title":"Towards understanding regularization in batch normalization","author":"Luo","year":"2018"},{"key":"2022083110581990600_bib20","doi-asserted-by":"crossref","first-page":"107489","DOI":"10.1016\/j.cie.2021.107489","article-title":"Dynamic multi-objective scheduling for flexible job shop by deep reinforcement learning","volume":"159","author":"Luo","year":"2021","journal-title":"Computers & Industrial Engineering"},{"key":"2022083110581990600_bib28","article-title":"Likelihood quantile networks for coordinating multi-agent reinforcement learning","author":"Lyu","year":"2018"},{"key":"2022083110581990600_bib25","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1504\/IJIEI.2018.091875","article-title":"Multi-agent model based on combination of chemical reaction optimisation metaheuristic with tabu search for flexible job shop scheduling problem","volume":"6","author":"Marzouki","year":"2018","journal-title":"International Journal of Intelligent Engineering Informatics"},{"key":"2022083110581990600_bib30","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1017\/S0269888912000057","article-title":"Independent reinforcement learners in cooperative Markov games: A survey regarding coordination problems","volume":"27","author":"Matignon","year":"2012","journal-title":"The Knowledge Engineering Review"},{"key":"2022083110581990600_bib26","first-page":"505","article-title":"Job-shop scheduling with genetic programming","volume-title":"Proceedings of the 2nd Annual Conference on Genetic and Evolutionary Computation","author":"Miyashita","year":"2000"},{"key":"2022083110581990600_bib9","doi-asserted-by":"crossref","first-page":"101390","DOI":"10.1109\/ACCESS.2021.3097254","article-title":"Deep reinforcement learning for minimizing tardiness in parallel machine scheduling with sequence dependent family setups","volume":"9","author":"Paeng","year":"2021","journal-title":"IEEE Access"},{"key":"2022083110581990600_bib18","first-page":"1420","article-title":"A reinforcement learning approach to robust scheduling of semiconductor manufacturing facilities","volume":"17","author":"Park","year":"2019","journal-title":"IEEE Transactions on Automation Science and Engineering"},{"key":"2022083110581990600_bib17","doi-asserted-by":"crossref","first-page":"3360","DOI":"10.1080\/00207543.2020.1870013","article-title":"Learning to schedule job-shop problems: Representation and policy learning using graph neural network and reinforcement learning","volume":"59","author":"Park","year":"2021","journal-title":"International Journal of Production Research"},{"key":"2022083110581990600_bib7","doi-asserted-by":"crossref","first-page":"3202","DOI":"10.1016\/j.cor.2007.02.014","article-title":"A genetic algorithm for the flexible job-shop scheduling problem","volume":"35","author":"Pezzella","year":"2008","journal-title":"Computers & Operations Research"},{"key":"2022083110581990600_bib1","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4614-2361-4","volume-title":"Scheduling","author":"Pinedo","year":"2012"},{"key":"2022083110581990600_bib13","first-page":"764","article-title":"A neural reinforcement learning approach to learn local dispatching policies in production scheduling","volume-title":"Proceedings of the 16th international joint conference on Artificial intelligence","author":"Riedmiller","year":"1999"},{"key":"2022083110581990600_bib40","article-title":"High-dimensional continuous control using generalized advantage estimation","author":"Schulman","year":"2015"},{"key":"2022083110581990600_bib34","article-title":"Value-decomposition networks for cooperative multi-agent learning","author":"Sunehag","year":"2017"},{"key":"2022083110581990600_bib33","volume-title":"Reinforcement learning: An introduction","author":"Sutton","year":"2018"},{"key":"2022083110581990600_bib11","first-page":"1114","article-title":"A reinforcement learning approach to job-shop scheduling","volume":"95","author":"Zhang","year":"1995","journal-title":"IJCAI"},{"key":"2022083110581990600_bib12","first-page":"1024","article-title":"High-performance job-shop scheduling with a timedelay TD (\u03bb) network","volume":"8","author":"Zhang","year":"1996","journal-title":"Advances in Neural Information Processing Systems"}],"container-title":["Journal of Computational Design and Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/9\/4\/1157\/45630626\/qwac044.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/9\/4\/1157\/45630626\/qwac044.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,8,31]],"date-time":"2022-08-31T10:59:06Z","timestamp":1661943546000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jcde\/article\/9\/4\/1157\/6633004"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,6]]},"references-count":39,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,7,6]]}},"URL":"https:\/\/doi.org\/10.1093\/jcde\/qwac044","relation":{},"ISSN":["2288-5048"],"issn-type":[{"value":"2288-5048","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,8]]},"published":{"date-parts":[[2022,7,6]]}}}