{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T19:44:59Z","timestamp":1784403899824,"version":"3.55.0"},"reference-count":52,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2025,10,30]],"date-time":"2025-10-30T00:00:00Z","timestamp":1761782400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"DOI":"10.13039\/100007431","name":"NRF","doi-asserted-by":"publisher","award":["RS-2025-00555741"],"award-info":[{"award-number":["RS-2025-00555741"]}],"id":[{"id":"10.13039\/100007431","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007431","name":"NRF","doi-asserted-by":"publisher","award":["RS-2025-02372996"],"award-info":[{"award-number":["RS-2025-02372996"]}],"id":[{"id":"10.13039\/100007431","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003052","name":"MOTIE","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100003052","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003661","name":"KIAT","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100003661","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,1,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>This paper addresses a critical research gap in applying deep reinforcement learning to job shop scheduling problems by focusing on state feature design, which has been largely treated as an empirical matter. We present three key contributions: (1) a state feature design framework for reinforcement learning application in scheduling domain: dynamics equation\u00a0(DE) and predicting state value based on nominal reward (PSVNR) perspectives, (2) the concept of \u2018undercooked\u2019 lower bounds that balance computational efficiency with bound tightness, and (3) successful integration of graph attention networks, transformer architecture, and the proposed \u2018undercooked\u2019 lower bounds into a unified framework. Our approach outperforms existing methods on 81$\\%$ of the benchmark data sets, demonstrating exceptional generalizability and scalability. Through extensive experimentation, we explore and analyse the relationship between lower bound tightness and state value prediction accuracy, suggesting a meaningful connection between classical optimization techniques and modern learning-based approaches. The theoretical framework we develop provides clear guidelines for state feature design in reinforcement learning applications to scheduling problems, while our practical implementation shows significant performance improvements over state-of-the-art methods.<\/jats:p>","DOI":"10.1093\/jcde\/qwaf118","type":"journal-article","created":{"date-parts":[[2025,10,30]],"date-time":"2025-10-30T13:12:04Z","timestamp":1761829924000},"page":"24-44","source":"Crossref","is-referenced-by-count":4,"title":["Framework for state features design in job shop scheduling with deep reinforcement learning: Beyond empirical approaches"],"prefix":"10.1093","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-7656-6522","authenticated-orcid":false,"given":"Seung Heon","family":"Oh","sequence":"first","affiliation":[{"name":"Seoul National University Department of Naval Architecture and Ocean Engineering, , 1, Gwanak-ro, Gwanak-gu, Seoul 08826 ,","place":["Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1614-5226","authenticated-orcid":false,"given":"Young-in","family":"Cho","sequence":"additional","affiliation":[{"name":"Seoul National University Department of Naval Architecture and Ocean Engineering, , 1, Gwanak-ro, Gwanak-gu, Seoul 08826 ,","place":["Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyunjin","family":"Oh","sequence":"additional","affiliation":[{"name":"Seoul National University Department of Naval Architecture and Ocean Engineering, , 1, Gwanak-ro, Gwanak-gu, Seoul 08826 ,","place":["Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiwon","family":"Baek","sequence":"additional","affiliation":[{"name":"Seoul National University Department of Naval Architecture and Ocean Engineering, , 1, Gwanak-ro, Gwanak-gu, Seoul 08826 ,","place":["Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seung Woo","family":"Han","sequence":"additional","affiliation":[{"name":"Seoul National University Department of Naval Architecture and Ocean Engineering, , 1, Gwanak-ro, Gwanak-gu, Seoul 08826 ,","place":["Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7612-7361","authenticated-orcid":false,"given":"Jong Hun","family":"Woo","sequence":"additional","affiliation":[{"name":"Seoul National University Department of Naval Architecture and Ocean Engineering, , 1, Gwanak-ro, Gwanak-gu, Seoul 08826 ,","place":["Republic of Korea"]},{"name":"Research Institute of Marine Systems Engineering, Seoul National University , 1, Gwanak-ro, Gwanak-gu, Seoul 08826 ,","place":["Republic of Korea"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,10,30]]},"reference":[{"key":"2026010204363937000_bib1","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1287\/mnsc.34.3.391","article-title":"The shifting bottleneck procedure for job shop scheduling","volume":"34","author":"Adams","year":"1988","journal-title":"Management Science"},{"key":"2026010204363937000_bib2","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1287\/ijoc.3.2.149","article-title":"A computational study of the job-shop scheduling problem","volume":"3","author":"Applegate","year":"1991","journal-title":"ORSA Journal on Computing"},{"key":"2026010204363937000_bib3","doi-asserted-by":"publisher","first-page":"414","DOI":"10.1093\/jcde\/qwac122","article-title":"Heave reduction of payload through crane control based on deep reinforcement learning using dual offshore cranes","volume":"10","author":"Bae","year":"2023","journal-title":"Journal of Computational Design and Engineering"},{"key":"2026010204363937000_bib4","article-title":"Relational inductive biases, deep learning, and graph networks","author":"Battaglia","year":"2018"},{"key":"2026010204363937000_bib5","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1007\/BF01719250","article-title":"A generalized permutation approach to job shop scheduling with genetic algorithms","volume":"17","author":"Bierwirth","year":"1995","journal-title":"Operations-Research-Spektrum"},{"key":"2026010204363937000_bib6","doi-asserted-by":"publisher","first-page":"1322","DOI":"10.1109\/TII.2022.3167380","article-title":"A deep reinforcement learning framework based on an attention mechanism and disjunctive graph embedding for the job-shop scheduling problem","volume":"19","author":"Chen","year":"2022","journal-title":"IEEE Transactions on Industrial Informatics"},{"key":"2026010204363937000_bib7","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1093\/jcde\/qwab068","article-title":"Minimize makespan of permutation flowshop using pointer network","volume":"9","author":"Cho","year":"2022","journal-title":"Journal of Computational Design and Engineering"},{"key":"2026010204363937000_bib8","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1093\/jcde\/qwae002","article-title":"Locating algorithm of steel stock area with asynchronous advantage actor-critic reinforcement learning","volume":"11","author":"Cho","year":"2024","journal-title":"Journal of Computational Design and Engineering"},{"key":"2026010204363937000_bib9","first-page":"105528","article-title":"Self-labeling the job shop scheduling problem","volume-title":"Advances in Neural Information Processing Systems","author":"Corsini","year":"2024"},{"key":"2026010204363937000_bib10","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1016\/S0377-2217(97)00019-2","article-title":"Benchmarks for shop scheduling problems","volume":"109","author":"Demirkol","year":"1998","journal-title":"European Journal of Operational Research"},{"key":"2026010204363937000_bib11","article-title":"A survey of state representation learning for deep reinforcement learning","author":"Echchahed","year":"2025","journal-title":"Transactions on Machine Learning Research"},{"key":"2026010204363937000_bib12","article-title":"Probabilistic learning combinations of local job-shop scheduling rules","author":"Fisher","year":"1963","journal-title":"Industrial Scheduling"},{"key":"2026010204363937000_bib13","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1080\/00207543.2011.571443","article-title":"Distributed policy search reinforcement learning for job-shop scheduling tasks","volume":"50","author":"Gabel","year":"2012","journal-title":"International Journal of Production Research"},{"key":"2026010204363937000_bib14","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1093\/jcde\/qwae066","article-title":"Automated design and optimization of distributed filter circuits using reinforcement learning","volume":"11","author":"Gao","year":"2024","journal-title":"Journal of Computational Design and Engineering"},{"key":"2026010204363937000_bib15","article-title":"Predict then propagate: Graph neural networks meet personalized pagerank","author":"Gasteiger","year":"2018"},{"key":"2026010204363937000_bib16","doi-asserted-by":"publisher","first-page":"1577","DOI":"10.1109\/ICSMC.1994.400072","article-title":"Solving job-shop scheduling problems by genetic algorithm","volume-title":"Proceedings of IEEE International Conference on Systems, Man and Cybernetics","author":"Gen","year":"1994"},{"key":"2026010204363937000_bib17","first-page":"770","article-title":"Deep residual learning for image recognition","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"He","year":"2016"},{"key":"2026010204363937000_bib18","article-title":"A natural policy gradient","volume-title":"Advances in Neural Information Processing Systems","author":"Kakade","year":"2001"},{"key":"2026010204363937000_bib19","doi-asserted-by":"publisher","first-page":"318","DOI":"10.1093\/jcde\/qwad001","article-title":"Application of reinforcement learning based on curriculum learning for the pipe auto-routing of ships","volume":"10","author":"Kim","year":"2023","journal-title":"Journal of Computational Design and Engineering"},{"key":"2026010204363937000_bib20","article-title":"Semi-supervised classification with graph convolutional networks","volume":"abs\/1609.02907","author":"Kipf","year":"2016","journal-title":"CoRR"},{"key":"2026010204363937000_bib21","article-title":"Resource constrained project scheduling: an experimental investigation of heuristic scheduling techniques","author":"Lawrance","year":"1984","journal-title":"GSIA, Carnegie Mellon University"},{"key":"2026010204363937000_bib22","first-page":"741","article-title":"Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model","volume":"33","author":"Lee","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2026010204363937000_bib23","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.1109\/TII.2023.3272661","article-title":"Large-scale dynamic scheduling for flexible job-shop with random arrivals of new jobs by hierarchical reinforcement learning","volume":"20","author":"Lei","year":"2023","journal-title":"IEEE Transactions on Industrial Informatics"},{"key":"2026010204363937000_bib24","doi-asserted-by":"publisher","first-page":"3222","DOI":"10.1109\/SMC53654.2022.9945585","article-title":"Learning to schedule job-shop problems via hierarchical reinforcement learning","volume-title":"2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC)","author":"Liao","year":"2022"},{"key":"2026010204363937000_bib25","doi-asserted-by":"publisher","first-page":"71752","DOI":"10.1109\/ACCESS.2020.2987820","article-title":"Actor-critic deep reinforcement learning for solving job shop scheduling problems","volume":"8","author":"Liu","year":"2020","journal-title":"IEEE Access"},{"key":"2026010204363937000_bib26","doi-asserted-by":"publisher","first-page":"8662","DOI":"10.1109\/TII.2024.3371489","article-title":"Dynamic job-shop scheduling via graph attention networks and deep reinforcement learning","volume":"20","author":"Liu","year":"2024","journal-title":"IEEE Transactions on Industrial Informatics"},{"key":"2026010204363937000_bib27","doi-asserted-by":"publisher","first-page":"106294","DOI":"10.1016\/j.cor.2023.106294","article-title":"A deep multi-agent reinforcement learning approach to solve dynamic job shop scheduling problem","volume":"159","author":"Liu","year":"2023","journal-title":"Computers & Operations Research"},{"key":"2026010204363937000_bib28","article-title":"Awac: Accelerating online reinforcement learning with offline datasets","author":"Nair","year":"2020"},{"key":"2026010204363937000_bib29","doi-asserted-by":"publisher","first-page":"1157","DOI":"10.1093\/jcde\/qwac044","article-title":"Distributional reinforcement learning with the independent learners for flexible job shop scheduling problem with high variability","volume":"9","author":"Oh","year":"2022","journal-title":"Journal of Computational Design and Engineering"},{"key":"2026010204363937000_bib30","first-page":"1420","article-title":"A reinforcement learning approach to robust scheduling of semiconductor manufacturing facilities","volume":"17","author":"Park","year":"2019","journal-title":"IEEE Transactions on Automation Science and Engineering"},{"key":"2026010204363937000_bib31","article-title":"Schedulenet: Learn to solve multi-agent scheduling problems with reinforcement learning","author":"Park","year":"2021"},{"key":"2026010204363937000_bib32","doi-asserted-by":"publisher","first-page":"3360","DOI":"10.1080\/00207543.2020.1870013","article-title":"Learning to schedule job-shop problems: representation and policy learning using graph neural network and reinforcement learning","volume":"59","author":"Park","year":"2021","journal-title":"International Journal of Production Research"},{"key":"2026010204363937000_bib33","doi-asserted-by":"publisher","first-page":"1127","DOI":"10.1287\/mnsc.24.11.1127","article-title":"Modified policy iteration algorithms for discounted markov decision problems","volume":"24","author":"Puterman","year":"1978","journal-title":"Management Science"},{"key":"2026010204363937000_bib34","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1109\/TNN.2008.2005605","article-title":"The graph neural network model","volume":"20","author":"Scarselli","year":"2008","journal-title":"IEEE Transactions on Neural Networks"},{"key":"2026010204363937000_bib35","article-title":"Artificial intelligence in combat decision-making: weapon target assignment via reinforcement learning and graph neural networks","author":"Seung\u00a0Heon","year":"2024","journal-title":"Authorea Preprints"},{"key":"2026010204363937000_bib36","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1093\/jcde\/qwaf045","article-title":"Sd2ac: A reinforcement learning framework using distribution evaluation and sequential decision-making for ucav combat","volume":"12","author":"Shi","year":"2025","journal-title":"Journal of Computational Design and Engineering"},{"key":"2026010204363937000_bib37","doi-asserted-by":"publisher","first-page":"8260","DOI":"10.1080\/00207543.2024.2335663","article-title":"An efficient and adaptive design of reinforcement learning environment to solve job shop scheduling problem with soft actor-critic algorithm","volume":"62","author":"Si","year":"2024","journal-title":"International Journal of Production Research"},{"key":"2026010204363937000_bib38","doi-asserted-by":"publisher","first-page":"1600","DOI":"10.1109\/TII.2022.3189725","article-title":"Flexible job-shop scheduling via graph neural network and deep reinforcement learning","volume":"19","author":"Song","year":"2022","journal-title":"IEEE Transactions on Industrial Informatics"},{"key":"2026010204363937000_bib39","doi-asserted-by":"publisher","first-page":"1495","DOI":"10.1287\/mnsc.38.10.1495","article-title":"New search spaces for sequencing problems with application to job shop scheduling","volume":"38","author":"Storer","year":"1992","journal-title":"Management Science"},{"key":"2026010204363937000_bib40","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"2018"},{"key":"2026010204363937000_bib41","article-title":"Policy gradient methods for reinforcement learning with function approximation","volume-title":"Advances in Neural Information Processing Systems","author":"Sutton","year":"1999"},{"key":"2026010204363937000_bib42","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1016\/0377-2217(93)90182-M","article-title":"Benchmarks for basic scheduling problems","volume":"64","author":"Taillard","year":"1993","journal-title":"European Journal of Operational Research"},{"key":"2026010204363937000_bib43","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2026010204363937000_bib44","article-title":"Graph attention networks","author":"Veli\u010dkovi\u0107","year":"2017"},{"key":"2026010204363937000_bib45","doi-asserted-by":"publisher","first-page":"107969","DOI":"10.1016\/j.comnet.2021.107969","article-title":"Dynamic job-shop scheduling in smart manufacturing using deep reinforcement learning","volume":"190","author":"Wang","year":"2021","journal-title":"Computer Networks"},{"key":"2026010204363937000_bib46","first-page":"1","article-title":"Design patterns of deep reinforcement learning models for job shop scheduling problems","author":"Wang","year":"2024","journal-title":"Journal of Intelligent Manufacturing"},{"key":"2026010204363937000_bib47","first-page":"281","article-title":"A genetic algorithm applicable to large-scale job-shop problems","volume-title":"Parallel Problem Solving from Nature","author":"Yamada","year":"1992"},{"key":"2026010204363937000_bib48","doi-asserted-by":"publisher","first-page":"110436","DOI":"10.1016\/j.asoc.2023.110436","article-title":"Solving job shop scheduling problems via deep reinforcement learning","volume":"143","author":"Yuan","year":"2023","journal-title":"Applied Soft Computing"},{"key":"2026010204363937000_bib50","article-title":"Deep reinforcement learning guided improvement heuristic for job shop scheduling","volume-title":"The Twelfth International Conference on Learning Representations","author":"Zhang","year":"2024"},{"key":"2026010204363937000_bib49","first-page":"1621","article-title":"Learning to dispatch for job shop scheduling via deep reinforcement learning","volume":"33","author":"Zhang","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2026010204363937000_bib51","doi-asserted-by":"publisher","first-page":"614","DOI":"10.1016\/j.jmsy.2024.08.015","article-title":"A novel soft actor\u2013critic framework with disjunctive graph embedding and autoencoder mechanism for job shop scheduling problems","volume":"76","author":"Zhang","year":"2024","journal-title":"Journal of Manufacturing Systems"},{"key":"2026010204363937000_bib52","doi-asserted-by":"publisher","first-page":"1315","DOI":"10.1016\/j.cor.2011.07.019","article-title":"Minimizing mean weighted tardiness in unrelated parallel machine scheduling with reinforcement learning","volume":"39","author":"Zhang","year":"2012","journal-title":"Computers & Operations Research"}],"container-title":["Journal of Computational Design and Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jcde\/advance-article-pdf\/doi\/10.1093\/jcde\/qwaf118\/65042073\/qwaf118.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/13\/1\/24\/65042073\/qwaf118.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jcde\/article-pdf\/13\/1\/24\/65042073\/qwaf118.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,2]],"date-time":"2026-01-02T09:36:46Z","timestamp":1767346606000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jcde\/article\/13\/1\/24\/8306862"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,30]]},"references-count":52,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,1,2]]}},"URL":"https:\/\/doi.org\/10.1093\/jcde\/qwaf118","relation":{},"ISSN":["2288-5048"],"issn-type":[{"value":"2288-5048","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,1]]},"published":{"date-parts":[[2025,10,30]]}}}