{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T22:06:55Z","timestamp":1790806015132,"version":"4.1.0"},"reference-count":25,"publisher":"Springer Science and Business Media LLC","issue":"5-6","license":[{"start":{"date-parts":[[2023,12,1]],"date-time":"2023-12-01T00:00:00Z","timestamp":1701388800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T00:00:00Z","timestamp":1702857600000},"content-version":"vor","delay-in-days":17,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"University of Bern"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Informatik Spektrum"],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The operation of inventory systems plays an important role in the success of manufacturing companies, making it a\u00a0highly relevant domain for optimization. In particular, the domain lends itself to being approached via Deep Reinforcement Learning (DRL) models due to it requiring sequential reorder decisions based on uncertainty to minimize cost. In this paper, we evaluate state-of-the-art optimization approaches to determine whether Deep Reinforcement Learning can be applied to the multi-echelon inventory optimization (MEIO) framework in a\u00a0practically feasible manner to generate fully dynamic reorder policies. We investigate how it performs in comparison to an optimized static reorder policy, how robust it is when it comes to structural changes in the environment, and whether the use of DRL is safe in terms of risk in real-world applications. Our results show promising performance for DRL with potential for improvement in terms of minimizing risky behavior.<\/jats:p>","DOI":"10.1007\/s00287-023-01556-6","type":"journal-article","created":{"date-parts":[[2023,12,18]],"date-time":"2023-12-18T12:02:02Z","timestamp":1702900922000},"page":"240-251","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Fully dynamic reorder policies with deep reinforcement learning for multi-echelon inventory management"],"prefix":"10.1007","volume":"46","author":[{"given":"Patric","family":"Hammler","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicolas","family":"Riesterer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Torsten","family":"Braun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,12,18]]},"reference":[{"issue":"3","key":"1556_CR1","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1111\/j.1745-493X.2012.03276.x","volume":"48","author":"BM Greer","year":"2012","unstructured":"Greer\u00a0BM, Theuri\u00a0P (2012) Linking supply chain management superiority to multifaceted firm financial performance. J\u00a0Supply Chain Manag 48(3):97\u2013106","journal-title":"J Supply Chain Manag"},{"issue":"13","key":"1556_CR2","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1108\/IJOPM-12-2021-0757","volume":"42","author":"J Matinheikki","year":"2022","unstructured":"Matinheikki\u00a0J, Kauppi\u00a0K, Brandon-Jones\u00a0A, Raaij\u00a0EM (2022) Making agency theory work for supply chain relationships: a\u00a0systematic review across four disciplines. Int J Oper Prod Manag 42(13):299\u2013334","journal-title":"Int J Oper Prod Manag"},{"issue":"3","key":"1556_CR3","doi-asserted-by":"publisher","first-page":"955","DOI":"10.1016\/j.ejor.2018.02.047","volume":"269","author":"T Kok","year":"2018","unstructured":"Kok\u00a0T, Grob\u00a0C, Laumanns\u00a0M, Minner\u00a0S, Rambau\u00a0J, Schade\u00a0K (2018) A\u00a0typology and literature review on stochastic multi-echelon inventory models. Eur J Oper Res 269(3):955\u2013983","journal-title":"Eur J Oper Res"},{"key":"1556_CR4","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1007\/978-3-031-11814-2_5","volume-title":"Quantitative models in life science business: from value creation to business processes","author":"P Hammler","year":"2023","unstructured":"Hammler\u00a0P, Riesterer\u00a0N, Mu\u00a0G, Braun\u00a0T (2023) Multi-echelon inventory optimization using deep reinforcement learning. In: Canci\u00a0JK, Mekler\u00a0P, Mu\u00a0G (eds) Quantitative models in life science business: from value creation to business processes. Springer, Cham, pp 73\u201393 https:\/\/doi.org\/10.1007\/978-3-031-11814-2_5"},{"issue":"5","key":"1556_CR5","doi-asserted-by":"publisher","first-page":"597","DOI":"10.1287\/mnsc.40.5.597","volume":"40","author":"WH Hausman","year":"1994","unstructured":"Hausman\u00a0WH, Erkip\u00a0NK (1994) Multi-echelon vs. single-echelon inventory control policies for low-demand items. Manage Sci 40(5):597\u2013602","journal-title":"Manage Sci"},{"issue":"4","key":"1556_CR6","doi-asserted-by":"publisher","first-page":"475","DOI":"10.1287\/mnsc.6.4.475","volume":"6","author":"AJ Clark","year":"1960","unstructured":"Clark\u00a0AJ, Scarf\u00a0H (1960) Optimal policies for a\u00a0multi-echelon inventory problem. Manage Sci 6(4):475\u2013490","journal-title":"Manage Sci"},{"key":"1556_CR7","volume-title":"Math programming based reinforcement learning for multi-echelon inventory management","author":"P Harsha","year":"2021","unstructured":"Harsha\u00a0P, Jagmohan\u00a0A, Kalagnanam\u00a0JR, Quanz\u00a0B, Singhvi\u00a0D (2021) Math programming based reinforcement learning for multi-echelon inventory management (arXiv preprint arXiv:2112.02215)"},{"issue":"1","key":"1556_CR8","doi-asserted-by":"publisher","first-page":"102","DOI":"10.3390\/pr9010102","volume":"9","author":"HD Perez","year":"2021","unstructured":"Perez\u00a0HD, Hubbs\u00a0CD, Li\u00a0C, Grossmann\u00a0IE (2021) Algorithmic approaches to inventory management optimization. Processes 9(1):102","journal-title":"Processes"},{"issue":"2","key":"1556_CR9","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1016\/S0272-6963(96)00098-8","volume":"15","author":"R Metters","year":"1997","unstructured":"Metters\u00a0R (1997) Quantifying the bullwhip effect in supply chains. J\u00a0Oper Manag 15(2):89\u2013100","journal-title":"J Oper Manag"},{"key":"1556_CR10","volume-title":"Reinforcement learning for multi-product multi-node inventory management in supply chains","author":"NN Sultana","year":"2020","unstructured":"Sultana\u00a0NN, Meisheri\u00a0H, Baniwal\u00a0V, Nath\u00a0S, Ravindran\u00a0B, Khadilkar\u00a0H (2020) Reinforcement learning for multi-product multi-node inventory management in supply chains (arXiv preprint arXiv:2006.04037)"},{"key":"1556_CR11","doi-asserted-by":"publisher","DOI":"10.1287\/msom.2021.1064","author":"J Gijsbrechts","year":"2021","unstructured":"Gijsbrechts\u00a0J, Boute\u00a0RN, Van Mieghem\u00a0JA, Zhang\u00a0D (2021) Can deep reinforcement learning improve inventory management? performance on dual sourcing, lost sales and multi-echelon problems. Manuf Serv Oper Manag. https:\/\/doi.org\/10.1287\/msom.2021.1064","journal-title":"Manuf Serv Oper Manag"},{"key":"1556_CR12","series-title":"Reinforcement Learning","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1007\/978-3-642-27645-3_1","volume-title":"Reinforcement learning and markov decision processes","author":"M van Otterlo","year":"2012","unstructured":"van Otterlo\u00a0M, Wiering\u00a0M (2012) Reinforcement learning and markov decision processes. Reinforcement Learning. Springer, pp\u00a03\u201342"},{"issue":"1","key":"1556_CR13","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1007\/s10479-005-6241-9","volume":"135","author":"FY Chen","year":"2005","unstructured":"Chen\u00a0FY, Wang\u00a0T, Xu\u00a0TZ (2005) Integrated inventory replenishment and temporal shipment consolidation: a\u00a0comparison of quantity-based and time-based models. Ann Oper Res 135(1):197\u2013210","journal-title":"Ann Oper Res"},{"key":"1556_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.compchemeng.2014.10.008","volume":"73","author":"Y Chu","year":"2015","unstructured":"Chu\u00a0Y, You\u00a0F, Wassick\u00a0JM, Agarwal\u00a0A (2015) Simulation-based optimization framework for multi-echelon inventory systems under uncertainty. Comput Chem Eng 73:1\u201316. https:\/\/doi.org\/10.1016\/j.compchemeng.2014.10.008","journal-title":"Comput Chem Eng"},{"issue":"6","key":"1556_CR15","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1109\/MSP.2017.2743240","volume":"34","author":"K Arulkumaran","year":"2017","unstructured":"Arulkumaran\u00a0K, Deisenroth\u00a0MP, Brundage\u00a0M, Bharath\u00a0AA (2017) Deep reinforcement learning: a\u00a0brief survey. IEEE Signal Process Mag 34(6):26\u201338","journal-title":"IEEE Signal Process Mag"},{"key":"1556_CR16","volume-title":"Reinforcement learning: an introduction","author":"RS Sutton","year":"2018","unstructured":"Sutton\u00a0RS, Barto\u00a0AG (2018) Reinforcement learning: an introduction, 2nd\u00a0edn. MIT Press","edition":"2"},{"key":"1556_CR17","volume-title":"Asynchronous methods for deep reinforcement learning","author":"V Mnih","year":"2016","unstructured":"Mnih\u00a0V, Badia\u00a0AP, Mirza\u00a0M, Graves\u00a0A, Lillicrap\u00a0TP, Harley\u00a0T, Silver\u00a0D, Kavukcuoglu\u00a0K (2016) Asynchronous methods for deep reinforcement learning"},{"key":"1556_CR18","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.1707.06347","volume-title":"Proximal policy optimization algorithms","author":"J Schulman","year":"2017","unstructured":"Schulman\u00a0J, Wolski\u00a0F, Dhariwal\u00a0P, Radford\u00a0A, Klimov\u00a0O (2017) Proximal policy optimization algorithms https:\/\/doi.org\/10.48550\/ARXIV.1707.06347"},{"key":"1556_CR19","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.1509.02971","volume-title":"Continuous control with deep reinforcement learning","author":"TP Lillicrap","year":"2015","unstructured":"Lillicrap\u00a0TP, Hunt\u00a0JJ, Pritzel\u00a0A, Heess\u00a0N, Erez\u00a0T, Tassa\u00a0Y, Silver\u00a0D, Wierstra\u00a0D (2015) Continuous control with deep reinforcement learning https:\/\/doi.org\/10.48550\/ARXIV.1509.02971"},{"key":"1556_CR20","series-title":"International conference on machine learning","first-page":"1587","volume-title":"Addressing function approximation error in actor-critic methods","author":"S Fujimoto","year":"2018","unstructured":"Fujimoto\u00a0S, Hoof\u00a0H, Meger\u00a0D (2018) Addressing function approximation error in actor-critic methods. International conference on machine learning, pp 1587\u20131596"},{"key":"1556_CR21","series-title":"Proceedings of the 36th\u00a0IEEE Conference on Decision and Control","first-page":"4052","volume-title":"A\u00a0neuro-dynamic programming approach to retailer inventory management","author":"B Van Roy","year":"1997","unstructured":"Van Roy\u00a0B, Bertsekas\u00a0DP, Lee\u00a0Y, Tsitsiklis\u00a0JN (1997) A\u00a0neuro-dynamic programming approach to retailer inventory management. Proceedings of the 36th\u00a0IEEE Conference on Decision and Control, vol 4. IEEE, pp 4052\u20134057"},{"issue":"1","key":"1556_CR22","doi-asserted-by":"publisher","first-page":"480","DOI":"10.1016\/j.ejor.2005.01.037","volume":"174","author":"S Axs\u00e4ter","year":"2006","unstructured":"Axs\u00e4ter\u00a0S (2006) A\u00a0simple procedure for determining order quantities under a\u00a0fill rate constraint and normally distributed lead-time demand. Eur J Oper Res 174(1):480\u2013491","journal-title":"Eur J Oper Res"},{"issue":"268","key":"1556_CR23","first-page":"1","volume":"22","author":"A Raffin","year":"2021","unstructured":"Raffin\u00a0A, Hill\u00a0A, Gleave\u00a0A, Kanervisto\u00a0A, Ernestus\u00a0M, Dormann\u00a0N (2021) Stable-baselines3: reliable reinforcement learning implementations. J\u00a0Mach Learn Res 22(268):1\u20138","journal-title":"J Mach Learn Res"},{"key":"1556_CR24","series-title":"Proceedings of the 25rd\u00a0ACM SIGKDD international conference on knowledge discovery and data mining","first-page":"2623","volume-title":"Optuna: a\u00a0nextgeneration hyperparameter optimization framework","author":"T Akiba","year":"2019","unstructured":"Akiba\u00a0T, Sano\u00a0S, Yanase\u00a0T, Ohta\u00a0T, Koyama\u00a0M (2019) Optuna: a\u00a0nextgeneration hyperparameter optimization framework. Proceedings of the 25rd\u00a0ACM SIGKDD international conference on knowledge discovery and data mining, pp\u00a02623\u20132631"},{"key":"1556_CR25","volume-title":"A2c is a\u00a0special case of ppo","author":"S Huang","year":"2022","unstructured":"Huang\u00a0S, Kanervisto\u00a0A, Raffin\u00a0A, Wang\u00a0W, Onta\u00f1\u00f3n\u00a0S, Dossa\u00a0RFJ (2022) A2c is a\u00a0special case of ppo (arXiv preprint arXiv:2205.09123)"}],"container-title":["Informatik Spektrum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00287-023-01556-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00287-023-01556-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00287-023-01556-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,17]],"date-time":"2024-01-17T15:03:11Z","timestamp":1705503791000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00287-023-01556-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12]]},"references-count":25,"journal-issue":{"issue":"5-6","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["1556"],"URL":"https:\/\/doi.org\/10.1007\/s00287-023-01556-6","relation":{},"ISSN":["0170-6012","1432-122X"],"issn-type":[{"value":"0170-6012","type":"print"},{"value":"1432-122X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12]]},"assertion":[{"value":"4 November 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 December 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"P.\u00a0Hammler, N.\u00a0Riesterer, and T.\u00a0Braun declare that they have no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}