{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,13]],"date-time":"2025-11-13T06:37:13Z","timestamp":1763015833104,"version":"3.37.3"},"reference-count":29,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2021,4,20]],"date-time":"2021-04-20T00:00:00Z","timestamp":1618876800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,4,20]],"date-time":"2021-04-20T00:00:00Z","timestamp":1618876800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Ulster University (GB) VCRS","award":["E4G3EN3G39F0A4"],"award-info":[{"award-number":["E4G3EN3G39F0A4"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["SN COMPUT. SCI."],"published-print":{"date-parts":[[2021,5]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Reinforcement learning (RL) problems with continuous states and discrete actions (CSDA) can be found in classic examples such as Cart Pole and Puck World, as well as real world applications such as Market Making. Solutions to CSDA problems typically involve a function approximation (FA) of the mapping from states to actions and can be linear or nonlinear. Linear FAs such as tile-coding (Sutton and Barto in Reinforcement learning, 2nd ed, 2009) suffer from state information loss due to state discretization, whilst non-linear FAs such as DQN (Mnih et al. in Playing atari with deep reinforcement learning,<jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/arxiv.org\/abs\/1312.5602\">https:\/\/arxiv.org\/abs\/1312.5602<\/jats:ext-link>, 2013) are practically infeasible in infinitely large state spaces due to their cubic time complexity (<jats:inline-formula><jats:alternatives><jats:tex-math>$$O(n^3)$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mrow><mml:mi>O<\/mml:mi><mml:mo>(<\/mml:mo><mml:msup><mml:mi>n<\/mml:mi><mml:mn>3<\/mml:mn><\/mml:msup><mml:mo>)<\/mml:mo><\/mml:mrow><\/mml:math><\/jats:alternatives><\/jats:inline-formula>). In this paper, we propose a novel, general solution to CSDA problems, called Gaussian distribution based non-linear function approximation (GBNLFA). Experimentation on three CSDA RL problems (Cart Pole, Puck World, Market Marking) demonstrates the superiority of GBNLFA over state-of-the-art FAs, namely tile-coding and DQN. In particular, GBNLFA resolves the state information loss problem with linear FAs and provides an asymptotically faster algorithm (<jats:italic>O<\/jats:italic>(<jats:italic>n<\/jats:italic>)) than linear FAs (<jats:inline-formula><jats:alternatives><jats:tex-math>$$O(n^2)$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mrow><mml:mi>O<\/mml:mi><mml:mo>(<\/mml:mo><mml:msup><mml:mi>n<\/mml:mi><mml:mn>2<\/mml:mn><\/mml:msup><mml:mo>)<\/mml:mo><\/mml:mrow><\/mml:math><\/jats:alternatives><\/jats:inline-formula>) and neural network based nonlinear FAs (<jats:inline-formula><jats:alternatives><jats:tex-math>$$O(n^3)$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\"><mml:mrow><mml:mi>O<\/mml:mi><mml:mo>(<\/mml:mo><mml:msup><mml:mi>n<\/mml:mi><mml:mn>3<\/mml:mn><\/mml:msup><mml:mo>)<\/mml:mo><\/mml:mrow><\/mml:math><\/jats:alternatives><\/jats:inline-formula>).<\/jats:p>","DOI":"10.1007\/s42979-021-00642-4","type":"journal-article","created":{"date-parts":[[2021,4,20]],"date-time":"2021-04-20T12:49:31Z","timestamp":1618922971000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Gaussian Based Non-linear Function Approximation for Reinforcement Learning"],"prefix":"10.1007","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1802-3086","authenticated-orcid":false,"given":"Abbas","family":"Haider","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Glenn","family":"Hawe","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hui","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bryan","family":"Scotney","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,4,20]]},"reference":[{"key":"642_CR1","unstructured":"Anschel O, Baram N, Shimkin N. Averaged-dqn: variance reduction and stabilization for deep reinforcement learning. In: Proceedings of the 34th international conference on machine learning, PMLR, vol.\u00a070. 2017. p. 176\u201385."},{"issue":"3","key":"642_CR2","doi-asserted-by":"publisher","first-page":"217","DOI":"10.1080\/14697680701381228","volume":"8","author":"M Avellaneda","year":"2008","unstructured":"Avellaneda M, Stoikov S. High-frequency trading in a limit order book. Quant Finance. 2008;8(3):217\u201324.","journal-title":"Quant Finance"},{"key":"642_CR3","volume-title":"Neuro-dynamic programming","author":"DP Bertsekas","year":"1996","unstructured":"Bertsekas DP, Tsitsiklis JN. Neuro-dynamic programming. Nashua: Athena Scientific; 1996."},{"key":"642_CR4","unstructured":"Davies S. Multidimensional triangulation and interpolation for reinforcement learning. https:\/\/scottdavies.net\/nips96.pdf. 1997."},{"key":"642_CR5","doi-asserted-by":"crossref","unstructured":"Geist M, Pietquin O, Fricout G. Kalman temporal differences: the deterministic case. In: 2009 IEEE symposium on adaptive dynamic programming and reinforcement learning. 2009.","DOI":"10.1109\/ADPRL.2009.4927543"},{"key":"642_CR6","unstructured":"Ghiassian S, Yu H, Rafiee B, Sutton RS. Two geometric input transformation methods for fast online reinforcement learning with neural nets. 2018."},{"key":"642_CR7","doi-asserted-by":"crossref","unstructured":"Haider A, Wang H, Scotney B, Hawe G. Effect of market spread over reinforcement learning based market maker. In: Machine learning, optimization, and data science, vol. 11943. Cham: Springer; 2019. p. 143\u201353.","DOI":"10.1007\/978-3-030-37599-7_13"},{"issue":"6","key":"642_CR8","doi-asserted-by":"publisher","first-page":"3723","DOI":"10.1109\/TII.2019.2890849","volume":"15","author":"X Han","year":"2019","unstructured":"Han X, Liu H, Sun F XZ. Active object detection with multistep action prediction using deep q-network. IEEE Trans Ind Inform. 2019;15(6:3723\u201331.","journal-title":"IEEE Trans Ind Inform"},{"key":"642_CR9","doi-asserted-by":"crossref","unstructured":"van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double q-learning. In: Thirtieth AAAI conference on artificial intelligence. 2016.","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"642_CR10","first-page":"1107","volume":"4","author":"MG Lagoudakis","year":"2003","unstructured":"Lagoudakis MG, Parr R. Least-squares policy iteration. J Mach Learn Res. 2003;4:1107\u201349.","journal-title":"J Mach Learn Res"},{"issue":"11","key":"642_CR11","doi-asserted-by":"publisher","first-page":"2621","DOI":"10.1109\/JSAC.2019.2933761","volume":"37","author":"W Li","year":"2019","unstructured":"Li W, Zhang H, Gao S, Xue C, Wang X, Lu S. Smartcc: a reinforcement learning approach for multipath tcp congestion control in heterogeneous networks. IEEE J Select Areas Commun. 2019;37(11):2621\u201333.","journal-title":"IEEE J Select Areas Commun"},{"key":"642_CR12","unstructured":"Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M. Playing atari with deep reinforcement learning. https:\/\/arxiv.org\/abs\/1312.5602 2013."},{"key":"642_CR13","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1023\/A:1017992615625","volume":"49","author":"R Munos","year":"2002","unstructured":"Munos R, Moore A. Variable resolution discretization in optimal control. Springer Mach Learn. 2002;49:291\u2013323.","journal-title":"Springer Mach Learn"},{"key":"642_CR14","unstructured":"Narvekar S, Stone P. Learning curriculum policies for reinforcement learning. In: Proc. of the 18th international conference on autonomous agents and multiagent systems (AAMAS 2019). 2019."},{"key":"642_CR15","unstructured":"Oroojlooyjadid A, Nazari MR, Snyder LV, Takac M. A deep q-network for the beer game: using machine learning to solve inventory optimization problems. 2017."},{"key":"642_CR16","doi-asserted-by":"crossref","unstructured":"Otterlo M, Wiering M. Adaptation, learning, and optimization, vol.\u00a012, chap Reinforcement Learning and Markov Decision Processes. Berlin: Springer; 2012.","DOI":"10.1007\/978-3-642-27645-3"},{"key":"642_CR17","doi-asserted-by":"crossref","unstructured":"Riedmiller M. Neural fitted q iteration\u2014first experiences with a data efficient neural reinforcement learning method. In: ECML:Lecture notes in Computer Science, vol. 3720. Berlin: Springer; 2005. p. 317\u201328.","DOI":"10.1007\/11564096_32"},{"key":"642_CR18","unstructured":"Riedmiller M, Braun H. A direct adaptive method for faster backpropagation learning: the rprop algorithm. In: IEEE international conference on neural networks, IEEE. 1993."},{"key":"642_CR19","unstructured":"Rummery GA, Niranjan M. Online q learning using connectionist systems. http:\/\/mi.eng.cam.ac.uk\/reports\/svr-ftp\/auto-pdf\/rummery_tr166.pdf. 1994."},{"key":"642_CR20","doi-asserted-by":"crossref","unstructured":"Russo G, Cardellini V, Presti FL. Reinforcement learning based policies for elastic stream processing on heterogeneous resources. In: DEBS \u201919: proceedings of the 13th ACM international conference on distributed and event-based systems, 2019. p. 31\u201342.","DOI":"10.1145\/3328905.3329506"},{"key":"642_CR21","unstructured":"Schaul T, Quan J, Antonoglou I, Silver D. Prioritized experience replay. https:\/\/arxiv.org\/abs\/1511.05952. 2016."},{"key":"642_CR22","unstructured":"Spooner T, Fearnley J, Savani R, Koukorinis A. Market making via reinforcement learning. In: AAMAS2018 conference proceedings, 2018. p 434\u201342."},{"key":"642_CR23","first-page":"9","volume":"3","author":"RS Sutton","year":"1988","unstructured":"Sutton RS. Learning to predict by the methods of temporal differences. Springer Mach Learn. 1988;3:9\u201344.","journal-title":"Springer Mach Learn"},{"key":"642_CR24","volume-title":"Reinforcement learning","author":"RS Sutton","year":"2018","unstructured":"Sutton RS, Barto AG. Reinforcement learning. 2nd ed. Cambridge: The MIT Press; 2018.","edition":"2"},{"key":"642_CR25","doi-asserted-by":"crossref","unstructured":"Tesauro G. Applications of neural networks, Springer, Boston, MA, chap TD-Gammon: a self-teaching backgammon program. 1995.","DOI":"10.1007\/978-1-4757-2379-3_11"},{"key":"642_CR26","doi-asserted-by":"publisher","first-page":"39974","DOI":"10.1109\/ACCESS.2019.2902846","volume":"7","author":"Y Wang","year":"2019","unstructured":"Wang Y, Liu H, Zheng W, Xia Y, Li Y, Chen P, Guo K, Xie H. Multi-objective workflow scheduling with deep-q-network-based multi-agent reinforcement learning. IEEE Access. 2019;7:39974\u201382.","journal-title":"IEEE Access"},{"key":"642_CR27","unstructured":"Wang Z, Schaul T, Hessel M, Hasselt H, Lanctot M, Freitas N. Dueling network architectures for deep reinforcement learning. In: Proceedings of the 33rd international conference on machine learning, PMLR. 2016."},{"key":"642_CR28","unstructured":"Watkins C. Learning from delayed rewards. PhD thesis, Computer Science. 1989."},{"key":"642_CR29","first-page":"279","volume":"8","author":"CJ Watkins","year":"1992","unstructured":"Watkins CJ, Dayan P. Technical note: Q-learning. Springer Mach Learn. 1992;8:279\u201392.","journal-title":"Springer Mach Learn"}],"container-title":["SN Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-021-00642-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42979-021-00642-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-021-00642-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,24]],"date-time":"2022-12-24T20:49:46Z","timestamp":1671914986000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42979-021-00642-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,20]]},"references-count":29,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,5]]}},"alternative-id":["642"],"URL":"https:\/\/doi.org\/10.1007\/s42979-021-00642-4","relation":{},"ISSN":["2662-995X","2661-8907"],"issn-type":[{"type":"print","value":"2662-995X"},{"type":"electronic","value":"2661-8907"}],"subject":[],"published":{"date-parts":[[2021,4,20]]},"assertion":[{"value":"11 November 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 April 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 April 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"On behalf of all authors, the corresponding author states that there is no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}],"article-number":"223"}}