{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T07:21:16Z","timestamp":1778656876740,"version":"3.51.4"},"reference-count":26,"publisher":"SAGE Publications","issue":"2","license":[{"start":{"date-parts":[[1997,9,1]],"date-time":"1997-09-01T00:00:00Z","timestamp":873072000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Adaptive Behavior"],"published-print":{"date-parts":[[1997,9]]},"abstract":"<jats:p>A key element in the solution of reinforcement learning problems is the value function. The purpose of this function is to measure the long-term utility or value of any given state. The function is important because an agent can use this measure to decide what to do next. A common problem in reinforcement learning when applied to systems having continuous states and action spaces is that the value function must operate with a domain consisting of real-valued variables, which means that it should be able to represent the value of infinitely many state and action pairs. For this reason, function approximators are used to represent the value function when a close-form solution of the optimal policy is not available. In this article, we extend a previously proposed reinforcement learning algorithm so that it can be used with function approximators that generalize the value of individual experiences across both state and action spaces. In particular, we discuss the benefits of using sparse coarse-coded function approximators to represent value functions and describe in detail three implementations: cerebellar model articulation controllers, instance-based, and case-based. Additionally, we discuss how function approximators having different degrees of resolution in different regions of the state and action spaces may influence the performance and learning efficiency of the agent. We propose a simple and modular technique that can be used to implement function approximators with nonuniform degrees of resolution so that the value function can be represented with higher accuracy in important regions of the state and action spaces. We performed extensive experiments in the double-integrator and pendulum swing-up systems to demonstrate the proposed ideas. '<\/jats:p>","DOI":"10.1177\/105971239700600201","type":"journal-article","created":{"date-parts":[[2007,3,11]],"date-time":"2007-03-11T01:45:25Z","timestamp":1173577525000},"page":"163-217","source":"Crossref","is-referenced-by-count":107,"title":["Experiments with Reinforcement Learning in Problems with Continuous State and Action Spaces"],"prefix":"10.1177","volume":"6","author":[{"given":"Juan C.","family":"Santamaria","sequence":"first","affiliation":[{"name":"Georgia Institute of Technology"}]},{"given":"Richard S.","family":"Sutton","sequence":"additional","affiliation":[{"name":"University of Massachusetts"}]},{"given":"Ashwin","family":"Ram","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology"}]}],"member":"179","published-online":{"date-parts":[[1997,9,1]]},"reference":[{"key":"atypb1","doi-asserted-by":"publisher","DOI":"10.1115\/1.3426922"},{"key":"atypb2","volume-title":"Proceedings of the 1991 American Control Conference","author":"Atkeson, C.G."},{"key":"atypb3","first-page":"835","volume":"13","author":"Barto, A.G.","year":"1983","journal-title":"IEEE Transactions on Systems, Man, and Cybernetics"},{"key":"atypb4","volume-title":"Dynamic programming","author":"Bellman, R.","year":"1957"},{"key":"atypb5","volume-title":"Dynamic programming and optimal control","author":"Bertsekas, D.P.","year":"1995"},{"key":"atypb6","first-page":"287","author":"Cishosz, P.","year":"1996","journal-title":"Journal of Artificial Intelligence Research"},{"key":"atypb7","volume-title":"Sparse distributed memory and related models","author":"Kanerva, P.","year":"1993"},{"key":"atypb8","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-8640.1989.tb00315.x"},{"key":"atypb9","first-page":"3","volume":"8","author":"Lin, L.J.","year":"1992","journal-title":"Machine Learning"},{"key":"atypb10","volume-title":"Proceedings of the Eighth International Workshop on Machine Learning","author":"Mahadevan, S."},{"key":"atypb11","first-page":"377","volume":"7","author":"McCallum, R.A.","year":"1995","journal-title":"Advances in Neural Information Processing Systems"},{"key":"atypb12","doi-asserted-by":"publisher","DOI":"10.1007\/BF00993591"},{"key":"atypb13","volume-title":"Stable adaptive systems","author":"Narendra, K.S.","year":"1989"},{"key":"atypb14","volume-title":"Efficient dynamic programming-based learning for control","author":"Peng, J.","year":"1993"},{"key":"atypb15","volume-title":"Machine Learning: Proceedings of the Eleventh International Conference","author":"Peng, J."},{"key":"atypb16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/S0004-3702(96)00018-5","volume":"90","author":"Ram, A.","year":"1997","journal-title":"Artificial Intelligence"},{"key":"atypb17","volume-title":"An introduction to dynamics and control","author":"Richards, R.J.","year":"1979"},{"key":"atypb18","volume-title":"On-line q-learning using connectionist systems (Tech. Rep. No. CUED\/F-INFEG\/TR66)","author":"Rummery, G.A.","year":"1994"},{"key":"atypb19","volume-title":"Digital and analog communication systems","author":"Shanmugam, K.S.","year":"1979"},{"key":"atypb20","doi-asserted-by":"publisher","DOI":"10.1007\/BF00114726"},{"key":"atypb21","volume-title":"Optimal control and estimation. Mineola","author":"Stengel, R.E.","year":"1994"},{"key":"atypb22","doi-asserted-by":"publisher","DOI":"10.1007\/BF00115009"},{"key":"atypb23","first-page":"1038","volume":"8","author":"Sutton, R.S.","year":"1996","journal-title":"Advances in Neural Information Processing Systems"},{"key":"atypb24","doi-asserted-by":"publisher","DOI":"10.1016\/0921-8890(95)00005-Z"},{"key":"atypb25","doi-asserted-by":"publisher","DOI":"10.1109\/9.580874"},{"key":"atypb26","volume-title":"Learning from delayed rewards","author":"Watkins, C.J.C.H.","year":"1989"}],"container-title":["Adaptive Behavior"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/105971239700600201","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/105971239700600201","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T16:18:23Z","timestamp":1777393103000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/105971239700600201"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[1997,9]]},"references-count":26,"journal-issue":{"issue":"2","published-print":{"date-parts":[[1997,9]]}},"alternative-id":["10.1177\/105971239700600201"],"URL":"https:\/\/doi.org\/10.1177\/105971239700600201","relation":{},"ISSN":["1059-7123","1741-2633"],"issn-type":[{"value":"1059-7123","type":"print"},{"value":"1741-2633","type":"electronic"}],"subject":[],"published":{"date-parts":[[1997,9]]}}}