{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T06:17:03Z","timestamp":1768457823503,"version":"3.49.0"},"reference-count":48,"publisher":"MIT Press","issue":"12","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,11,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>The current reinforcement learning framework focuses exclusively on performance, often at the expense of efficiency. In contrast, biological control achieves remarkable performance while also optimizing computational energy expenditure and decision frequency. We propose a decision-bounded Markov decision process (DB-MDP) that constrains the number of decisions and computational energy available to agents in reinforcement learning environments. Our experiments demonstrate that existing reinforcement learning algorithms struggle within this framework, leading to either failure or suboptimal performance. To address this, we introduce a biologically inspired, temporally layered architecture (TLA), enabling agents to manage computational costs through two layers with distinct timescales and energy requirements. TLA achieves optimal performance in decision-bounded environments and in continuous control environments, matching state-of-the-art performance while using a fraction of the computing cost. Compared to current reinforcement learning algorithms that solely prioritize performance, our approach significantly lowers computational energy expenditure while maintaining performance. These findings establish a benchmark and pave the way for future research on energy and time-aware control.<\/jats:p>","DOI":"10.1162\/neco_a_01718","type":"journal-article","created":{"date-parts":[[2024,10,9]],"date-time":"2024-10-09T16:23:06Z","timestamp":1728490986000},"page":"2734-2763","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":1,"title":["Optimizing Attention and Cognitive Control Costs Using Temporally Layered Architectures"],"prefix":"10.1162","volume":"36","author":[{"given":"Devdhar","family":"Patel","sequence":"first","affiliation":[{"name":"Manning College of Information and Computer Science, University of Massachusetts, Amherst, MA 01003, U.S.A. devdharpatel@gmail.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Terrence","family":"Sejnowski","sequence":"additional","affiliation":[{"name":"Computational Neurobiology Laboratory, Salk Institute for Biological Studies, La\u00a0Jolla, CA 92037, U.S.A."},{"name":"Institute for Neural Computation and Department of Neurobiology, University of California, San Diego, La Jolla, CA 92093, U.S.A. terry@snl.salk.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hava","family":"Siegelmann","sequence":"additional","affiliation":[{"name":"Manning College of Information and Computer Science, University of Massachusetts, Amherst, MA 01003, U.S.A. hava@umass.edu"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2024,11,19]]},"reference":[{"key":"2024111519195558300_bib1","first-page":"914","article-title":"TempoRL: Learning when to act","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Biedenkapp","year":"2021"},{"key":"2024111519195558300_bib2","article-title":"Frame skip is a powerful parameter for learning to play Atari","volume-title":"Proceedings of the AAAI Workshop: Learning for General Competency in Video Games","author":"Braylan","year":"2015"},{"key":"2024111519195558300_bib3","author":"Brockman","year":"2016","journal-title":"OpenAI Gym"},{"key":"2024111519195558300_bib4","article-title":"Transition point dynamic programming","volume-title":"Advances in neural information processing systems","author":"Buckland","year":"1993"},{"key":"2024111519195558300_bib5","article-title":"Learning in a small world","volume-title":"Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems","author":"Chaganty","year":"2012"},{"key":"2024111519195558300_bib6","author":"Dabney","year":"2021","journal-title":"Temporally-extended \u03f5-greedy exploration"},{"key":"2024111519195558300_bib7","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1016\/j.brainres.2014.09.074","article-title":"Oscillatory brain responses to own names uttered by unfamiliar and familiar voices","volume":"1591","author":"Del Giudice","year":"2014","journal-title":"Brain Research"},{"key":"2024111519195558300_bib8","first-page":"1587","article-title":"Addressing function approximation error in actor-critic methods","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Fujimoto","year":"2018"},{"key":"2024111519195558300_bib9","first-page":"1861","article-title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Haarnoja","year":"2018"},{"key":"2024111519195558300_bib10","article-title":"Reinforcement learning for mixed open-loop and closed-loop control","volume-title":"Advances in neural information processing systems","author":"Hansen","year":"1996"},{"issue":"3","key":"2024111519195558300_bib11","doi-asserted-by":"publisher","first-page":"379","DOI":"10.1002\/cne.903240308","article-title":"Corticotectal projections in the cat: Anterograde transport studies of twenty-five cortical areas","volume":"324","author":"Harting","year":"1992","journal-title":"Journal of Comparative Neurology"},{"key":"2024111519195558300_bib12","doi-asserted-by":"publisher","DOI":"10.3389\/fnins.2014.00150","article-title":"The speed-accuracy tradeoff: History, physiology, methodology, and behavior","volume":"8","author":"Heitz","year":"2014","journal-title":"Frontiers in Neuroscience"},{"key":"2024111519195558300_bib13","author":"Jacq","year":"2022","journal-title":"Lazy-MDPs: Towards interpretable reinforcement learning by learning when to act"},{"issue":"2","key":"2024111519195558300_bib14","doi-asserted-by":"publisher","first-page":"124","DOI":"10.4103\/2229-516X.157168","article-title":"A comparative study of visual and auditory reaction times on the basis of gender and physical activity levels of medical first year students","volume":"5","author":"Jain","year":"2015","journal-title":"International Journal of Applied and Basic Medical Research"},{"key":"2024111519195558300_bib15","first-page":"6023","article-title":"Residual reinforcement learning for robot control","volume-title":"Proceedings of the 2019 International Conference on Robotics and Automation","author":"Johannink","year":"2019"},{"key":"2024111519195558300_bib16","author":"Kalyanakrishnan","year":"2021","journal-title":"An analysis of frame-skipping in reinforcement learning"},{"issue":"39","key":"2024111519195558300_bib17","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2300445120","article-title":"Internal feedback in the cortical perception\u2013action loop enables fast and accurate behavior","volume":"120","author":"Li","year":"2023","journal-title":"Proceedings of the National Academy of Sciences, USA"},{"key":"2024111519195558300_bib18","author":"Machado","year":"2021","journal-title":"Temporal abstraction in reinforcement learning with the successor representation"},{"key":"2024111519195558300_bib19","author":"Masset","year":"2023","journal-title":"Multi-timescale reinforcement learning in the brain"},{"key":"2024111519195558300_bib20","author":"McCallum","year":"1996","journal-title":"Reinforcement learning with selective perception and hidden state"},{"key":"2024111519195558300_bib21","article-title":"Roles of macro-actions in accelerating reinforcement learning","author":"McGovern","year":"1997","journal-title":"Proceedings of the 1997 Grace Hopper Celebration of Women in Computing"},{"key":"2024111519195558300_bib22","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"2024111519195558300_bib23","author":"Moore","year":"1990","journal-title":"Efficient memory-based learning for robot control"},{"key":"2024111519195558300_bib24","doi-asserted-by":"publisher","DOI":"10.1098\/rspb.2018.0613","article-title":"Scaling of sensorimotor delays in terrestrial mammals","volume":"285","author":"More","year":"2018","journal-title":"Proceedings of the Royal Society B: Biological Sciences"},{"key":"2024111519195558300_bib25","doi-asserted-by":"publisher","first-page":"2342","DOI":"10.1523\/JNEUROSCI.0836-15.2016","article-title":"Temporal prediction in lieu of periodic stimulation","volume":"36","author":"Morillon","year":"2016","journal-title":"Journal of Neuroscience"},{"key":"2024111519195558300_bib26","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1916367118","article-title":"Diversity-enabled sweet spots in layered architectures and speed\u2013accuracy trade-offs in sensorimotor control","volume":"118","author":"Nakahira","year":"2021","journal-title":"Proceedings of the National Academy of Sciences USA"},{"key":"2024111519195558300_bib27","author":"Oroojlooyjadid","year":"2019","journal-title":"A review of cooperative multi-agent deep reinforcement learning"},{"key":"2024111519195558300_bib28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3459991","article-title":"A survey of reinforcement learning algorithms for dynamically varying environments","volume":"54","author":"Padakandla","year":"2020","journal-title":"ACM Computing Surveys"},{"key":"2024111519195558300_bib29","first-page":"4045","article-title":"Time limits in reinforcement learning","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Pardo","year":"2018"},{"key":"2024111519195558300_bib30","author":"Patel","year":"2022","journal-title":"Quicknets: Saving training and preventing over-confidence in early-exit neural architectures"},{"key":"2024111519195558300_bib31","article-title":"Temporal abstraction in reinforcement learning","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Precup","year":"2000"},{"key":"2024111519195558300_bib32","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1016\/S0927-0507(05)80172-0","article-title":"Markov decision processes","volume-title":"Handbook in operations research and management science","author":"Puterman","year":"1990"},{"key":"2024111519195558300_bib33","article-title":"Learning macro-actions in reinforcement learning","volume-title":"Advances in neural information processing systems","author":"Randl\u00f8v","year":"1998"},{"issue":"5","key":"2024111519195558300_bib34","doi-asserted-by":"publisher","first-page":"954","DOI":"10.1007\/s12559-020-09734-4","article-title":"Why should we add early exits to neural networks?","volume":"12","author":"Scardapane","year":"2020","journal-title":"Cognitive Computation"},{"key":"2024111519195558300_bib35","author":"Schulman","year":"2017","journal-title":"Proximal policy optimization algorithms"},{"key":"2024111519195558300_bib36","author":"Sharma","year":"2017","journal-title":"Learning to repeat: Fine grained action repetition for deep reinforcement learning"},{"key":"2024111519195558300_bib37","author":"Silver","year":"2018","journal-title":"Residual policy learning"},{"key":"2024111519195558300_bib38","article-title":"Dynamic action repetition for deep reinforcement learning","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Srinivas","year":"2017"},{"key":"2024111519195558300_bib39","volume-title":"Reinforcement learning: An introduction","author":"Sutton","year":"2018"},{"key":"2024111519195558300_bib40","doi-asserted-by":"crossref","first-page":"2917","DOI":"10.1007\/978-3-540-36841-0_737","article-title":"Relationship between jerk cost function and energy consumption during walking","volume-title":"Proceedings of the World Congress on Medical Physics and Biomedical Engineering 2006","author":"Tack","year":"2007"},{"key":"2024111519195558300_bib41","article-title":"Cost-sensitive reinforcement learning for adaptive classification and control","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Tan","year":"1991"},{"key":"2024111519195558300_bib42","first-page":"5026","article-title":"Mujoco: A physics engine for model-based control","volume-title":"Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems","author":"Todorov","year":"2012"},{"key":"2024111519195558300_bib43","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1016\/j.neuroscience.2022.11.025","article-title":"Effects of natural scene inversion on visual-evoked brain potentials and pupillary responses: A matter of effortful processing of unfamiliar configurations","volume":"509","author":"van Helden","year":"2023","journal-title":"Neuroscience"},{"key":"2024111519195558300_bib44","doi-asserted-by":"crossref","DOI":"10.1109\/IEMBS.1999.802610","article-title":"Minimum jerk theory revisited","volume-title":"Proceedings of the First Joint BMES\/EMBS Conference","author":"Voros","year":"1999"},{"issue":"4","key":"2024111519195558300_bib45","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/s41593-019-0336-0","article-title":"Spinal stretch reflexes support efficient hand control","volume":"22","author":"Weiler","year":"2019","journal-title":"Nature Neuroscience"},{"key":"2024111519195558300_bib46","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.00801","article-title":"Skill learning strengthens cortical representations of motor sequences","volume":"2","author":"Wiestler","year":"2013","journal-title":"eLife"},{"key":"2024111519195558300_bib47","article-title":"TAAC: Temporally abstract actor-critic for continuous control","volume-title":"Advances in neural information processing systems","author":"Yu","year":"2021"},{"key":"2024111519195558300_bib48","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1007\/978-3-030-60990-0_12","article-title":"Multi-agent reinforcement learning: A selective overview of the ories and algorithms","author":"Zhang","year":"2021","journal-title":"Handbook of reinforcement learning and control"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/36\/12\/2734\/2479577\/neco_a_01718.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/36\/12\/2734\/2479577\/neco_a_01718.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,15]],"date-time":"2024-11-15T19:20:13Z","timestamp":1731698413000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/36\/12\/2734\/124826\/Optimizing-Attention-and-Cognitive-Control-Costs"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,19]]},"references-count":48,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2024,11,19]]},"published-print":{"date-parts":[[2024,11,19]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01718","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,12]]},"published":{"date-parts":[[2024,11,19]]}}}