{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T05:08:23Z","timestamp":1775711303650,"version":"3.50.1"},"reference-count":14,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2022,11,24]],"date-time":"2022-11-24T00:00:00Z","timestamp":1669248000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Conselho Nacional de Desenvolvimento Cient\u00edfico e Tecnol\u00f3gico (CNPq)","award":["403457\/2020-3"],"award-info":[{"award-number":["403457\/2020-3"]}]},{"name":"Brazil and the National Institute for Research in Digital Science and Technology (INRIA, France)","award":["403457\/2020-3"],"award-info":[{"award-number":["403457\/2020-3"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This study introduces a novel controller based on a Reinforcement Learning (RL) algorithm for real-time adaptation of the stimulation pattern during FES-cycling. Core to our approach is the introduction of an RL agent that interacts with the cycling environment and learns through trial and error how to modulate the electrical charge applied to the stimulated muscle groups according to a predefined policy and while tracking a reference cadence. Instead of a static stimulation pattern to be modified by a control law, we hypothesized that a non-stationary baseline set of parameters would better adjust the amount of injected electrical charge to the time-varying characteristics of the musculature. Overground FES-assisted cycling sessions were performed by a subject with spinal cord injury (SCI AIS-A, T8). For tracking a predefined pedaling cadence, two closed-loop control laws were simultaneously used to modulate the pulse intensity of the stimulation channels responsible for evoking the muscle contractions. First, a Proportional-Integral (PI) controller was used to control the current amplitude of the stimulation channels over an initial parameter setting with predefined pulse amplitude, width and fixed frequency parameters. In parallel, an RL algorithm with a decayed-epsilon-greedy strategy was implemented to randomly explore nine different variations of pulse amplitude and width parameters over the same stimulation setting, aiming to adjust the injected electrical charge according to a predefined policy. The performance of this global control strategy was evaluated in two different RL settings and explored in two different cycling scenarios. The participant was able to pedal overground for distances over 3.5 km, and the results evidenced the RL agent learned to modify the stimulation pattern according to the predefined policy and was simultaneously able to track a predefined pedaling cadence. Despite the simplicity of our approach and the existence of more sophisticated RL algorithms, our method can be used to reduce the time needed to define stimulation patterns. Our results suggest interesting research possibilities to be explored in the future to improve cycling performance since more efficient stimulation cost dynamics can be explored and implemented for the agent to learn.<\/jats:p>","DOI":"10.3390\/s22239126","type":"journal-article","created":{"date-parts":[[2022,11,25]],"date-time":"2022-11-25T03:34:24Z","timestamp":1669347264000},"page":"9126","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["A Novel Functional Electrical Stimulation-Induced Cycling Controller Using Reinforcement Learning to Optimize Online Muscle Activation Pattern"],"prefix":"10.3390","volume":"22","author":[{"given":"Tiago","family":"Coelho-Magalh\u00e3es","sequence":"first","affiliation":[{"name":"Graduate Program in Electrical Engineering, Universidade Federal de Minas Gerais, Av, Ant\u00f4nio Carlos 6627, Belo Horizonte 31270-901, MG, Brazil"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7379-8004","authenticated-orcid":false,"given":"Christine","family":"Azevedo Coste","sequence":"additional","affiliation":[{"name":"National Institute for Research in Computer Science and Automation (Inria), Camin Team, 34090 Montpellier, France"}]},{"given":"Henrique","family":"Resende-Martins","sequence":"additional","affiliation":[{"name":"Graduate Program in Electrical Engineering, Universidade Federal de Minas Gerais, Av, Ant\u00f4nio Carlos 6627, Belo Horizonte 31270-901, MG, Brazil"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1016\/j.arcontrol.2017.09.014","article-title":"Sensing motion and muscle activity for feedback control of functional electrical stimulation: Ten years of experience in Berlin","volume":"44","author":"Schauer","year":"2017","journal-title":"Annu. Rev. Control."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1186\/s12984-021-00882-8","article-title":"Functional electrical stimulation cycling exercise after spinal cord injury: A systematic review of health and fitness-related outcomes","volume":"18","author":"Valentino","year":"2021","journal-title":"J. NeuroEngineering Rehabil."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1179\/2045772314Y.0000000244","article-title":"The effects of electrical stimulation on body composition and metabolic profile after spinal cord injury\u2014Part II","volume":"38","author":"Gorgey","year":"2015","journal-title":"J. Spinal Cord Med."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1093\/ptj\/85.4.358","article-title":"Recruitment Patterns in Human Skeletal Muscle During Electrical Stimulation","volume":"85","author":"Gregory","year":"2005","journal-title":"Phys. Ther."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1146\/annurev.bioeng.6.040803.140103","article-title":"Functional electrical stimulation for neuromuscular applications","volume":"7","author":"Peckham","year":"2005","journal-title":"Annu. Rev. Biomed. Eng."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1016\/j.clinbiomech.2014.11.007","article-title":"Technical developments of functional electrical stimulation to correct drop foot: Sensing, actuation and control strategies","volume":"30","author":"Melo","year":"2015","journal-title":"Clin. Biomech. (Bristol Avon)"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Cousin, C., Duenas, V., and Dixon, W. (2021). FES Cycling and Closed-Loop Feedback Control for Rehabilitative Human\u2014Robot Interaction. Robotics, 10.","DOI":"10.3390\/robotics10020061"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Sijobert, B., Le Guillou, R., Fattal, C., and Azevedo-Coste, C. (2019). FES-Induced Cycling in Complete SCI: A Simpler Control Method Based on Inertial Sensors. Sensors, 19.","DOI":"10.3390\/s19194268"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Coelho-Magalh\u00e3es, T., Fachin-Martins, E., Silva, A., Azevedo Coste, C., and Resende-Martins, H. (2022). Development of a High-Power Capacity Open Source Electrical Stimulation System to Enhance Research into FES-Assisted Devices: Validation of FES Cycling. Sensors, 22.","DOI":"10.3390\/s22020531"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1084","DOI":"10.1109\/TCYB.2018.2882755","article-title":"Distributed Repetitive Learning Control for Cooperative Cadence Tracking in Functional Electrical Stimulation Cycling","volume":"50","author":"Duenas","year":"2020","journal-title":"IEEE Trans. Cybern."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"108852","DOI":"10.1016\/j.automatica.2020.108852","article-title":"Torque and cadence tracking in functional electrical stimulation induced cycling using passivity-based spatial repetitive learning control","volume":"115","author":"Duenas","year":"2020","journal-title":"Automatica"},{"key":"ref_12","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, The MIT Press. [2nd ed.]."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Wannawas, N., Subramanian, M., and Faisal, A.A. (2021, January 4\u20136). Neuromechanics-based Deep Reinforcement Learning of Neurostimulation Control in FES cycling. Proceedings of the 2021 10th International IEEE\/EMBS Conference on Neural Engineering (NER), Virtual Event.","DOI":"10.1109\/NER49283.2021.9441354"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1186\/s12984-022-01018-2","article-title":"OIDA: An optimal interval detection algorithm for automatized determination of stimulation patterns for FES-Cycling in individuals with SCI","volume":"19","author":"Schmoll","year":"2022","journal-title":"J. NeuroEng. Rehabil."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/23\/9126\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:26:00Z","timestamp":1760145960000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/23\/9126"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,24]]},"references-count":14,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["s22239126"],"URL":"https:\/\/doi.org\/10.3390\/s22239126","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,24]]}}}