{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,23]],"date-time":"2026-01-23T23:17:03Z","timestamp":1769210223116,"version":"3.49.0"},"reference-count":54,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2020,9,27]],"date-time":"2020-09-27T00:00:00Z","timestamp":1601164800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["U1913207, 61233015, 61473131"],"award-info":[{"award-number":["U1913207, 61233015, 61473131"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Program on Key Basic Research Project of China","award":["2013CB329506"],"award-info":[{"award-number":["2013CB329506"]}]},{"name":"Postdoctoral Science Foundation of China","award":["2019M652652"],"award-info":[{"award-number":["2019M652652"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Background: For the nonstationarity of neural recordings in intracortical brain\u2013machine interfaces, daily retraining in a supervised manner is always required to maintain the performance of the decoder. This problem can be improved by using a reinforcement learning (RL) based self-recalibrating decoder. However, quickly exploring new knowledge while maintaining a good performance remains a challenge in RL-based decoders. Methods: To solve this problem, we proposed an attention-gated RL-based algorithm combining transfer learning, mini-batch, and weight updating schemes to accelerate the weight updating and avoid over-fitting. The proposed algorithm was tested on intracortical neural data recorded from two monkeys to decode their reaching positions and grasping gestures. Results: The decoding results showed that our proposed algorithm achieved an approximate 20% increase in classification accuracy compared to that obtained by the non-retrained classifier and even achieved better classification accuracy than the daily retraining classifier. Moreover, compared with a conventional RL method, our algorithm improved the accuracy by approximately 10% and the online weight updating speed by approximately 70 times. Conclusions: This paper proposed a self-recalibrating decoder which achieved a good and robust decoding performance with fast weight updating and might facilitate its application in wearable device and clinical practice.<\/jats:p>","DOI":"10.3390\/s20195528","type":"journal-article","created":{"date-parts":[[2020,9,28]],"date-time":"2020-09-28T08:02:58Z","timestamp":1601280178000},"page":"5528","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Reinforcement Learning Based Fast Self-Recalibrating Decoder for Intracortical Brain\u2013Machine Interface"],"prefix":"10.3390","volume":"20","author":[{"given":"Peng","family":"Zhang","sequence":"first","affiliation":[{"name":"Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan 430074, China"}]},{"given":"Lianying","family":"Chao","sequence":"additional","affiliation":[{"name":"Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan 430074, China"}]},{"given":"Yuting","family":"Chen","sequence":"additional","affiliation":[{"name":"Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan 430074, China"}]},{"given":"Xuan","family":"Ma","sequence":"additional","affiliation":[{"name":"Department of physiology, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA"}]},{"given":"Weihua","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China"}]},{"given":"Jiping","family":"He","sequence":"additional","affiliation":[{"name":"Advanced Innovation Center for Intelligent Robots and Systems, Beijing Institute of Technology, Beijing 100081, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6267-8824","authenticated-orcid":false,"given":"Jian","family":"Huang","sequence":"additional","affiliation":[{"name":"School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China"}]},{"given":"Qiang","family":"Li","sequence":"additional","affiliation":[{"name":"Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan 430074, China"}]}],"member":"1968","published-online":{"date-parts":[[2020,9,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1016\/j.eij.2015.06.002","article-title":"Brain computer interfacing: Applications and challenges","volume":"16","author":"Abdulkader","year":"2015","journal-title":"Egypt. Inform. J."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1038\/nrn3724","article-title":"Restoring sensorimotor function through intracortical interfaces: Progress and looming challenges","volume":"15","author":"Bensmaia","year":"2014","journal-title":"Nat. Rev. Neurosci."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1038\/nature04970","article-title":"Neuronal ensemble control of prosthetic devices by a human with tetraplegia","volume":"442","author":"Hochberg","year":"2006","journal-title":"Nature"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1038\/nature04968","article-title":"A high-performance braincomputer interface","volume":"442","author":"Santhanam","year":"2006","journal-title":"Nature"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1098","DOI":"10.1038\/nature06996","article-title":"Cortical control of a prosthetic arm for self-feeding","volume":"453","author":"Meel","year":"2008","journal-title":"Nature"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"557","DOI":"10.1016\/S0140-6736(12)61816-9","article-title":"High-performance neuroprosthetic control by an individual with tetraplegia","volume":"381","author":"Collinger","year":"2013","journal-title":"Lancet"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"906","DOI":"10.1126\/science.aaa5417","article-title":"Decoding motor imagery from the posterior parietal cortex of a tetraplegic human","volume":"348","author":"Tyson","year":"2015","journal-title":"Science"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2060","DOI":"10.1056\/NEJMoa1608085","article-title":"Fully implanted braincomputer interface in a locked-in patient with als","volume":"375","author":"Vansteensel","year":"2016","journal-title":"N. Engl. J. Med."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1038\/nature11076","article-title":"Reach and grasp by people with tetraplegia using a neurally controlled robotic arm","volume":"485","author":"Hochberg","year":"2013","journal-title":"Nature"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"e18554","DOI":"10.7554\/eLife.18554","article-title":"High performance communication by people with paralysis using an intracortical braincomputer interface","volume":"6","author":"Chethan","year":"2017","journal-title":"Elife"},{"key":"ref_11","first-page":"313ra179","article-title":"Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer interface","volume":"7","author":"Beata","year":"2015","journal-title":"Sci. Transl. Med."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"462","DOI":"10.1177\/1545968314554624","article-title":"Neural point-and-click communication by a person with incomplete locked-in syndrome","volume":"29","author":"Bacher","year":"2015","journal-title":"Neurorehabil. Neural Repair"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1669","DOI":"10.1038\/s41591-018-0171-y","article-title":"Meeting brain-computer interface user performance expectations using a deep neural network decoding framework","volume":"24","author":"Schwemmer","year":"2018","journal-title":"Nat. Med."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1752","DOI":"10.1038\/nn.3265","article-title":"A high-performance neural prosthesis enabled by control algorithm design","volume":"15","author":"Gilja","year":"2012","journal-title":"Nat. Neurosci."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"026019","DOI":"10.1088\/1741-2552\/aaa8a4","article-title":"Decoder calibration with ultra small current sample set for intracortical brainmachine interface","volume":"15","author":"Zhang","year":"2018","journal-title":"J. Neural Eng."},{"key":"ref_16","unstructured":"Farshchian, A., Gallego, J.A., Cohen, J.P., Bengio, Y., Miller, L.E., and Solla, S.A. (2019). Adversarial domain adaptation for stable brain\u2013machine interfaces. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1705","DOI":"10.1109\/TNSRE.2016.2612001","article-title":"Decoding local field potentials for neural interfaces","volume":"25","author":"Jackson","year":"2017","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"36004","DOI":"10.1088\/1741-2560\/10\/3\/036004","article-title":"Intra-day signal instabilities affect decoding performance in an intracortical neural interface system","volume":"10","author":"Perge","year":"2013","journal-title":"J. Neural Eng."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"13749","DOI":"10.1038\/ncomms13749","article-title":"Making brain machine interfaces robust to future neural variability","volume":"7","author":"Sussillo","year":"2016","journal-title":"Nat. Commun."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"66003","DOI":"10.1088\/1741-2560\/11\/6\/066003","article-title":"Performance sustaining intracortical neural prostheses","volume":"11","author":"Paul","year":"2014","journal-title":"J. Neural Eng."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"26001","DOI":"10.1088\/1741-2560\/11\/2\/026001","article-title":"Self-recalibrating classifiers for intracortical brain-computer interfaces","volume":"11","author":"Bishop","year":"2014","journal-title":"J. Neural Eng."},{"key":"ref_22","unstructured":"Zhang, X., Yao, L., Wang, X., Monaghan, J., Mcalpine, D., and Zhang, Y. (2019). A survey on deep learning based brain computer interface: Recent advances and new frontiers. arXiv."},{"key":"ref_23","first-page":"10868","article-title":"Adversarial Representation Learning for Robust Patient-Independent Epileptic Seizure Detection","volume":"1909","author":"Zhang","year":"2019","journal-title":"IEEE J. Biome. Health Inform."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"3162","DOI":"10.1162\/NECO_a_00207","article-title":"Adaptive decoding for brain\u2013machine interfaces through bayesian parameter updates","volume":"23","author":"Li","year":"2014","journal-title":"Neural Comput."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2456","DOI":"10.1152\/jn.01118.2006","article-title":"General purpose filter design for neural prosthetic devices","volume":"98","author":"Srinivasan","year":"2007","journal-title":"J. Neurophysiol."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1109\/TNSRE.2008.922679","article-title":"Real-time decoding of nonstationary neural activity in motor cortex","volume":"16","author":"Wu","year":"2008","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1109\/TCDS.2018.2869587","article-title":"Using high-frequency local field potentials from multicortex to decode reaching and grasping movements in monkey","volume":"11","author":"Zhang","year":"2019","journal-title":"IEEE T. Cogn. Dev. Syst."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Pohlmeyer, E.A., Mahmoudi, B., Geng, S., Prins, N.W., Sanchez, J.C., and Cymbalyuk, G. (2014). Using reinforcement learning to provide stable brainmachine interface control despite neural input reorganization. PLoS ONE, 9.","DOI":"10.1371\/journal.pone.0087253"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"873","DOI":"10.1109\/TNNLS.2015.2493079","article-title":"Quantized attention-gated kernel reinforcement learning for brainmachine interface decoding","volume":"28","author":"Wang","year":"2017","journal-title":"IEEE Trans. Neural. Netw. Learn Syst."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1684","DOI":"10.1109\/TNSRE.2019.2934176","article-title":"Clustering neural patterns in kernel reinforcement learning assists fast brain control in brain\u2013machine interfaces","volume":"27","author":"Zhang","year":"2019","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Mahmoudi, B., and Sanchez, J.C. (2011). A symbiotic brain\u2013machine interface through value-based decision making. PLoS ONE, 6.","DOI":"10.1371\/journal.pone.0014760"},{"key":"ref_32","unstructured":"Bower, G.H. (1981). Theories of Learning, Prentice-Hall. [5th ed.]."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press.","DOI":"10.1109\/TNN.1998.712192"},{"key":"ref_34","first-page":"1","article-title":"Neural control of a tracking task via attention-gated reinforcement learning for brain\u2013machine interfaces","volume":"23","author":"Wang","year":"2014","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1109\/TBME.2008.926699","article-title":"Coadaptive brainmachine interface via reinforcement learning","volume":"56","author":"Digiovanna","year":"2009","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Sanchez, J.C., Tarigoppula, A., Choi, J.S., Marsh, B.T., and Francis, J.T. (May, January 27). Control of a center-out reaching task using a reinforcement learning brain\u2013machine interface. Proceedings of the 5th International IEEE\/EMBS Conference on Neural Engineering (NER 2011), Canun, Mexico.","DOI":"10.1109\/NER.2011.5910601"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"066005","DOI":"10.1088\/1741-2560\/10\/6\/066005","article-title":"Towards autonomous neuroprosthetic control using hebbian reinforcement learning","volume":"10","author":"Mahmoudi","year":"2013","journal-title":"J. Neural Eng."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"887","DOI":"10.1038\/nn1279","article-title":"Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops","volume":"7","author":"Tanaka","year":"2004","journal-title":"Nat. Neurosci."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"410","DOI":"10.1038\/nn2077","article-title":"Modulators of decision making","volume":"11","author":"Doya","year":"2008","journal-title":"Nat. Neurosci."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"7374","DOI":"10.1523\/JNEUROSCI.1802-14.2015","article-title":"Toward an autonomous brain machine interface: Integrating sensorimotor reward modulation and reinforcement learning","volume":"35","author":"Marsh","year":"2015","journal-title":"J. Neurosci."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"2176","DOI":"10.1162\/0899766054615699","article-title":"Attention-gated reinforcement learning of internal representations for classification","volume":"17","author":"Roelfsema","year":"2005","journal-title":"Neural Comput."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_43","unstructured":"Jin, H., Xuan, X., and He, J. (2012, January 5\u20138). A neurobehavioral device to study the neural mechanism in reach to grasp task. Proceedings of the IEEE International Conference on Mechatronics and Automation (ICMA 2012), Chengdu, China."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"36011","DOI":"10.1088\/1741-2552\/ab0bfb","article-title":"Reliability of motor and sensory neural decoding by threshold crossings for intracortical brain\u2013machine interface","volume":"16","author":"Dai","year":"2019","journal-title":"J. Neural Eng."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1961189.1961199","article-title":"Libsvm: A library for support vector machines","volume":"2","author":"Chang","year":"2011","journal-title":"ACM Trans. Intell. Syst. Technol."},{"key":"ref_46","unstructured":"Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016). Prioritized experience replay. arXiv."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Krauledat, M., Tangermann, M., Blankertz, B., and M\u00fcller, K.-R. (2008). Towards Zero Training for Brain-Computer Interfacing. PLoS ONE, 3.","DOI":"10.1371\/journal.pone.0002967"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"683","DOI":"10.1109\/LSP.2009.2022557","article-title":"Composite Common Spatial Pattern for Subject-to-Subject Transfer","volume":"16","author":"Hyohyeong","year":"2009","journal-title":"IEEE Signal Process Lett."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1016\/j.neunet.2014.05.012","article-title":"Bayesian common spatial patterns for multi-subject EEG classification Neural Networks","volume":"57","author":"Kang","year":"2014","journal-title":"Neural Netw."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1109\/TBME.2010.2082539","article-title":"Regularizing Common Spatial Patterns to Improve BCI Designs: Unified Theory and New Algorithms","volume":"58","author":"Lotte","year":"2011","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"026013","DOI":"10.1088\/1741-2560\/9\/2\/026013","article-title":"Stationary common spatial patterns for brain\u2013computer interfacing","volume":"9","author":"Samek","year":"2012","journal-title":"J. Neural Eng."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1463","DOI":"10.1080\/01691864.2015.1070748","article-title":"Reinforcement learning-based shared control for walking-aid robot and its experimental verification","volume":"29","author":"Xu","year":"2015","journal-title":"Adv. Robot."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Huang, J., Yu, X., Wang, Y., and Xiao, X. (2016). An Integrated Wireless Wearable Sensor System for Posture Recognition and Indoor Localization. Sensors, 16.","DOI":"10.3390\/s16111825"},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1016\/j.robot.2014.11.013","article-title":"Posture estimation and human support using wearable sensors and walking-aid robot","volume":"73","author":"Huang","year":"2015","journal-title":"Rob. Autom. Syst."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/19\/5528\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:14:10Z","timestamp":1760177650000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/19\/5528"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,27]]},"references-count":54,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2020,10]]}},"alternative-id":["s20195528"],"URL":"https:\/\/doi.org\/10.3390\/s20195528","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,9,27]]}}}