{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T15:43:09Z","timestamp":1781797389618,"version":"3.54.5"},"reference-count":49,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,1,16]],"date-time":"2022-01-16T00:00:00Z","timestamp":1642291200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2020R1A6A1A03047902"],"award-info":[{"award-number":["2020R1A6A1A03047902"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Korea government (Ministry of Science and ICT, MSIT)","award":["2020R1A2C2005385, 2020R1A2C2004764"],"award-info":[{"award-number":["2020R1A2C2005385, 2020R1A2C2004764"]}]},{"name":"Korean government (MSIT, the Ministry of Trade, Industry and Energy, the Ministry of Health &amp; Welfare, and the Ministry of Food and Drug Safety)","award":["202017D01"],"award-info":[{"award-number":["202017D01"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Motion classification can be performed using biometric signals recorded by electroencephalography (EEG) or electromyography (EMG) with noninvasive surface electrodes for the control of prosthetic arms. However, current single-modal EEG and EMG based motion classification techniques are limited owing to the complexity and noise of EEG signals, and the electrode placement bias, and low-resolution of EMG signals. We herein propose a novel system of two-dimensional (2D) input image feature multimodal fusion based on an EEG\/EMG-signal transfer learning (TL) paradigm for detection of hand movements in transforearm amputees. A feature extraction method in the frequency domain of the EEG and EMG signals was adopted to establish a 2D image. The input images were used for training on a model based on the convolutional neural network algorithm and TL, which requires 2D images as input data. For the purpose of data acquisition, five transforearm amputees and nine healthy controls were recruited. Compared with the conventional single-modal EEG signal trained models, the proposed multimodal fusion method significantly improved classification accuracy in both the control and patient groups. When the two signals were combined and used in the pretrained model for EEG TL, the classification accuracy increased by 4.18\u20134.35% in the control group, and by 2.51\u20133.00% in the patient group.<\/jats:p>","DOI":"10.3390\/s22020680","type":"journal-article","created":{"date-parts":[[2022,1,16]],"date-time":"2022-01-16T20:45:21Z","timestamp":1642365921000},"page":"680","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":26,"title":["Enhanced Recognition of Amputated Wrist and Hand Movements by Deep Learning Method Using Multimodal Fusion of Electromyography and Electroencephalography"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3295-4701","authenticated-orcid":false,"given":"Sehyeon","family":"Kim","sequence":"first","affiliation":[{"name":"Department of Convergence IT Engineering, Pohang University of Science and Technology, Pohang 37673, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1085-5507","authenticated-orcid":false,"given":"Dae Youp","family":"Shin","sequence":"additional","affiliation":[{"name":"Department of Rehabilitation Medicine, College of Medicine, Dankook University, Cheonan 31116, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4233-8574","authenticated-orcid":false,"given":"Taekyung","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Medical Device Management and Research, SAIHST, Sungkyunkwan University, Seoul 03063, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sangsook","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Rehabilitation Medicine, Daejeon Hospital, Daejeon 34383, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9254-4424","authenticated-orcid":false,"given":"Jung Keun","family":"Hyun","sequence":"additional","affiliation":[{"name":"Department of Rehabilitation Medicine, College of Medicine, Dankook University, Cheonan 31116, Korea"},{"name":"Department of Nanobiomedical Science & BK21 NBM Global Research Center for Regenerative Medicine, Dankook University, Cheonan 31116, Korea"},{"name":"Institute of Tissue Regeneration Engineering (ITREN), Dankook University, Cheonan 31116, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8359-8110","authenticated-orcid":false,"given":"Sung-Min","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Convergence IT Engineering, Pohang University of Science and Technology, Pohang 37673, Korea"},{"name":"Department of Electrical Engineering, Pohang University of Science and Technology, Pohang 37673, Korea"},{"name":"Department of Mechanical Engineering, Pohang University of Science and Technology, Pohang 37673, Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1016\/j.apmr.2007.11.005","article-title":"Estimating the Prevalence of Limb Loss in the United States: 2005 to 2050","volume":"89","author":"MacKenzie","year":"2008","journal-title":"Arch. Phys. Med. Rehabil."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e0038","DOI":"10.2106\/JBJS.OA.17.00038","article-title":"Incidence and Characterization of Major Upper-Extremity Amputations in the National Trauma Data Bank","volume":"3","author":"Inkellis","year":"2018","journal-title":"JB JS Open Access"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"656","DOI":"10.1007\/s10926-018-9757-y","article-title":"Factors Influencing Functional Outcomes and Return-to-Work After Amputation: A Review of the Literature","volume":"28","author":"Darter","year":"2018","journal-title":"J. Occup. Rehabil."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1302\/0301-620X.62B3.7410469","article-title":"The below-elbow myo-electric prosthesis. A comparison of the Otto Bock myo-electric prosthesis with the hook and functional hand","volume":"62","author":"Ball","year":"1980","journal-title":"J. Bone Jt. Surgery. Br. Vol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12984-017-0340-0","article-title":"Case-study of a user-driven prosthetic arm design: Bionic hand versus customized body-powered technology in a highly demanding work environment","volume":"15","author":"Schweitzer","year":"2018","journal-title":"J. Neuroeng. Rehabil."},{"key":"ref_6","unstructured":"(2021, December 01). LUKE Arm Detail Page\u2014Mobius Bionics. Available online: http:\/\/www.mobiusbionics.com\/luke-arm\/."},{"key":"ref_7","unstructured":"User Brochure: I-Limb Quantum (2021, December 01). Touch Bionics. Available online: https:\/\/training.touchbionics.com\/index-en.html."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Pohlmeyer, E.A., Fifer, M., Rich, M., Pino, J., Wester, B., Johannes, M., Dohopolski, C., Helder, J., D\u2019Angelo, D., and Beaty, J. (2017). Beyond intuitive anthropomorphic control: Recent achievements using brain computer interface technologies. Micro-and Nanotechnology Sensors, Systems, and Applications IX, International Society for Optics and Photonics.","DOI":"10.1117\/12.2263886"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"101637","DOI":"10.1016\/j.bspc.2019.101637","article-title":"Hand gesture recognition based on motor unit spike trains decoded from high-density electromyography","volume":"55","author":"Chen","year":"2019","journal-title":"Biomed. Signal Process. Control."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1038\/s41551-016-0025","article-title":"Man\/machine interface based on the discharge timings of spinal motor neurons after targeted muscle reinnervation","volume":"1","author":"Farina","year":"2017","journal-title":"Nat. Biomed. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1186\/s12984-018-0361-3","article-title":"Evaluation of EMG pattern recognition for upper limb prosthesis control: A case study in comparison with direct myoelectric control","volume":"15","author":"Resnik","year":"2018","journal-title":"J. Neuroeng. Rehabil."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"101838","DOI":"10.1016\/j.bspc.2019.101838","article-title":"Generalization of a wavelet-based algorithm to adaptively detect activation intervals in weak and noisy myoelectric signals","volume":"58","author":"Varrecchia","year":"2020","journal-title":"Biomed. Signal Process. Control."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"907","DOI":"10.5535\/arm.2011.35.6.907","article-title":"A Survey on Activities of Daily Living and Occupations of Upper Extremity Amputees","volume":"35","author":"Jang","year":"2011","journal-title":"Ann. Rehabil. Med."},{"key":"ref_14","first-page":"294","article-title":"Prosthesis rejection in acquired major upper-limb amputees: A population-based survey","volume":"7","author":"Franklin","year":"2011","journal-title":"Disabil. Rehabil. Assist. Technol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"26274","DOI":"10.1073\/pnas.1902276116","article-title":"From thought to action: The brain\u2013machine interface in posterior parietal cortex","volume":"116","author":"Andersen","year":"2019","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Uejima, T., Kita, K., Fujii, T., Kato, R., Takita, M., and Yokoi, H. (2009, January 3\u20136). Motion classification using epidural electrodes for low-invasive brain-machine interface. Proceedings of the 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN, USA.","DOI":"10.1109\/IEMBS.2009.5333547"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1016\/j.mayocp.2011.12.008","article-title":"Brain-Computer Interfaces in Medicine","volume":"87","author":"Shih","year":"2012","journal-title":"Mayo Clin. Proc."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"934","DOI":"10.1162\/neco.2007.19.4.934","article-title":"Model Selection for Convolutive ICA with an Application to Spatiotemporal Analysis of EEG","volume":"19","author":"Dyrholm","year":"2007","journal-title":"Neural Comput."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1109\/RBME.2019.2950897","article-title":"Selection of Features and Classifiers for EMG-EEG-Based Upper Limb Assistive Devices\u2014A Review","volume":"13","author":"Khan","year":"2020","journal-title":"IEEE Rev. Biomed. Eng."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1109\/TNSRE.2016.2601240","article-title":"A Deep Learning Scheme for Motor Imagery Classification based on Restricted Boltzmann Machines","volume":"25","author":"Lu","year":"2017","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"906","DOI":"10.1126\/science.aaa5417","article-title":"Decoding motor imagery from the posterior parietal cortex of a tetraplegic human","volume":"348","author":"Aflalo","year":"2015","journal-title":"Science"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1645","DOI":"10.1109\/TCBB.2021.3052811","article-title":"EEG-Based Brain-Computer Interfaces (BCIs): A Survey of Recent Studies on Signal Sensing Technologies and Computational Intelligence Approaches and Their Applications","volume":"18","author":"Gu","year":"2021","journal-title":"IEEE\/ACM Trans. Comput. Biol. Bioinform."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"011001","DOI":"10.1088\/1741-2552\/14\/1\/011001","article-title":"Support vector machines to detect physiological patterns for EEG and EMG-based human\u2013computer interaction: A review","volume":"14","author":"Quitadamo","year":"2017","journal-title":"J. Neural Eng."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Ebrahimi, F., Mikaeili, M., Estrada, E., and Nazeran, H. (2008, January 20\u201325). Automatic sleep stage classification based on EEG signals by using neural networks and wavelet packet coefficients. Proceedings of the 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vancouver, BC, Canada.","DOI":"10.1109\/IEMBS.2008.4649365"},{"key":"ref_25","unstructured":"Said, A.B., Mohamed, A., Elfouly, T., Harras, K., and Wang, Z.J. (2017, January 19\u201322). Multimodal Deep Learning Approach for Joint EEG-EMG Data Compression and Classification. Proceedings of the 2017 IEEE Wireless Communications and Networking Conference (WCNC), San Francisco, CA, USA."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"zsaa106","DOI":"10.1093\/sleep\/zsaa106","article-title":"Estimating daytime sleepiness with previous night electroencephalography, electrooculography, and electromyography spectrograms in patients with suspected sleep apnea using a convolutional neural network","volume":"43","author":"Nikkonen","year":"2020","journal-title":"Sleep"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"38565","DOI":"10.1038\/srep38565","article-title":"Noninvasive Electroencephalogram Based Control of a Robotic Arm for Reach and Grasp Tasks","volume":"6","author":"Meng","year":"2016","journal-title":"Sci. Rep."},{"key":"ref_28","unstructured":"Perotto, A.O. (2011). Anatomical Guide for the Electromyographer: The Limbs and Trunk, Charles C Thomas Pub Ltd.. [5th ed.]."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"935364","DOI":"10.1155\/2011\/935364","article-title":"BioSig: The Free and Open Source Software Library for Biomedical Signal Processing","volume":"2011","author":"Vidaurre","year":"2011","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Park, T., Lee, M., Jeong, T., Shin, Y.-I., and Park, S.-M. (2020). Quantitative Analysis of EEG Power Spectrum and EMG Median Power Frequency Changes after Continuous Passive Motion Mirror Therapy System. Sensors, 20.","DOI":"10.3390\/s20082354"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"458","DOI":"10.3389\/fnhum.2019.00458","article-title":"The Application of EEG Mu Rhythm Measures to Neurophysiological Research in Stuttering","volume":"13","author":"Jenson","year":"2019","journal-title":"Front. Hum. Neurosci."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"22364","DOI":"10.1038\/s41598-021-01368-2","article-title":"Cortical sensorimotor activity in the execution and suppression of discrete and rhythmic movements","volume":"11","author":"Hervault","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_33","unstructured":"De Luca, C.J. (2002). Surface Electromyography: Detection and Recording, DelSys Incorporated."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"2352","DOI":"10.1162\/neco_a_00990","article-title":"Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review","volume":"29","author":"Rawat","year":"2017","journal-title":"Neural Comput."},{"key":"ref_35","unstructured":"Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. (2014, January 8\u201313). How transferable are features in deep neural networks?. arXiv preprint arXiv:1411.1792. Proceedings of the Advances in Neural Information Processing Systems 27 (NIPS 2014), Montr\u00e9al, QC, Canada."},{"key":"ref_36","unstructured":"Marcelino, P. (2021, October 23). Transfer Learning from Pre-Trained Models. Available online: https:\/\/towardsdatascience.com\/transfer-learning-from-pre-trained-models-f2393f124751."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"646","DOI":"10.1016\/j.compeleceng.2017.04.003","article-title":"Pattern recognition of electromyography signals based on novel time domain features for amputees\u2019 limb motion classification","volume":"67","author":"Samuel","year":"2018","journal-title":"Comput. Electr. Eng."},{"key":"ref_38","unstructured":"Chen, T., Moreau, T., Jiang, Z., Zheng, L., Yan, E., Shen, H., Cowan, M., Wang, L., Hu, Y., and Ceze, L. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. Proceedings of Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"314","DOI":"10.3389\/fnins.2020.00314","article-title":"Motor Cortex Reorganization in Limb Amputation: A Systematic Review of TMS Motor Mapping Studies","volume":"14","author":"Gunduz","year":"2020","journal-title":"Front. Neurosci."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"518","DOI":"10.1053\/ampr.2003.50127","article-title":"Evaluation of the validity of the prosthetic upper extremity functional index for children","volume":"84","author":"Wright","year":"2003","journal-title":"Arch. Phys. Med. Rehabil."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13673-019-0194-5","article-title":"Multi-sensor fusion based on multiple classifier systems for human activity identification","volume":"9","author":"Nweke","year":"2019","journal-title":"Hum.-Cent. Comput. Inf. Sci."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1123\/mc.2020-0041","article-title":"Generalized Finger Motion Classification Model Based on Motor Unit Voting","volume":"25","author":"Liu","year":"2020","journal-title":"Mot. Control."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1109\/JBHI.2019.2926307","article-title":"Finger Joint Angle Estimation Based on Motoneuron Discharge Activities","volume":"24","author":"Dai","year":"2019","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1186\/s12984-016-0212-z","article-title":"A motion-classification strategy based on sEMG-EEG signal combination for upper-limb amputees","volume":"14","author":"Li","year":"2017","journal-title":"J. Neuroeng. Rehabil."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"066030","DOI":"10.1088\/1741-2552\/ab4063","article-title":"Regression-based reconstruction of human grip force trajectories with noninvasive scalp electroencephalography","volume":"16","author":"Paek","year":"2019","journal-title":"J. Neural Eng."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"116943","DOI":"10.1016\/j.neuroimage.2020.116943","article-title":"Brain (re)organisation following amputation: Implications for phantom limb pain","volume":"218","author":"Makin","year":"2020","journal-title":"NeuroImage"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"11770","DOI":"10.3390\/s140711770","article-title":"Physiological Sensor Signals Classification for Healthcare Using Sensor Data Fusion and Case-Based Reasoning","volume":"14","author":"Begum","year":"2014","journal-title":"Sensors"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"2295","DOI":"10.1109\/JPROC.2017.2761740","article-title":"Efficient Processing of Deep Neural Networks: A Tutorial and Survey","volume":"105","author":"Sze","year":"2017","journal-title":"Proc. IEEE"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"5455","DOI":"10.1007\/s10462-020-09825-6","article-title":"A survey of the recent architectures of deep convolutional neural networks","volume":"53","author":"Khan","year":"2020","journal-title":"Artif. Intell. Rev."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/2\/680\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:02:13Z","timestamp":1760133733000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/2\/680"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,16]]},"references-count":49,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,1]]}},"alternative-id":["s22020680"],"URL":"https:\/\/doi.org\/10.3390\/s22020680","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,16]]}}}