{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T20:12:58Z","timestamp":1780344778953,"version":"3.54.1"},"reference-count":52,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2017,10,12]],"date-time":"2017-10-12T00:00:00Z","timestamp":1507766400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61472216"],"award-info":[{"award-number":["61472216"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Many people suffer from high mental workload which may threaten human health and cause serious accidents. Mental workload estimation is especially important for particular people such as pilots, soldiers, crew and surgeons to guarantee the safety and security. Different physiological signals have been used to estimate mental workload based on the n-back task which is capable of inducing different mental workload levels. This paper explores a feature weight driven signal fusion method and proposes interactive mutual information modeling (IMIM) to increase the mental workload classification accuracy. We used EEG and ECG signals to validate the effectiveness of the proposed method for heterogeneous bio-signal fusion. The experiment of mental workload estimation consisted of signal recording, artifact removal, feature extraction, feature weight calculation, and classification. Ten subjects were invited to take part in easy, medium and hard tasks for the collection of EEG and ECG signals in different mental workload levels. Therefore, heterogeneous physiological signals of different mental workload states were available for classification. Experiments reveal that ECG can be utilized as a supplement of EEG to optimize the fusion model and improve mental workload estimation. Classification results show that the proposed bio-signal fusion method IMIM can increase the classification accuracy in both feature level and classifier level fusion. This study indicates that multi-modal signal fusion is promising to identify the mental workload levels and the fusion strategy has potential application of mental workload estimation in cognitive activities during daily life.<\/jats:p>","DOI":"10.3390\/s17102315","type":"journal-article","created":{"date-parts":[[2017,10,11]],"date-time":"2017-10-11T12:17:47Z","timestamp":1507724267000},"page":"2315","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["Feature Weight Driven Interactive Mutual Information Modeling for Heterogeneous Bio-Signal Fusion to Estimate Mental Workload"],"prefix":"10.3390","volume":"17","author":[{"given":"Pengbo","family":"Zhang","sequence":"first","affiliation":[{"name":"State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing 100084, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xue","family":"Wang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing 100084, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junfeng","family":"Chen","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing 100084, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"You","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijing 100084, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2017,10,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.measurement.2015.07.008","article-title":"Sparse EEG compressive sensing for web-enabled person identification","volume":"74","author":"Dai","year":"2015","journal-title":"Measurement"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1016\/j.neuroimage.2011.07.047","article-title":"Adaptive training using an artificial neural network and EEG metrics for within- and cross-task workload classification","volume":"59","author":"Baldwin","year":"2012","journal-title":"Neuroimage"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"743","DOI":"10.1109\/TBME.2016.2574812","article-title":"Drowsiness Detection by Bayesian-Copula Discriminant Classifier Based on EEG Signals During Daytime Short Nap","volume":"64","author":"Qian","year":"2017","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"026019","DOI":"10.1088\/1741-2560\/13\/2\/026019","article-title":"Efficient mental workload estimation using task-independent EEG features","volume":"13","author":"Roy","year":"2016","journal-title":"J. Neural Eng."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"715","DOI":"10.1109\/JBHI.2016.2532354","article-title":"Driver Fatigue Classification With Independent Component by Entropy Rate Bound Minimization Analysis in an EEG-Based System","volume":"21","author":"Chai","year":"2017","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1016\/j.bspc.2016.11.013","article-title":"Cross-session classification of mental workload levels using EEG and an adaptive deep learning model","volume":"33","author":"Yin","year":"2017","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.patrec.2017.05.020","article-title":"Deep long short-term memory structures model temporal dependencies improving cognitive workload estimation","volume":"94","author":"Hefron","year":"2017","journal-title":"Pattern Recognit. Lett."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1016\/j.bspc.2011.07.004","article-title":"Real-time detection of workload changes using heart rate variability","volume":"7","author":"Hoover","year":"2012","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"400","DOI":"10.1016\/j.bspc.2013.01.007","article-title":"Multimodal information improves the rapid detection of mental fatigue","volume":"8","author":"Laurent","year":"2013","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1016\/j.ijpsycho.2015.04.006","article-title":"Evaluation of 3D cognitive fatigue using heart\u2013brain synchronization","volume":"97","author":"Park","year":"2015","journal-title":"Int. J. Psychophysiol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1140","DOI":"10.1016\/j.apergo.2014.02.001","article-title":"Assessment of early onset of driver fatigue using multimodal fatigue measures in a static simulator","volume":"45","author":"Jagannath","year":"2014","journal-title":"Appl. Ergon."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"176","DOI":"10.3389\/fnbeh.2015.00176","article-title":"Dissociation between mental fatigue and motivational state during prolonged mental activity","volume":"9","author":"Gergelyfi","year":"2015","journal-title":"Front. Behav. Neurosci."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"354","DOI":"10.3390\/s7030354","article-title":"An improved co-evolutionary particle swarm optimization for wireless sensor networks with dynamic deployment","volume":"7","author":"Wang","year":"2007","journal-title":"Sensors"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1016\/j.neuroimage.2013.11.007","article-title":"Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals","volume":"102","author":"Verma","year":"2014","journal-title":"Neuroimage"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1016\/j.inffus.2016.09.005","article-title":"Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges","volume":"35","author":"Gravina","year":"2017","journal-title":"Inf. Fusion"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"322","DOI":"10.3389\/fnins.2014.00322","article-title":"Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload","volume":"8","author":"Hogervorst","year":"2014","journal-title":"Front. Neurosci."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1016\/j.cmpb.2013.09.007","article-title":"Operator functional state classification using least-square support vector machine based recursive feature elimination technique","volume":"113","author":"Yin","year":"2014","journal-title":"Comput. Meth. Programs Biomed."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1007\/s10916-015-0219-1","article-title":"Mutual information analysis of sleep eeg in detecting psycho-physiological insomnia","volume":"39","author":"Tunga","year":"2015","journal-title":"J. Med. Syst."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1293","DOI":"10.1111\/psyp.12500","article-title":"When flanker meets the n-back: What EEG and pupil dilation data reveal about the interplay between the two central-executive working memory functions inhibition and updating","volume":"52","author":"Scharinger","year":"2015","journal-title":"Psychophysiology"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1037\/h0043688","article-title":"Age differences in short-term retention of rapidly changing information","volume":"55","author":"Kirchner","year":"1958","journal-title":"J. Exp. Psychol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1007\/s10648-010-9130-y","article-title":"Using Electroencephalography to Measure Cognitive Load","volume":"22","author":"Antonenko","year":"2010","journal-title":"Educ. Psychol. Rev."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"424","DOI":"10.1109\/THMS.2015.2476818","article-title":"Using Wireless EEG Signals to Assess Memory Workload in the n-Back Task","volume":"46","author":"Wang","year":"2016","journal-title":"IEEE Trans. Hum.-Mach. Syst."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Al-Qazzaz, N., Hamid Bin Mohd Ali, S., Ahmad, S., Islam, M., and Escudero, J. (2017). Automatic Artifact Removal in EEG of Normal and Demented Individuals Using ICA\u2013WT during Working Memory Tasks. Sensors, 17.","DOI":"10.3390\/s17061326"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.jneumeth.2003.10.009","article-title":"EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis","volume":"134","author":"Delorme","year":"2004","journal-title":"J. Neurosci. Methods"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1111\/j.1469-8986.2010.01061.x","article-title":"ADJUST: An automatic EEG artifact detector based on the joint use of spatial and temporal features","volume":"48","author":"Mognon","year":"2011","journal-title":"Psychophysiology"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1016\/j.ijpsycho.2015.10.004","article-title":"Towards an effective cross-task mental workload recognition model using electroencephalography based on feature selection and support vector machine regression","volume":"98","author":"Ke","year":"2015","journal-title":"Int. J. Psychophysiol."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"2840","DOI":"10.1109\/TBME.2014.2331189","article-title":"In-Flight Automatic Detection of Vigilance States Using a Single EEG Channel","volume":"61","author":"Sauvet","year":"2014","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.neuroimage.2011.07.091","article-title":"The effects of day-to-day variability of physiological data on operator functional state classification","volume":"59","author":"Christensen","year":"2012","journal-title":"Neuroimage"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"703","DOI":"10.3389\/fnhum.2014.00703","article-title":"An EEG-based mental workload estimator trained on working memory task can work well under simulated multi-attribute task","volume":"8","author":"Ke","year":"2014","journal-title":"Front. Hum. Neurosci."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"54","DOI":"10.3389\/fnins.2015.00054","article-title":"Electrode replacement does not affect classification accuracy in dual-session use of a passive brain-computer interface for assessing cognitive workload","volume":"9","author":"Estepp","year":"2015","journal-title":"Front. Neurosci."},{"key":"ref_31","first-page":"114","article-title":"EEG-based workload estimation across affective contexts","volume":"8","author":"Jeunet","year":"2014","journal-title":"Front. Neurosci."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"372","DOI":"10.3389\/fnins.2014.00372","article-title":"Artificial Neural Network classification of operator workload with an assessment of time variation and noise-enhancement to increase performance","volume":"8","author":"Casson","year":"2014","journal-title":"Front. Neurosci."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1016\/j.neuroimage.2011.07.094","article-title":"Cross-subject workload classification with a hierarchical Bayes model","volume":"59","author":"Wang","year":"2012","journal-title":"Neuroimage"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Zhao, Y.X., and Chou, C.H. (2016). Feature Selection Method Based on Neighborhood Relationships: Applications in EEG Signal Identification and Chinese Character Recognition. Sensors, 16.","DOI":"10.3390\/s16060871"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"308","DOI":"10.3389\/fnhum.2015.00308","article-title":"Effects of user mental state on EEG-BCI performance","volume":"9","author":"Myrden","year":"2015","journal-title":"Front. Hum. Neurosci."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/j.cmpb.2014.04.011","article-title":"Identification of temporal variations in mental workload using locally-linear- embedding-based EEG feature reduction and support-vector-machine-based clustering and classification techniques","volume":"115","author":"Yin","year":"2014","journal-title":"Comput. Meth. Programs Biomed."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"045008","DOI":"10.1088\/1741-2560\/9\/4\/045008","article-title":"Estimating workload using EEG spectral power and ERPs in the n-back task","volume":"9","author":"Brouwer","year":"2012","journal-title":"J. Neural Eng."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1016\/j.ijpsycho.2014.12.008","article-title":"The efficacy of auditory probes in indexing cognitive workload is dependent on stimulus complexity","volume":"95","author":"Dyke","year":"2015","journal-title":"Int. J. Psychophysiol."},{"key":"ref_39","unstructured":"Ramshur, J. (2010). Design, Evaluation, and Application of Heart Rate Variability Analysis Software (HRVAS). [Master\u2019s Thesis, University of Memphis]."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Zhang, P., Wang, X., Li, X., and Dai, P. (2016, January 23\u201326). EEG feature selection based on weighted-normalized mutual information for mental fatigue classification. Proceedings of the 2016 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), Taipei, Taiwan.","DOI":"10.1109\/I2MTC.2016.7520423"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1007\/BF02289159","article-title":"Multivariate information transmission","volume":"19","author":"McGill","year":"1954","journal-title":"Psychometrika"},{"key":"ref_42","unstructured":"Jakulin, A., and Bratko, I. (2004, March 02). Quantifying and Visualizing Attribute Interactions: An Approach Based on Entropy. Available online: http:\/\/arxiv.org\/abs\/cs.AI\/0308002."},{"key":"ref_43","unstructured":"Bell, A. (2003, January 1\u20134). The co-information lattice. Proceedings of the 4th international symposium on Independent Component Analysis and Blind Source Separation (ICA 2003), Nara, Japan."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1529","DOI":"10.1109\/TPAMI.2014.2372791","article-title":"A Semidefinite Programming Based Search Strategy for Feature Selection with Mutual Information Measure","volume":"37","author":"Naghibi","year":"2015","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1007\/BF01588795","article-title":"An O(n3L) primal interior point algorithm for convex quadratic programming","volume":"49","author":"Goldfarb","year":"1990","journal-title":"Math. Programm."},{"key":"ref_46","unstructured":"Lofberg, J. (2004, January 2\u20134). YALMIP: A toolbox for modeling and optimization in MATLAB. Proceedings of the 2004 IEEE International Symposium on Computer Aided Control Systems Design, New Orleans, LA, USA."},{"key":"ref_47","unstructured":"MOSEK-ApS (2017, August 24). The MOSEK Optimization Toolbox for MATLAB Manual. Available online: http:\/\/docs.mosek.com\/8.0\/toolbox\/."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1109\/TBME.2007.890733","article-title":"A feature selection method for multilevel mental fatigue EEG classification","volume":"54","author":"Shen","year":"2007","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Vedaldi, A., Gulshan, V., Varma, M., and Zisserman, A. (October, January 29). Multiple Kernels for Object Detection. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan.","DOI":"10.1109\/ICCV.2009.5459183"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1135","DOI":"10.1109\/TPAMI.2012.198","article-title":"Linear Dependency Modeling for Classifier Fusion and Feature Combination","volume":"35","author":"Ma","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Gehler, P., and Nowozin, S. (October, January 29). On Feature Combination for Multiclass Object Classification. Proceedings of the 2009 IEEE 12th International Conference on Computer Vision, Kyoto, Japan.","DOI":"10.1109\/ICCV.2009.5459169"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1109\/TAMD.2015.2441960","article-title":"Beyond Subjective Self-Rating: EEG Signal Classification of Cognitive Workload","volume":"7","author":"Zarjam","year":"2015","journal-title":"IEEE Trans. Auton. Ment. Dev."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/17\/10\/2315\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:47:00Z","timestamp":1760208420000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/17\/10\/2315"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,10,12]]},"references-count":52,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2017,10]]}},"alternative-id":["s17102315"],"URL":"https:\/\/doi.org\/10.3390\/s17102315","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,10,12]]}}}