{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T07:29:19Z","timestamp":1780471759934,"version":"3.54.1"},"reference-count":52,"publisher":"MDPI AG","issue":"13","license":[{"start":{"date-parts":[[2021,7,2]],"date-time":"2021-07-02T00:00:00Z","timestamp":1625184000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Cognitive workload, being a quantitative measure of mental effort, draws significant interest of researchers, as it allows to monitor the state of mental fatigue. Estimation of cognitive workload becomes especially important for job positions requiring outstanding engagement and responsibility, e.g., air-traffic dispatchers, pilots, car or train drivers. Cognitive workload estimation finds its applications also in the field of education material preparation. It allows to monitor the difficulty degree for specific tasks enabling to adjust the level of education materials to typical abilities of students. In this study, we present the results of research conducted with the goal of examining the influence of various fuzzy or non-fuzzy aggregation functions upon the quality of cognitive workload estimation. Various classic machine learning models were successfully applied to the problem. The results of extensive in-depth experiments with over 2000 aggregation operators shows the applicability of the approach based on the aggregation functions. Moreover, the approach based on aggregation process allows for further improvement of classification results. A wide range of aggregation functions is considered and the results suggest that the combination of classical machine learning models and aggregation methods allows to achieve high quality of cognitive workload level recognition preserving low computational cost.<\/jats:p>","DOI":"10.3390\/s21134542","type":"journal-article","created":{"date-parts":[[2021,7,2]],"date-time":"2021-07-02T10:06:34Z","timestamp":1625220394000},"page":"4542","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["On the Improvement of Eye Tracking-Based Cognitive Workload Estimation Using Aggregation Functions"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4618-7937","authenticated-orcid":false,"given":"Monika","family":"Kaczorowska","sequence":"first","affiliation":[{"name":"Department of Computer Science, Lublin University of Technology, 20-618 Lublin, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pawe\u0142","family":"Karczmarek","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Lublin University of Technology, 20-618 Lublin, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1055-5344","authenticated-orcid":false,"given":"Ma\u0142gorzata","family":"Plechawska-W\u00f3jcik","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Lublin University of Technology, 20-618 Lublin, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mikhail","family":"Tokovarov","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Lublin University of Technology, 20-618 Lublin, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,7,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1016\/j.eng.2018.11.025","article-title":"Neural mechanisms of mental fatigue revisited: New insights from the brain connectome","volume":"5","author":"Qi","year":"2019","journal-title":"Engineering"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1016\/j.eswa.2017.01.040","article-title":"Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers","volume":"85","author":"Chen","year":"2017","journal-title":"Expert Syst. Appl."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1016\/j.neuroimage.2011.07.094","article-title":"Cross-subject workload classification with a hierarchical Bayes model","volume":"59","author":"Wang","year":"2012","journal-title":"NeuroImage"},{"key":"ref_4","unstructured":"Walter, C., Wolter, P., Rosenstiel, W., Bogdan, M., and Sp\u00fcler, M. (2014, January 16\u201321). Towards cross-subject workload prediction. Proceedings of the 6th International Brain-Computer Interface Conference, Graz, Austria."},{"key":"ref_5","unstructured":"Thodoroff, P., Pineau, J., and Lim, A. (2016, January 19\u201320). Learning robust features using deep learning for automatic seizure detection. Proceedings of the Machine learning for healthcare conference, Los Angeles, CA, USA."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"295","DOI":"10.3389\/fnhum.2019.00295","article-title":"Theories and methods for labeling cognitive workload: Classification and transfer learning","volume":"13","author":"McKendrick","year":"2019","journal-title":"Front. Hum. Neurosci."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Fridman, L., Reimer, B., Mehler, B., and Freeman, W.T. (2018, January 21\u201326). Cognitive load estimation in the wild. Proceedings of the 2018 Chi Conference on Human Factors in Computing Systems, Montreal, QC, Canada.","DOI":"10.1145\/3173574.3174226"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Appel, T., Scharinger, C., Gerjets, P., and Kasneci, E. (2018, January 14\u201317). Cross-subject workload classification using pupil-related measures. Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, Warsaw, Poland.","DOI":"10.1145\/3204493.3204531"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Almogbel, M.A., Dang, A.H., and Kameyama, W. (2018, January 11\u201314). EEG-signals based cognitive workload detection of vehicle driver using deep learning. Proceedings of the 2018 20th International Conference on Advanced Communication Technology (ICACT), Chuncheon, Korea.","DOI":"10.23919\/ICACT.2018.8323715"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Hefron, R., Borghetti, B., Schubert Kabban, C., Christensen, J., and Estepp, J. (2018). Cross-participant EEG-based assessment of cognitive workload using multi-path convolutional recurrent neural networks. Sensors, 18.","DOI":"10.3390\/s18051339"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Lobo, J.L., Ser, J.D., De Simone, F., Presta, R., Collina, S., and Moravek, Z. (2016, January 14\u201316). Cognitive workload classification using eye-tracking and EEG data. Proceedings of the International Conference on Human-Computer Interaction in Aerospace, Paris, France.","DOI":"10.1145\/2950112.2964585"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Almogbel, M.A., Dang, A.H., and Kameyama, W. (2019, January 17\u201320). Cognitive workload detection from raw EEG-signals of vehicle driver using deep learning. Proceedings of the 2019 21st International Conference on Advanced Communication Technology (ICACT), PyeongChang, Korea.","DOI":"10.23919\/ICACT.2019.8702048"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2186","DOI":"10.1016\/j.compbiomed.2013.08.021","article-title":"Estimating cognitive workload using wavelet entropy-based features during an arithmetic task","volume":"43","author":"Zarjam","year":"2013","journal-title":"Comput. Biol. Med."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1016\/j.neuropsychologia.2019.04.004","article-title":"Exploring the fatigue affecting electroencephalography based functional brain networks during real driving in young males","volume":"129","author":"Chen","year":"2019","journal-title":"Neuropsychologia"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1016\/j.artmed.2018.06.005","article-title":"Detecting mental fatigue from eye-tracking data gathered while watching video: Evaluation in younger and older adults","volume":"91","author":"Yamada","year":"2018","journal-title":"Artif. Intell. Med."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1109\/TBME.2010.2077291","article-title":"Driver drowsiness classification using fuzzy wavelet-packet-based feature-extraction algorithm","volume":"58","author":"Khushaba","year":"2010","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"374","DOI":"10.1016\/j.neucom.2020.06.009","article-title":"Deep learning for EEG-based biometric recognition","volume":"410","author":"Maiorana","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1016\/j.image.2016.05.018","article-title":"EEG-based prediction of driver\u2019s cognitive performance by deep convolutional neural network","volume":"47","author":"Hajinoroozi","year":"2016","journal-title":"Signal Process. Image Commun."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Wobrock, D., Frey, J., Graeff, D., De La Rivi\u00e8re, J.B., Castet, J., and Lotte, F. (2015, January 14\u201318). Continuous mental effort evaluation during 3d object manipulation tasks based on brain and physiological signals. Proceedings of the IFIP Conference on Human-Computer Interaction, Bamberg, Germany.","DOI":"10.1007\/978-3-319-22701-6_35"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Bozkir, E., Geisler, D., and Kasneci, E. (2019, January 23\u201327). Person independent, privacy preserving, and real time assessment of cognitive load using eye tracking in a virtual reality setup. Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan.","DOI":"10.1109\/VR.2019.8797758"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Zhao, Z., Wang, C., Niu, Y., Shen, L., Ma, Z., and Wu, L. (2019, January 21\u201322). Adjustable Autonomy for Human-UAVs Collaborative Searching Using Fuzzy Cognitive Maps. Proceedings of the 2019 2nd China Symposium on Cognitive Computing and Hybrid Intelligence (CCHI), Xi\u2019an, China.","DOI":"10.1109\/CCHI.2019.8901937"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Naqvi, R.A., Arsalan, M., and Park, K.R. (2017). Fuzzy system-based target selection for a NIR camera-based gaze tracker. Sensors, 17.","DOI":"10.3390\/s17040862"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Yusuf, A.B., Kor, A.L., and Tawfik, H. (2019, January 14\u201319). Development of a Simulation Experiment to Investigate In-Flight Startle using Fuzzy Cognitive Maps and Pupillometry. Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary.","DOI":"10.1109\/IJCNN.2019.8851828"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Fatimah, B., Pramanick, D., and Shivashankaran, P. (2020, January 1\u20133). Automatic detection of mental arithmetic task and its difficulty level using EEG signals. Proceedings of the 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT), Kharagpur, India.","DOI":"10.1109\/ICCCNT49239.2020.9225647"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1016\/j.bbe.2019.12.002","article-title":"Classification of pilots\u2019 mental states using a multimodal deep learning network","volume":"40","author":"Han","year":"2020","journal-title":"Biocybern. Biomed. Eng."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Becerra-S\u00e1nchez, P., Reyes-Munoz, A., and Guerrero-Iba\u00f1ez, A. (2020). Feature selection model based on EEG signals for assessing the cognitive workload in drivers. Sensors, 20.","DOI":"10.20944\/preprints202009.0521.v1"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Dell\u2019Agnola, F., Momeni, N., Arza, A., and Atienza, D. (2020, January 19\u201324). Cognitive workload monitoring in virtual reality based rescue missions with drones. Proceedings of the International Conference on Human-Computer Interaction, Copenhagen, Denmark.","DOI":"10.1007\/978-3-030-49695-1_26"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Chakladar, D.D., Dey, S., Roy, P.P., and Iwamura, M. (2021, January 10\u201315). EEG-Based Cognitive State Assessment Using Deep Ensemble Model and Filter Bank Common Spatial Pattern. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.","DOI":"10.1109\/ICPR48806.2021.9412869"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Alsina, C., Schweizer, B., and Frank, M.J. (2006). Associative Functions: Triangular Norms and Copulas, World Scientific.","DOI":"10.1142\/9789812774200"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Karczmarek, P., Kiersztyn, A., and Pedrycz, W. (2019, January 16\u201320). Generalizations of Aggregation Functions for Face Recognition. Proceedings of the International Conference on Artificial Intelligence and Soft Computing, Zakopane, Poland.","DOI":"10.1007\/978-3-030-20915-5_17"},{"key":"ref_31","unstructured":"Beliakov, G., Pradera, A., and Calvo, T. (2007). Aggregation Functions: A Guide for Practitioners, Springer."},{"key":"ref_32","unstructured":"Calvo, T., Mayor, G., and Mesiar, R. (2012). Aggregation Operators: New Trends and Applications, Physica."},{"key":"ref_33","unstructured":"G\u0105golewski, M. (2015). Data Fusion: Theory, Methods, and Applications, Institute of Computer Science, Polish Academy of Sciences."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Grabisch, M., Marichal, J.L., Mesiar, R., and Pap, E. (2009). Aggregation Functions (No. 127), Cambridge University Press.","DOI":"10.1017\/CBO9781139644150"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1007\/978-3-540-73723-0_7","article-title":"A review of aggregation functions. Fuzzy sets and their extensions: Representation, aggregation and models","volume":"220","author":"Mesiar","year":"2008","journal-title":"Stud. Fuzziness Soft Comput."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ins.2010.08.043","article-title":"Aggregation functions: Means","volume":"181","author":"Grabisch","year":"2011","journal-title":"Inf. Sci."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.ins.2010.08.040","article-title":"Aggregation functions: Construction methods, conjunctive, disjunctive and mixed classes","volume":"181","author":"Grabisch","year":"2011","journal-title":"Inf. Sci."},{"key":"ref_38","unstructured":"Klement, E.P., Mesiar, R., and Pap, E. (2013). Triangular Norms, Springer Science & Business Media."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Klement, E.P., and Mesiar, R. (2005). Logical, Algebraic, Analytic and Probabilistic Aspects of Triangular Norms, Elsevier.","DOI":"10.1016\/B978-044451814-9\/50004-5"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Yager, R.R., and Kacprzyk, J. (2012). The Ordered Weighted Averaging Operators: Theory and Applications, Springer Science & Business Media.","DOI":"10.1007\/978-3-642-17910-5"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"131","DOI":"10.5802\/aif.53","article-title":"Theory of capacities","volume":"5","author":"Choquet","year":"1954","journal-title":"Annales de l\u2019institut Fourier"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1016\/0377-2217(95)00176-X","article-title":"The application of fuzzy integrals in multicriteria decision making","volume":"89","author":"Grabisch","year":"1996","journal-title":"Eur. J. Oper. Res."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Bustince, H., Sanz, J.A., Lucca, G., Dimuro, G.P., Bedregal, B., and Mesiar, R. (2013, January 7\u201310). Pre-aggregation functions: Definition, properties and construction methods. Proceedings of the 2016 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) 2016, Hyderabad, India.","DOI":"10.1109\/FUZZ-IEEE.2016.7737700"},{"key":"ref_44","first-page":"1","article-title":"A comprehensive experimental comparison of the aggregation techniques for face recognition","volume":"16","author":"Karczmarek","year":"2019","journal-title":"Iran. J. Fuzzy Syst."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Lucca, G., Sanz, J.A., Dimuro, G.P., Bedregal, B., Mesiar, R., Koles\u00e1rov\u00e1, A., and Bustince, H. (2015, January 21\u201323). The notion of pre-aggregation function. Proceedings of the International Conference on Modeling Decisions for Artificial Intelligence 2015, Sk\u00f6vde, Sweden.","DOI":"10.1007\/978-3-319-23240-9_3"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1047","DOI":"10.1007\/s40815-017-0355-5","article-title":"Generalized choquet integral for face recognition","volume":"20","author":"Karczmarek","year":"2018","journal-title":"Int. J. Fuzzy Syst."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1016\/j.inffus.2019.10.005","article-title":"The state-of-art of the generalizations of the Choquet integral: From aggregation and pre-aggregation to ordered directionally monotone functions","volume":"57","author":"Dimuro","year":"2020","journal-title":"Inf. Fusion"},{"key":"ref_48","unstructured":"Karczmarek, P. (2018). Selected Problems of Face Recognition and Decision-Making Theory, Wydawnictwo Politechniki Lubelskiej."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1076\/jcen.24.3.383.981","article-title":"From the Binet\u2013Simon to the Wechsler\u2013Bellevue: Tracing the history of intelligence testing","volume":"24","author":"Boake","year":"2002","journal-title":"J. Clin. Exp. Neuropsychol."},{"key":"ref_50","unstructured":"Pedrycz, W., and Gomide, F. (1988). An Introduction to Fuzzy Sets: Analysis and Design, MIT Press."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Torra, V., and Narukawa, Y. (2007). Modeling Decisions: Information Fusion and Aggregation Operators, Springer Science & Business Media.","DOI":"10.1007\/978-3-540-68791-7"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1080\/0025570X.1975.11976447","article-title":"Generalizations of the logarithmic mean","volume":"48","author":"Stolarsky","year":"1975","journal-title":"Math. Mag."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/13\/4542\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:25:00Z","timestamp":1760163900000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/13\/4542"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,2]]},"references-count":52,"journal-issue":{"issue":"13","published-online":{"date-parts":[[2021,7]]}},"alternative-id":["s21134542"],"URL":"https:\/\/doi.org\/10.3390\/s21134542","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,2]]}}}