{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:00:41Z","timestamp":1760148041667,"version":"build-2065373602"},"reference-count":47,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2023,3,23]],"date-time":"2023-03-23T00:00:00Z","timestamp":1679529600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>One of the emerging fields in functional magnetic resonance imaging (fMRI) is the decoding of different stimulations. The underlying idea is to reveal the hidden representative signal patterns of various fMRI tasks for achieving high task-classification performance. Unfortunately, when multiple tasks are processed, performance remains limited due to several challenges, which are rarely addressed since the majority of the state-of-the-art studies cover a single neuronal activity task. Accordingly, the first contribution of this study is the collection and release of a rigorously acquired dataset, which contains cognitive, behavioral, and affective fMRI tasks together with resting state. After a comprehensive analysis of the pitfalls of existing systems on this new dataset, we propose an automatic multitask classification (MTC) strategy using a feature fusion module (FFM). FFM aims to create a unique signature for each task by combining deep features with time-frequency representations. We show that FFM creates a feature space that is superior for representing task characteristics compared to their individual use. Finally, for MTC, we test a diverse set of deep-models and analyze their complementarity. Our results reveal higher classification accuracy compared to benchmarks. Both the dataset and the code are accessible to researchers for further developments.<\/jats:p>","DOI":"10.3390\/s23073382","type":"journal-article","created":{"date-parts":[[2023,3,24]],"date-time":"2023-03-24T03:16:46Z","timestamp":1679627806000},"page":"3382","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Diverse Task Classification from Activation Patterns of Functional Neuro-Images Using Feature Fusion Module"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2326-9438","authenticated-orcid":false,"given":"Osman Tayfun","family":"Bi\u015fkin","sequence":"first","affiliation":[{"name":"Department of Electrical and Electronics Engineering, Burdur Mehmet Akif Ersoy University, Burdur 15030, Turkey"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9850-137X","authenticated-orcid":false,"given":"Cemre","family":"Candemir","sequence":"additional","affiliation":[{"name":"International Computer Institute, Ege University, Izmir 35100, Turkey"},{"name":"Standardization of Computational Anatomy Techniques, SoCAT Lab, Ege University, Izmir 35100, Turkey"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3522-1359","authenticated-orcid":false,"given":"Ali Saffet","family":"Gonul","sequence":"additional","affiliation":[{"name":"Standardization of Computational Anatomy Techniques, SoCAT Lab, Ege University, Izmir 35100, Turkey"},{"name":"Department of Psychiatry, Medical Faculty, Ege University, Izmir 35100, Turkey"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8445-0388","authenticated-orcid":false,"given":"Mustafa Alper","family":"Selver","sequence":"additional","affiliation":[{"name":"Department of Electrical and Electronics Engineering and Izmir Health Technologies Development and Accelerator (BioIzmir), Dokuz Eylul University, Izmir 35160, Turkey"}]}],"member":"1968","published-online":{"date-parts":[[2023,3,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"9868","DOI":"10.1073\/pnas.87.24.9868","article-title":"Brain magnetic resonance imaging with contrast dependent on blood oxygenation","volume":"87","author":"Ogawa","year":"1990","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"28","DOI":"10.3389\/fnhum.2011.00028","article-title":"Statistical Analysis of fMRI Time-Series: A Critical Review of the GLM Approach","volume":"5","author":"Monti","year":"2011","journal-title":"Front. Hum. Neurosci."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"332","DOI":"10.1523\/JNEUROSCI.1302-13.2014","article-title":"Brain-Based Translation: fMRI Decoding of Spoken Words in Bilinguals Reveals Language-Independent Semantic Representations in Anterior Temporal Lobe","volume":"34","author":"Correia","year":"2014","journal-title":"J. Neurosci."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"309","DOI":"10.1007\/s12021-018-9358-0","article-title":"Decoding Auditory Saliency from Brain Activity Patterns during Free Listening to Naturalistic Audio Excerpts","volume":"16","author":"Zhao","year":"2018","journal-title":"Neuroinformatics"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1002\/hbm.23015","article-title":"Motor imagery of hand actions: Decoding the content of motor imagery from brain activity in frontal and parietal motor areas","volume":"37","author":"Pilgramm","year":"2015","journal-title":"Hum. Brain Mapp."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Shen, G., Horikawa, T., Majima, K., and Kamitani, Y. (2019). Deep image reconstruction from human brain activity. PLoS Comput. Biol., 15.","DOI":"10.1371\/journal.pcbi.1006633"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"915","DOI":"10.1016\/j.neuron.2008.11.004","article-title":"Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders","volume":"60","author":"Miyawaki","year":"2008","journal-title":"Neuron"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2425","DOI":"10.1126\/science.1063736","article-title":"Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex","volume":"293","author":"Haxby","year":"2001","journal-title":"Science"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1038\/nature06713","article-title":"Identifying natural images from human brain activity","volume":"452","author":"Kay","year":"2008","journal-title":"Nature"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"323","DOI":"10.1016\/j.cub.2006.11.072","article-title":"Reading Hidden Intentions in the Human Brain","volume":"17","author":"Haynes","year":"2007","journal-title":"Curr. Biol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.neuroimage.2014.03.018","article-title":"Neural portraits of perception: Reconstructing face images from evoked brain activity","volume":"94","author":"Cowen","year":"2014","journal-title":"NeuroImage"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"508","DOI":"10.1109\/TAFFC.2019.2952113","article-title":"Multi-Label Multi-Task Deep Learning for Behavioral Coding","volume":"13","author":"Gibson","year":"2022","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1109\/TAFFC.2019.2946540","article-title":"Facial Expression Recognition Using a Temporal Ensemble of Multi-Level Convolutional Neural Networks","volume":"13","author":"Nguyen","year":"2022","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"285","DOI":"10.1109\/TAFFC.2019.2926724","article-title":"All-in-One: Emotion, Sentiment and Intensity Prediction Using a Multi-Task Ensemble Framework","volume":"13","author":"Akhtar","year":"2022","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"43222","DOI":"10.1109\/ACCESS.2019.2907040","article-title":"Decoding Behavior Tasks from Brain Activity Using Deep Transfer Learning","volume":"7","author":"Gao","year":"2019","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1677","DOI":"10.1109\/JBHI.2019.2940695","article-title":"Decoding Brain States from fMRI Signals by Using Unsupervised Domain Adaptation","volume":"24","author":"Gao","year":"2020","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"616","DOI":"10.1016\/j.neuroimage.2010.05.081","article-title":"Decoding brain states from fMRI connectivity graphs","volume":"56","author":"Richiardi","year":"2011","journal-title":"NeuroImage"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1016\/j.neuroimage.2017.08.005","article-title":"Deconstructing multivariate decoding for the study of brain function","volume":"180","author":"Hebart","year":"2018","journal-title":"NeuroImage"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1016\/j.neuron.2015.05.025","article-title":"A Primer on Pattern-Based Approaches to fMRI: Principles, Pitfalls, and Perspectives","volume":"87","author":"Haynes","year":"2015","journal-title":"Neuron"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"813","DOI":"10.1016\/S1053-8119(03)00002-8","article-title":"Comparison of fMRI activation at 3 and 1.5T during perceptual, cognitive, and affective processing","volume":"18","author":"Krasnow","year":"2003","journal-title":"NeuroImage"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"54","DOI":"10.3389\/fninf.2018.00054","article-title":"Intra- and Inter-Scanner Reliability of Voxel-Wise Whole-Brain Analytic Metrics for Resting State fMRI","volume":"12","author":"Zhao","year":"2018","journal-title":"Front. Neuroinform."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1016\/j.neuroimage.2014.11.028","article-title":"Functional connectivity in BOLD and CBF data: Similarity and reliability of resting brain networks","volume":"106","author":"Jann","year":"2015","journal-title":"NeuroImage"},{"key":"ref_23","unstructured":"(2022, August 24). Working Memory in Healthy and Schizophrenic Individuals. Available online: https:\/\/openfmri.org\/dataset\/ds000115\/."},{"key":"ref_24","unstructured":"(2022, August 24). Prefrontal-Subcortical Pathways Mediating Successful Emotion Regulation. Available online: https:\/\/openfmri.org\/dataset\/ds000108\/."},{"key":"ref_25","unstructured":"(2022, August 24). Affective Videos. Available online: https:\/\/openfmri.org\/dataset\/ds000205\/."},{"key":"ref_26","unstructured":"(2022, August 24). Neural Processing of Emotional Musical and Nonmusical Stimuli in Depression. Available online: https:\/\/openfmri.org\/dataset\/ds000171\/."},{"key":"ref_27","unstructured":"(2022, August 24). EUPD Cyberball. Available online: https:\/\/openfmri.org\/dataset\/ds000214\/."},{"key":"ref_28","unstructured":"(2022, August 24). Incidental Encoding Task (Posner Cueing Paradigm). Available online: https:\/\/openfmri.org\/dataset\/ds000110\/."},{"key":"ref_29","unstructured":"(2022, August 24). Visual Imagery and False Memory for Pictures. Available online: https:\/\/openfmri.org\/dataset\/ds000203\/."},{"key":"ref_30","unstructured":"(2022, August 24). Block Tapping Task, Cognitive Atlas. Available online: http:\/\/www.cognitiveatlas.org\/task\/id\/tsk_4a57abb9498df\/."},{"key":"ref_31","unstructured":"(2022, August 24). Learning and Memory: Motor Skill Consolidation and Intermanual Transfer. Available online: https:\/\/openfmri.org\/dataset\/ds000170\/."},{"key":"ref_32","unstructured":"(2022, August 24). Goal-Directed Motor Task\u2014OpenNeuro. Available online: https:\/\/openneuro.org\/datasets\/ds004056\/versions\/1.0.2."},{"key":"ref_33","unstructured":"(2022, August 24). Visual and Audiovisual Speech Perception Associated with Increased Functional Connectivity between Sensory and Motor Regions\u2014OpenNeuro. Available online: https:\/\/openneuro.org\/datasets\/ds003717\/versions\/1.0.1."},{"key":"ref_34","unstructured":"(2022, August 24). A Multi-Modal Human Neuroimaging Dataset for Data Integration: Simultaneous EEG and fMRI Acquisition during a Motor Imagery Neurofeedback Task: XP2\u2014OpenNeuro. Available online: https:\/\/openneuro.org\/datasets\/ds002338\/versions\/2.0.2."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"70","DOI":"10.12688\/f1000research.29988.1","article-title":"rt-me-fMRI: A task and resting state dataset for real-time, multi-echo fMRI methods development and validation","volume":"10","author":"Heunis","year":"2021","journal-title":"F1000Research"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"706","DOI":"10.1109\/TAFFC.2021.3059965","article-title":"Automatic Detection of Emotional Changes Induced by Social Support Loss using fMRI","volume":"14","author":"Candemir","year":"2021","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1007\/s12021-013-9204-3","article-title":"A review of feature reduction techniques in neuroimaging","volume":"12","author":"Mwangi","year":"2014","journal-title":"Neuroinformatics"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"2033","DOI":"10.1016\/j.patcog.2011.06.001","article-title":"Brain decoding: Opportunities and challenges for pattern recognition","volume":"45","author":"Lee","year":"2012","journal-title":"Pattern Recognit."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"674","DOI":"10.1109\/34.192463","article-title":"A theory for multiresolution signal decomposition: The wavelet representation","volume":"11","author":"Mallat","year":"1989","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_40","unstructured":"(2021, November 30). ImageNet. Available online: https:\/\/image-net.org\/index.php."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Zhao, Z., Anand, R., and Wang, M. (2019). Maximum Relevance and Minimum Redundancy Feature Selection Methods for a Marketing Machine Learning Platform. arXiv.","DOI":"10.1109\/DSAA.2019.00059"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1142\/S0219720005001004","article-title":"Minimum Redundancy Feature Selection from Microarray Gene Expression Data","volume":"3","author":"Ding","year":"2005","journal-title":"J. Bioinform. Comput. Biol."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Gulgezen, G., Cataltepe, Z., and Yu, L. (2009, January 9\u201311). Stable feature selection using MRMR algorithm. Proceedings of the 2009 IEEE 17th Signal Processing and Communications Applications Conference, Antalya, Turkey.","DOI":"10.1109\/SIU.2009.5136466"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"430","DOI":"10.3389\/fnhum.2015.00430","article-title":"On the existence of a generalized non-specific task-dependent network","volume":"9","author":"Hugdahl","year":"2015","journal-title":"Front. Hum. Neurosci."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"21","DOI":"10.3389\/fpsyt.2018.00021","article-title":"Psychopathology Assessment Methods Revisited: On Translational Cross-Validation of Clinical Self-Evaluation Scale and fMRI","volume":"9","author":"Stoyanov","year":"2018","journal-title":"Front. Psychiatry"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1016\/j.tics.2022.12.015","article-title":"Improving the study of brain-behavior relationships by revisiting basic assumptions","volume":"27","author":"Westlin","year":"2023","journal-title":"Trends Cogn. Sci."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/7\/3382\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:01:25Z","timestamp":1760122885000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/7\/3382"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,23]]},"references-count":47,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2023,4]]}},"alternative-id":["s23073382"],"URL":"https:\/\/doi.org\/10.3390\/s23073382","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2023,3,23]]}}}