{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,15]],"date-time":"2026-03-15T05:05:25Z","timestamp":1773551125801,"version":"3.50.1"},"reference-count":40,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,8,17]],"date-time":"2023-08-17T00:00:00Z","timestamp":1692230400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Neurosci."],"abstract":"<jats:sec><jats:title>Introduction<\/jats:title><jats:p>As deep learning has achieved state-of-the-art performance for many tasks of EEG-based BCI, many efforts have been made in recent years trying to understand what have been learned by the models. This is commonly done by generating a heatmap indicating to which extent each pixel of the input contributes to the final classification for a trained model. Despite the wide use, it is not yet understood to which extent the obtained interpretation results can be trusted and how accurate they can reflect the model decisions.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We conduct studies to quantitatively evaluate seven different deep interpretation techniques across different models and datasets for EEG-based BCI.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The results reveal the importance of selecting a proper interpretation technique as the initial step. In addition, we also find that the quality of the interpretation results is inconsistent for individual samples despite when a method with an overall good performance is used. Many factors, including model structure and dataset types, could potentially affect the quality of the interpretation results.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion<\/jats:title><jats:p>Based on the observations, we propose a set of procedures that allow the interpretation results to be presented in an understandable and trusted way. We illustrate the usefulness of our method for EEG-based BCI with instances selected from different scenarios.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fncom.2023.1232925","type":"journal-article","created":{"date-parts":[[2023,8,18]],"date-time":"2023-08-18T07:40:55Z","timestamp":1692344455000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":19,"title":["Towards best practice of interpreting deep learning models for EEG-based brain computer interfaces"],"prefix":"10.3389","volume":"17","author":[{"given":"Jian","family":"Cui","sequence":"first","affiliation":[]},{"given":"Liqiang","family":"Yuan","sequence":"additional","affiliation":[]},{"given":"Zhaoxiang","family":"Wang","sequence":"additional","affiliation":[]},{"given":"Ruilin","family":"Li","sequence":"additional","affiliation":[]},{"given":"Tianzi","family":"Jiang","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2023,8,17]]},"reference":[{"key":"B1","article-title":"Towards better understanding of gradient-based attribution methods for deep neural networks.","author":"Ancona","year":"2017","journal-title":"arXiv"},{"key":"B2","year":"2015"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0130140","article-title":"On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.","volume":"10","author":"Bach","year":"2015","journal-title":"PLoS One"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2020.3048385","article-title":"Spatio-spectral feature representation for motor imagery classification using convolutional neural networks","author":"Bang","year":"2021","journal-title":"Proceedings of the IEEE transactions on neural networks and learning systems"},{"key":"B5","year":"2008"},{"key":"B6","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1016\/j.neunet.2020.05.032","article-title":"Interpretable and lightweight convolutional neural network for EEG decoding: Application to movement execution and imagination.","volume":"129","author":"Borra","year":"2020","journal-title":"Neural Netw."},{"key":"B7","author":"Britton","year":"2016","journal-title":"Electroencephalography (EEG): An introductory text and atlas of normal and abnormal findings in adults, children, and infants."},{"key":"B8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41597-019-0027-4","article-title":"Multi-channel EEG recordings during a sustained-attention driving task.","volume":"6","author":"Cao","year":"2019","journal-title":"Sci. Data"},{"key":"B9","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1016\/j.ymeth.2021.04.017","article-title":"A compact and interpretable convolutional neural network for cross-subject driver drowsiness detection from single-channel EEG.","volume":"202","author":"Cui","year":"","journal-title":"Methods"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2022.3147208","article-title":"EEG-based cross-subject driver drowsiness recognition with an interpretable convolutional neural network","author":"Cui","year":"2022","journal-title":"Proceedings of the IEEE transactions on neural networks and learning systems"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1109\/CW52790.2021.00041","article-title":"Subject-independent drowsiness recognition from single-channel EEG with an interpretable CNN-LSTM model","author":"Cui","year":"","journal-title":"Proceedings of the 2021 International Conference on Cyberworlds (CW)"},{"key":"B12","year":"2022","journal-title":"Towards-best-practice-of-interpreting-deep-learning-models-for-EEG-based-BCI."},{"key":"B13","doi-asserted-by":"publisher","first-page":"281","DOI":"10.1016\/S0079-6123(08)60995-4","article-title":"Models of neuronal populations: The basic mechanism of rhythmicity","author":"Da Silva","year":"1976","journal-title":"Perspectives of brain research, progress in brain research"},{"key":"B14","doi-asserted-by":"publisher","first-page":"5","DOI":"10.2340\/16501977-0020","article-title":"Motor imagery and stroke rehabilitation: A critical discussion.","volume":"39","author":"De Vries","year":"2007","journal-title":"J. Rehabil. Med."},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/aaf3f6","article-title":"Inter-subject transfer learning with an end-to-end deep convolutional neural network for EEG-based BCI.","volume":"16","author":"Fahimi","year":"2019","journal-title":"J. Neural Eng."},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.371","article-title":"Interpretable explanations of black boxes by meaningful perturbation","author":"Fong","year":"2017","journal-title":"Proceedings of the IEEE international conference on computer vision"},{"key":"B17","article-title":"A benchmark for interpretability methods in deep neural networks.","author":"Hooker","year":"2018","journal-title":"arXiv"},{"key":"B18","article-title":"Benchmarking deep learning interpretability in time series predictions.","author":"Ismail","year":"2020","journal-title":"arXiv"},{"key":"B19","article-title":"Adam: A method for stochastic optimization.","author":"Kingma","year":"2014","journal-title":"arXiv"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/aace8c","article-title":"EEGNet: A compact convolutional neural network for EEG-based brain\u2013computer interfaces.","volume":"15","author":"Lawhern","year":"2018","journal-title":"J. Neural Eng."},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.3389\/fnhum.2017.00134","article-title":"Cortical mechanisms of tongue sensorimotor functions in humans: A review of the magnetoencephalography approach.","volume":"11","author":"Maezawa","year":"2017","journal-title":"Front. Hum. Neurosci."},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1155\/2012\/578295","article-title":"Objective and subjective evaluation of online error correction during P300-based spelling.","volume":"2012","author":"Margaux","year":"2012","journal-title":"Adv. Hum. Comput. Interact."},{"key":"B23","doi-asserted-by":"publisher","first-page":"788","DOI":"10.1162\/jocn.1997.9.6.788","article-title":"Event-related brain potentials following incorrect feedback in a time-estimation task: Evidence for a \u201cgeneric\u201d neural system for error detection.","volume":"9","author":"Miltner","year":"1997","journal-title":"J. Cogn. Neurosci."},{"key":"B24","doi-asserted-by":"publisher","first-page":"2116","DOI":"10.1109\/TIP.2018.2881920","article-title":"Cnn fixations: An unraveling approach to visualize the discriminative image regions.","volume":"28","author":"Mopuri","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"B25","doi-asserted-by":"publisher","first-page":"27074","DOI":"10.1109\/ACCESS.2020.2971600","article-title":"Learning invariant representations from EEG via adversarial inference.","volume":"8","author":"\u00d6zdenizci","year":"2020","journal-title":"IEEE Access"},{"key":"B26","article-title":"Rise: Randomized input sampling for explanation of black-box models.","author":"Petsiuk","year":"2018","journal-title":"arXiv"},{"key":"B27","doi-asserted-by":"publisher","first-page":"1842","DOI":"10.1016\/s1388-2457(99)00141-8","article-title":"Event-related EEG\/MEG synchronization and desynchronization: Basic principles.","volume":"110","author":"Pfurtscheller","year":"1999","journal-title":"Clin. Neurophysiol."},{"key":"B28","article-title":"Not just a black box: Learning important features through propagating activation differences.","author":"Shrikumar","year":"2016","journal-title":"arXiv"},{"key":"B29","doi-asserted-by":"publisher","first-page":"1168","DOI":"10.1016\/j.clinph.2010.10.044","article-title":"EEG alpha spindle measures as indicators of driver fatigue under real traffic conditions.","volume":"122","author":"Simon","year":"2011","journal-title":"Clin. Neurophysiol."},{"key":"B30","article-title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","author":"Simonyan","year":"2014","journal-title":"Proceedings of the international conference on learning representations"},{"key":"B31","article-title":"Striving for simplicity: The all convolutional net.","author":"Springenberg","year":"2014","journal-title":"arXiv"},{"key":"B32","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1016\/j.jneumeth.2016.10.008","article-title":"Interpretable deep neural networks for single-trial EEG classification.","volume":"274","author":"Sturm","year":"2016","journal-title":"J. Neurosci. Methods"},{"key":"B33","article-title":"Axiomatic attribution for deep networks","author":"Sundararajan","year":"2017","journal-title":"Proceedings of the international 34th conference on machine learning"},{"key":"B34","article-title":"Quantifying explainability of saliency methods in deep neural networks.","author":"Tjoa","year":"2020","journal-title":"arXiv"},{"key":"B35","doi-asserted-by":"publisher","first-page":"400","DOI":"10.1109\/TNSRE.2018.2790359","article-title":"Toward drowsiness detection using non-hair-bearing EEG-based brain-computer interfaces.","volume":"26","author":"Wei","year":"2018","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2020.3028783","article-title":"Interpreting image classifiers by generating discrete masks","author":"Yuan","year":"2020","journal-title":"Proceedings of the IEEE transactions on pattern analysis and machine intelligence"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-10590-1_53","article-title":"Visualizing and understanding convolutional networks","author":"Zeiler","year":"2014","journal-title":"Proceedings of the European conference on computer vision"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.319","article-title":"Learning deep features for discriminative localization","author":"Zhou","year":"2016","journal-title":"Proceedings of the IEEE conference on computer vision and pattern recognition"},{"key":"B39","article-title":"Visualizing deep neural network decisions: Prediction difference analysis.","author":"Zintgraf","year":"2017","journal-title":"arXiv"},{"key":"B40","doi-asserted-by":"publisher","first-page":"1084","DOI":"10.1016\/j.neunet.2020.01.031","article-title":"Top-down neural attention by excitation backprop.","volume":"126","author":"Zou","year":"2018","journal-title":"Int. J. Comput. Vis."}],"container-title":["Frontiers in Computational Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2023.1232925\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,18]],"date-time":"2023-08-18T07:41:01Z","timestamp":1692344461000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2023.1232925\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,17]]},"references-count":40,"alternative-id":["10.3389\/fncom.2023.1232925"],"URL":"https:\/\/doi.org\/10.3389\/fncom.2023.1232925","relation":{},"ISSN":["1662-5188"],"issn-type":[{"value":"1662-5188","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,17]]},"article-number":"1232925"}}