{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,5]],"date-time":"2025-11-05T06:50:50Z","timestamp":1762325450896,"version":"build-2065373602"},"reference-count":59,"publisher":"MDPI AG","issue":"20","license":[{"start":{"date-parts":[[2022,10,14]],"date-time":"2022-10-14T00:00:00Z","timestamp":1665705600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Ministry of Science and ICT","award":["2017M3C7A1029485"],"award-info":[{"award-number":["2017M3C7A1029485"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Restless legs syndrome (RLS) is a sensorimotor disorder accompanied by a strong urge to move the legs and an unpleasant sensation in the legs, and is known to accompany prefrontal dysfunction. Here, we aimed to clarify the neural mechanism of working memory deficits associated with RLS using machine-learning-based analysis of single-trial neural activities. A convolutional neural network classifier was developed to discriminate the cortical activities between RLS patients and normal controls. A layer-wise relevance propagation was applied to the trained classifier in order to determine the critical nodes in the input layer for the output decision, i.e., the time\/location of cortical activities discriminating RLS patients and normal controls during working memory tasks. Our method provided high classification accuracy (~94%) from single-trial event-related potentials, which are known to suffer from high inter-trial\/inter-subject variation and low signal-to-noise ratio, after strict separation of training\/test\/validation data according to leave-one-subject-out cross-validation. The determined critical areas overlapped with the cortical substrates of working memory, and the neural activities in these areas were correlated with some significant clinical scores of RLS.<\/jats:p>","DOI":"10.3390\/s22207792","type":"journal-article","created":{"date-parts":[[2022,10,17]],"date-time":"2022-10-17T03:43:58Z","timestamp":1665978238000},"page":"7792","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Explainable Machine-Learning-Based Characterization of Abnormal Cortical Activities for Working Memory of Restless Legs Syndrome Patients"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8011-4037","authenticated-orcid":false,"given":"Minju","family":"Kim","sequence":"first","affiliation":[{"name":"Department of Biomedical Engineering, College of Health Science, Yonsei University, 1, Yeonsedae-gil, Heungeop-myeon, Wonju-si 26493, Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7265-8640","authenticated-orcid":false,"given":"Hyun","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, College of Health Science, Yonsei University, 1, Yeonsedae-gil, Heungeop-myeon, Wonju-si 26493, Korea"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5098-7446","authenticated-orcid":false,"given":"Pukyeong","family":"Seo","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, College of Health Science, Yonsei University, 1, Yeonsedae-gil, Heungeop-myeon, Wonju-si 26493, Korea"}]},{"given":"Ki-Young","family":"Jung","sequence":"additional","affiliation":[{"name":"Department of Neurology, Seoul National University Hospital, Seoul National University College of Medicine, 101, Daehak-ro, Jongno-gu, Seoul 03080, Korea"}]},{"given":"Kyung Hwan","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Biomedical Engineering, College of Health Science, Yonsei University, 1, Yeonsedae-gil, Heungeop-myeon, Wonju-si 26493, Korea"}]}],"member":"1968","published-online":{"date-parts":[[2022,10,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"860","DOI":"10.1016\/j.sleep.2014.03.025","article-title":"Restless legs syndrome\/Willis\u2013Ekbom disease diagnostic criteria: Updated International Restless Legs Syndrome Study Group (IRLSSG) consensus criteria\u2014History, rationale, description, and significance","volume":"15","author":"Allen","year":"2014","journal-title":"Sleep Med."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"772","DOI":"10.1093\/sleep\/32.6.772","article-title":"Polysomnographic and Health-related Quality of Life Correlates of Restless Legs Syndrome in the Sleep Heart Health Study","volume":"32","author":"Winkelman","year":"2009","journal-title":"Sleep"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1016\/j.sleep.2005.05.006","article-title":"Cognitive deficits associated with restless legs syndrome (RLS)","volume":"7","author":"Pearson","year":"2006","journal-title":"Sleep Med."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2641","DOI":"10.1002\/mds.23353","article-title":"Short-term attention and verbal fluency is decreased in restless legs syndrome patients","volume":"25","author":"Fulda","year":"2010","journal-title":"Mov. Disord."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"808","DOI":"10.1016\/j.sleep.2014.03.010","article-title":"Working memory deficit in patients with restless legs syndrome: An event-related potential study","volume":"15","author":"Kim","year":"2014","journal-title":"Sleep Med."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e13287","DOI":"10.1111\/jsr.13287","article-title":"Working memory deficits in patients with idiopathic restless legs syndrome are associated with abnormal theta-band neural synchrony","volume":"30","author":"Cha","year":"2021","journal-title":"J. Sleep Res."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1016\/S0926-6410(01)00009-X","article-title":"Neurophysiological signals of working memory in normal aging","volume":"11","author":"McEvoy","year":"2001","journal-title":"Cogn. Brain Res."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1093\/sleep\/28.1.55","article-title":"Decreased Cortical Response to Verbal Working Memory Following Sleep Deprivation","volume":"28","author":"Mu","year":"2005","journal-title":"Sleep"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"401","DOI":"10.5334\/tohm.322","article-title":"Restless Legs Syndrome: Current Concepts about Disease Pathophysiology","volume":"6","author":"Koo","year":"2016","journal-title":"Tremor Other Hyperkinetic Mov."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1016\/j.sleep.2016.07.018","article-title":"Brain imaging and networks in restless legs syndrome","volume":"31","author":"Rizzo","year":"2016","journal-title":"Sleep Med."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1751","DOI":"10.1002\/mds.21608","article-title":"Cortical grey matter alterations in idiopathic restless legs syndrome: An optimized voxel-based morphometry study","volume":"22","author":"Unrath","year":"2007","journal-title":"Mov. Disord."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"304","DOI":"10.1016\/j.ijpsycho.2014.12.005","article-title":"Neurocognitive function in patients with idiopathic Restless Legs Syndrome before and after treatment with dopamine-agonist","volume":"95","author":"Galbiati","year":"2014","journal-title":"Int. J. Psychophysiol."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/S0301-0082(02)00011-4","article-title":"Dopamine and the regulation of cognition and attention","volume":"67","author":"Nieoullon","year":"2002","journal-title":"Prog. Neurobiol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1016\/j.jneumeth.2017.07.015","article-title":"Strategies for statistical thresholding of source localization maps in magnetoencephalography and estimating source extent","volume":"290","author":"Maksymenko","year":"2017","journal-title":"J. Neurosci. Methods"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"5391","DOI":"10.1002\/hbm.23730","article-title":"Deep learning with convolutional neural networks for EEG decoding and visualization","volume":"38","author":"Schirrmeister","year":"2017","journal-title":"Hum. Brain Mapp."},{"key":"ref_16","unstructured":"Mayor-Torres, J.M., Medina-DeVilliers, S., Clarkson, T., Lerner, M.D., and Riccardi, G. (2021). Evaluation of Interpretability for Deep Learning Algorithms in EEG Emotion Recognition: A Case Study in Autism. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"031001","DOI":"10.1088\/1741-2552\/ab0ab5","article-title":"Deep learning for electroencephalogram (EEG) classification tasks: A review","volume":"16","author":"Craik","year":"2019","journal-title":"J. Neural Eng."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1016\/j.jneumeth.2016.10.008","article-title":"Interpretable deep neural networks for single-trial EEG classification","volume":"274","author":"Sturm","year":"2016","journal-title":"J. Neurosci. Methods"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"251","DOI":"10.3389\/fnins.2020.00251","article-title":"Deep Learning Convolutional Neural Networks Discriminate Adult ADHD from Healthy Individuals on the Basis of Event-Related Spectral EEG","volume":"14","author":"Ruffini","year":"2020","journal-title":"Front. Neurosci."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"806","DOI":"10.3389\/fneur.2019.00806","article-title":"Deep Learning with EEG Spectrograms in Rapid Eye Movement Behavior Disorder","volume":"10","author":"Ruffini","year":"2019","journal-title":"Front. Neurol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1038\/s42003-020-0846-z","article-title":"Applying deep learning to single-trial EEG data provides evidence for complementary theories on action control","volume":"3","author":"Vahid","year":"2020","journal-title":"Commun. Biol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1016\/S1389-9457(02)00258-7","article-title":"Validation of the International Restless Legs Syndrome Study Group rating scale for restless legs syndrome","volume":"4","author":"Horiguchi","year":"2003","journal-title":"Sleep Med."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1016\/0165-1781(89)90047-4","article-title":"Pittsburgh Sleep Quality Index (PSQI): A New Instrument for Psychiatric Research and Practice","volume":"28","author":"Buysse","year":"1989","journal-title":"Psychiatry Res."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"540","DOI":"10.1093\/sleep\/14.6.540","article-title":"A New Method for Measuring Daytime Sleepiness: The Epworth Sleepiness Scale","volume":"14","author":"Johns","year":"1991","journal-title":"Sleep"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1016\/S1389-9457(00)00065-4","article-title":"Validation of the Insomnia Severity Index as an outcome measure for insomnia research","volume":"2","author":"Bastien","year":"2001","journal-title":"Sleep Med."},{"key":"ref_26","first-page":"126","article-title":"Validation and Factor Structure of Korean Version of the Beck Depression Inventory Second Edition (BDI-II): In a University Student Sample","volume":"18","author":"Yu","year":"2011","journal-title":"Korean J. Biol. Psychiatry"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12955-017-0759-9","article-title":"Is the Hospital Anxiety and Depression Scale (HADS) a valid measure in a general population 65\u201380 years old? A psychometric evaluation study","volume":"15","author":"Djukanovic","year":"2017","journal-title":"Health Qual. Life Outcomes"},{"key":"ref_28","unstructured":"Simonyan, K., and Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Bach, S., Binder, A., Montavon, G., Klauschen, F., M\u00fcller, K.-R., and Samek, W. (2015). On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0130140"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Kang, X., Herron, T.J., Cate, A.D., Yund, E.W., and Woods, D.L. (2012). Hemispherically-Unified Surface Maps of Human Cerebral Cortex: Reliability and Hemispheric Asymmetries. PLoS ONE, 7.","DOI":"10.1371\/journal.pone.0045582"},{"key":"ref_31","first-page":"5","article-title":"Standardized Low-Resolution Brain Electromagnetic Tomography (SLORETA): Technical Details","volume":"24","year":"2002","journal-title":"Methods Find. Exp. Clin. Pharmacol."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2011\/879716","article-title":"Brainstorm: A User-Friendly Application for MEG\/EEG Analysis","volume":"2011","author":"Tadel","year":"2011","journal-title":"Comput. Intell. Neurosci."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"732","DOI":"10.1016\/j.clinph.2003.11.021","article-title":"A review of the evidence for P2 being an independent component process: Age, sleep and modality","volume":"115","author":"Colrain","year":"2004","journal-title":"Clin. Neurophysiol."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1037\/a0033753","article-title":"Components of working memory and visual selective attention","volume":"40","author":"Burnham","year":"2014","journal-title":"J. Exp. Psychol. Hum. Percept. Perform."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"241","DOI":"10.2307\/1784714","article-title":"A New Equal-Area Projection for World Maps","volume":"73","author":"Boggs","year":"1929","journal-title":"Geogr. J."},{"key":"ref_36","unstructured":"O\u2019Shea, K., and Nash, R. (2015). An Introduction to Convolutional Neural Networks. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Albawi, S., Mohammed, T.A., and Al-Zawi, S. (2017, January 21\u201323). Understanding of a Convolutional Neural Network. Proceedings of the 2017 International Conference on Engineering and Technology (ICET), Antalya, Turkey.","DOI":"10.1109\/ICEngTechnol.2017.8308186"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"056013","DOI":"10.1088\/1741-2552\/aace8c","article-title":"EEGNet: A compact convolutional neural network for EEG-based brain\u2013computer interfaces","volume":"15","author":"Lawhern","year":"2018","journal-title":"J. Neural Eng."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"60141","DOI":"10.1109\/ACCESS.2022.3176367","article-title":"An Interpretable Deep Learning Classifier for Epileptic Seizure Prediction Using EEG Data","volume":"10","author":"Jemal","year":"2022","journal-title":"IEEE Access"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Vilamala, A., Madsen, K.H., and Hansen, L.K. (2017, January 25\u201328). Deep Convolutional Neural Networks for Interpretable Analysis of EEG Sleep Stage Scoring. Proceedings of the 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP), Tokyo, Japan.","DOI":"10.1109\/MLSP.2017.8168133"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1515\/cdbme-2021-2038","article-title":"Balanced Leave-One-Subject-Out Cross- Validation for Microsleep Classification","volume":"7","author":"Pauli","year":"2021","journal-title":"Curr. Dir. Biomed. Eng."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"761","DOI":"10.1016\/S0893-6080(98)00010-0","article-title":"Automatic early stopping using cross validation: Quantifying the criteria","volume":"11","author":"Prechelt","year":"1998","journal-title":"Neural Netw."},{"key":"ref_43","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_44","first-page":"281","article-title":"Random Search for Hyper-Parameter Optimization","volume":"13","author":"Bergstra","year":"2012","journal-title":"J. Mach. Learn. Res."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"193","DOI":"10.1007\/978-3-030-28954-6_10","article-title":"Layer-Wise Relevance Propagation: An Overview","volume":"1","author":"Montavon","year":"2019","journal-title":"Explain. AI Interpret. Explain. Vis. Deep Learn."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1016\/j.tics.2011.11.014","article-title":"Top-down modulation: Bridging selective attention and working memory","volume":"16","author":"Gazzaley","year":"2012","journal-title":"Trends Cogn. Sci."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"i125","DOI":"10.1093\/cercor\/bhm113","article-title":"Functional Interactions between Prefrontal and Visual Association Cortex Contribute to Top-Down Modulation of Visual Processing","volume":"17","author":"Gazzaley","year":"2007","journal-title":"Cereb. Cortex"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"eabe8212","DOI":"10.1126\/sciadv.abe8212","article-title":"Output planning at the input stage in visual working memory","volume":"7","author":"Boettcher","year":"2021","journal-title":"Sci. Adv."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"57","DOI":"10.3389\/fnhum.2011.00057","article-title":"Effects of Working Memory Load on Visual Selective Attention: Behavioral and Electrophysiological Evidence","volume":"5","author":"Pratt","year":"2011","journal-title":"Front. Hum. Neurosci."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"1377","DOI":"10.1126\/science.1962197","article-title":"A Neural Mechanism for Working and Recognition Memory in Inferior Temporal Cortex","volume":"254","author":"Miller","year":"1991","journal-title":"Science"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1146\/annurev.psych.49.1.289","article-title":"The Cogntive Neuroscience of Constructive Memory","volume":"49","author":"Schacter","year":"1998","journal-title":"Annu. Rev. Psychol."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"268","DOI":"10.1002\/ana.10812","article-title":"Deterioration of naming nouns versus verbs in primary progressive aphasia","volume":"55","author":"Hillis","year":"2004","journal-title":"Ann. Neurol."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1080\/02643290244000248","article-title":"Neural correlates of conceptual knowledge for actions","volume":"20","author":"Tranel","year":"2003","journal-title":"Cogn. Neuropsychol."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1136\/jnnp.67.4.532","article-title":"Verbal memory impairment after left insular cortex infarction","volume":"67","author":"Manes","year":"1999","journal-title":"J. Neurol. Neurosurg. Psychiatry"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1177\/155005940803900217","article-title":"Semantic Activation and Verbal Working Memory Maintenance in Schizophrenic Thought Disorder: Insights from Electrophysiology and Lexical Amibiguity","volume":"39","author":"Salisbury","year":"2008","journal-title":"Clin. EEG Neurosci."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"005","DOI":"10.1055\/s-2008-1061621","article-title":"Semantic Memory and Language Processing: A Primer","volume":"29","author":"Antonucci","year":"2008","journal-title":"Semin. Speech Lang."},{"key":"ref_57","doi-asserted-by":"crossref","first-page":"10607","DOI":"10.1523\/JNEUROSCI.5578-12.2013","article-title":"Tired and Apprehensive: Anxiety Amplifies the Impact of Sleep Loss on Aversive Brain Anticipation","volume":"33","author":"Goldstein","year":"2013","journal-title":"J. Neurosci."},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Pascual-Marqui, R.D., Faber, P., Kinoshita, T., Kochi, K., Milz, P., Nishida, K., and Yoshimura, M. (2018). Comparing EEG\/MEG Neuroimaging Methods Based on Localization Error, False Positive Activity, and False Positive Connectivity. bioRxiv, 269753.","DOI":"10.1101\/269753"},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1016\/j.jneumeth.2015.08.015","article-title":"EEG source localization: Sensor density and head surface coverage","volume":"256","author":"Song","year":"2015","journal-title":"J. Neurosci. Methods"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/20\/7792\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T00:53:49Z","timestamp":1760144029000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/20\/7792"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,14]]},"references-count":59,"journal-issue":{"issue":"20","published-online":{"date-parts":[[2022,10]]}},"alternative-id":["s22207792"],"URL":"https:\/\/doi.org\/10.3390\/s22207792","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,10,14]]}}}