{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,18]],"date-time":"2026-11-18T18:31:27Z","timestamp":1795026687941,"version":"4.10.2"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,10,3]],"date-time":"2023-10-03T00:00:00Z","timestamp":1696291200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,10,3]],"date-time":"2023-10-03T00:00:00Z","timestamp":1696291200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100013129","name":"Ministry of SMEs and Startups","doi-asserted-by":"publisher","award":["S3197996"],"award-info":[{"award-number":["S3197996"]}],"id":[{"id":"10.13039\/501100013129","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100013129","name":"Ministry of SMEs and Startups","doi-asserted-by":"publisher","award":["S3197996"],"award-info":[{"award-number":["S3197996"]}],"id":[{"id":"10.13039\/501100013129","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2021M3E5D2A01019547"],"award-info":[{"award-number":["2021M3E5D2A01019547"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100010418","name":"Institute for Information and Communications Technology Promotion","doi-asserted-by":"publisher","award":["2017-0-00451"],"award-info":[{"award-number":["2017-0-00451"]}],"id":[{"id":"10.13039\/501100010418","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Deep-learning approaches with data augmentation have been widely used when developing neuroimaging-based computer-aided diagnosis (CAD) systems. To prevent the inflated diagnostic performance caused by data leakage, a correct cross-validation (CV) method should be employed, but this has been still overlooked in recent deep-learning-based CAD studies. The goal of this study was to investigate the impact of correct and incorrect CV methods on the diagnostic performance of deep-learning-based CAD systems after data augmentation. To this end, resting-state electroencephalogram (EEG) data recorded from post-traumatic stress disorder patients and healthy controls were augmented using a cropping method with different window sizes, respectively. Four different CV approaches were used to estimate the diagnostic performance of the CAD system, i.e., subject-wise CV (sCV), overlapped sCV (oSCV), trial-wise CV (tCV), and overlapped tCV (otCV). Diagnostic performances were evaluated using two deep-learning models based on convolutional neural network. Data augmentation can increase the performance with all CVs, but inflated diagnostic performances were observed when using incorrect CVs (tCV and otCV) due to data leakage. Therefore, the correct CV (sCV and osCV) should be used to develop a deep-learning-based CAD system. We expect that our investigation can provide deep-insight for researchers who plan to develop neuroimaging-based CAD systems for psychiatric disorders using deep-learning algorithms with data augmentation.<\/jats:p>","DOI":"10.1038\/s41598-023-43542-8","type":"journal-article","created":{"date-parts":[[2023,10,3]],"date-time":"2023-10-03T10:02:55Z","timestamp":1696327375000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":28,"title":["Risk of data leakage in estimating the diagnostic performance of a deep-learning-based computer-aided system for psychiatric disorders"],"prefix":"10.1038","volume":"13","author":[{"given":"Hyung-Tak","family":"Lee","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hye-Ran","family":"Cheon","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seung-Hwan","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Miseon","family":"Shim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Han-Jeong","family":"Hwang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,10,3]]},"reference":[{"key":"43542_CR1","doi-asserted-by":"publisher","first-page":"314","DOI":"10.1016\/j.schres.2016.05.007","volume":"176","author":"M Shim","year":"2016","unstructured":"Shim, M., Hwang, H. J., Kim, D. W., Lee, S. H. & Im, C. H. Machine-learning-based diagnosis of schizophrenia using combined sensor-level and source-level EEG features. Schizophr. Res. 176, 314\u2013319 (2016).","journal-title":"Schizophr. Res."},{"key":"43542_CR2","doi-asserted-by":"publisher","first-page":"101707","DOI":"10.1016\/j.bspc.2019.101707","volume":"56","author":"HU Amin","year":"2020","unstructured":"Amin, H. U., Yusoff, M. Z. & Ahmad, R. F. A novel approach based on wavelet analysis and arithmetic coding for automated detection and diagnosis of epileptic seizure in EEG signals using machine learning techniques. Biomed Signal Process Control 56, 101707 (2020).","journal-title":"Biomed Signal Process Control"},{"key":"43542_CR3","doi-asserted-by":"publisher","first-page":"102227","DOI":"10.1016\/j.bspc.2020.102227","volume":"63","author":"A Ahmadi","year":"2021","unstructured":"Ahmadi, A., Kashefi, M., Shahrokhi, H. & Nazari, M. A. Computer aided diagnosis system using deep convolutional neural networks for ADHD subtypes. Biomed Signal Process Control 63, 102227 (2021).","journal-title":"Biomed Signal Process Control"},{"key":"43542_CR4","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1177\/1550059420916634","volume":"52","author":"C Uyulan","year":"2021","unstructured":"Uyulan, C. et al. Major depressive disorder classification based on different convolutional neural network models: Deep learning approach. Clin. EEG Neurosci. 52, 38\u201351 (2021).","journal-title":"Clin. EEG Neurosci."},{"key":"43542_CR5","doi-asserted-by":"crossref","unstructured":"Wang, F., Zhong, S., Peng, J., Jiang, J. & Liu, Y. Data augmentation for EEG-based emotion recognition with deep convolutional neural networks. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) vol. 10705 LNCS 82\u201393 (Springer Verlag, 2018).","DOI":"10.1007\/978-3-319-73600-6_8"},{"key":"43542_CR6","doi-asserted-by":"publisher","first-page":"102152","DOI":"10.1016\/j.bspc.2020.102152","volume":"62","author":"PH Gubert","year":"2020","unstructured":"Gubert, P. H., Costa, M. H., Silva, C. D. & Trofino-Neto, A. The performance impact of data augmentation in CSP-based motor-imagery systems for BCI applications. Biomed Signal Process Control 62, 102152 (2020).","journal-title":"Biomed Signal Process Control"},{"key":"43542_CR7","unstructured":"Le Guennec, A., Malinowski, S. & Tavenard, R. Data augmentation for time series classification using convolutional neural networks. In ECML\/PKDD Workshop on Advanced Analytics and Learning on Temporal Data (2016)."},{"key":"43542_CR8","doi-asserted-by":"publisher","first-page":"108312","DOI":"10.1016\/j.jneumeth.2019.108312","volume":"324","author":"Z Mousavi","year":"2019","unstructured":"Mousavi, Z., Yousefi Rezaii, T., Sheykhivand, S., Farzamnia, A. & Razavi, S. N. Deep convolutional neural network for classification of sleep stages from single-channel EEG signals. J. Neurosci. Methods 324, 108312 (2019).","journal-title":"J. Neurosci. Methods"},{"key":"43542_CR9","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/j.eswa.2018.04.021","volume":"107","author":"I Ullah","year":"2018","unstructured":"Ullah, I. et al. An automated system for epilepsy detection using EEG brain signals based on deep learning approach. Expert Syst. Appl. 107, 61\u201371 (2018).","journal-title":"Expert Syst. Appl."},{"key":"43542_CR10","doi-asserted-by":"crossref","unstructured":"Avcu, M. T., Zhang, Z. & Shih Chan, D. W. Seizure detection using least eeg channels by deep convolutional neural network. In ICASSP 2019\u20132019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 1120\u20131124 (IEEE, 2019).","DOI":"10.1109\/ICASSP.2019.8683229"},{"key":"43542_CR11","unstructured":"Lu, D. & Triesch, J. Residual deep convolutional neural network for EEG signal classification in epilepsy (2019)."},{"key":"43542_CR12","doi-asserted-by":"publisher","first-page":"7980","DOI":"10.1038\/s41598-021-87157-3","volume":"11","author":"M Shim","year":"2021","unstructured":"Shim, M., Lee, S. H. & Hwang, H. J. Inflated prediction accuracy of neuropsychiatric biomarkers caused by data leakage in feature selection. Sci. Rep. 11, 7980 (2021).","journal-title":"Sci. Rep."},{"key":"43542_CR13","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1016\/j.ijpsycho.2006.07.009","volume":"64","author":"R Miller","year":"2007","unstructured":"Miller, R. Theory of the normal waking EEG: From single neurones to waveforms in the alpha, beta and gamma frequency ranges. Int. J. Psychophysiol. 64, 18\u201323 (2007).","journal-title":"Int. J. Psychophysiol."},{"key":"43542_CR14","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1109\/JBHI.2020.2984238","volume":"25","author":"D Ahmedt-Aristizabal","year":"2021","unstructured":"Ahmedt-Aristizabal, D. et al. Identification of children at risk of schizophrenia via deep learning and EEG responses. IEEE J. Biomed. Health Inform. 25, 69\u201376 (2021).","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"43542_CR15","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1016\/j.compbiomed.2018.05.019","volume":"99","author":"\u039a Tsiouris","year":"2018","unstructured":"Tsiouris, \u039a et al. A long short-term memory deep learning network for the prediction of epileptic seizures using EEG signals. Comput. Biol. Med. 99, 24\u201337 (2018).","journal-title":"Comput. Biol. Med."},{"key":"43542_CR16","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1177\/0954411920966937","volume":"235","author":"K Singh","year":"2021","unstructured":"Singh, K., Singh, S. & Malhotra, J. Spectral features based convolutional neural network for accurate and prompt identification of schizophrenic patients. Proc. Inst. Mech. Eng. H 235, 167\u2013184 (2021).","journal-title":"Proc. Inst. Mech. Eng. H"},{"key":"43542_CR17","doi-asserted-by":"publisher","first-page":"1736","DOI":"10.3390\/s19071736","volume":"19","author":"I Majidov","year":"2019","unstructured":"Majidov, I. & Whangbo, T. Efficient classification of motor imagery electroencephalography signals using deep learning methods. Sensors (Switz.) 19, 1736 (2019).","journal-title":"Sensors (Switz.)"},{"key":"43542_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s13755-021-00139-7","volume":"9","author":"H Akbari","year":"2021","unstructured":"Akbari, H., Sadiq, M. T. & Rehman, A. U. Classification of normal and depressed EEG signals based on centered correntropy of rhythms in empirical wavelet transform domain. Health Inf. Sci. Syst. 9, 1\u201315 (2021).","journal-title":"Health Inf. Sci. Syst."},{"key":"43542_CR19","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-0716-1418-1","volume-title":"An Introduction to Statistical Learning","author":"G James","year":"2021","unstructured":"James, G., Witten, D., Hastie, T. & Tibshirani, R. An Introduction to Statistical Learning (Springer, 2021)."},{"key":"43542_CR20","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/TIM.2021.3053999","volume":"70","author":"A Seal","year":"2021","unstructured":"Seal, A. et al. DeprNet: A deep convolution neural network framework for detecting depression using EEG. IEEE Trans. Instrum. Meas. 70, 1\u201313 (2021).","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"43542_CR21","first-page":"1","volume":"9","author":"M Rashed-Al-Mahfuz","year":"2021","unstructured":"Rashed-Al-Mahfuz, M. et al. A deep convolutional neural network method to detect seizures and characteristic frequencies using epileptic electroencephalogram (EEG) data. IEEE J. Transl. Eng. Health Med. 9, 1\u201312 (2021).","journal-title":"IEEE J. Transl. Eng. Health Med."},{"key":"43542_CR22","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1016\/j.cmpb.2018.04.012","volume":"161","author":"UR Acharya","year":"2018","unstructured":"Acharya, U. R. et al. Automated EEG-based screening of depression using deep convolutional neural network. Comput. Methods Progr. Biomed. 161, 103\u2013113 (2018).","journal-title":"Comput. Methods Progr. Biomed."},{"key":"43542_CR23","doi-asserted-by":"publisher","first-page":"056013","DOI":"10.1088\/1741-2552\/aace8c","volume":"15","author":"VJ Lawhern","year":"2018","unstructured":"Lawhern, V. J. et al. EEGNet: A compact convolutional neural network for EEG-based brain-computer interfaces. J. Neural Eng. 15, 056013 (2018).","journal-title":"J. Neural Eng."},{"key":"43542_CR24","unstructured":"Van Der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9 (2008)."},{"key":"43542_CR25","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4419-9878-1","volume-title":"Robust Data Mining","author":"P Xanthopoulos","year":"2013","unstructured":"Xanthopoulos, P., Pardalos, P. M. & Trafalis, T. B. Robust Data Mining (Springer, 2013)."},{"key":"43542_CR26","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1145\/3446776","volume":"64","author":"C Zhang","year":"2021","unstructured":"Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Commun. ACM 64, 107\u2013115 (2021).","journal-title":"Commun. ACM"}],"updated-by":[{"DOI":"10.1038\/s41598-024-52295-x","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2024,1,25]],"date-time":"2024-01-25T00:00:00Z","timestamp":1706140800000}}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-43542-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-43542-8","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-43542-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,25]],"date-time":"2024-01-25T07:15:53Z","timestamp":1706166953000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-023-43542-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,3]]},"references-count":26,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["43542"],"URL":"https:\/\/doi.org\/10.1038\/s41598-023-43542-8","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,10,3]]},"assertion":[{"value":"24 October 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 September 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 October 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 January 2024","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A Correction to this paper has been published:","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"https:\/\/doi.org\/10.1038\/s41598-024-52295-x","URL":"https:\/\/doi.org\/10.1038\/s41598-024-52295-x","order":7,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"16633"}}