{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,15]],"date-time":"2026-04-15T20:42:56Z","timestamp":1776285776923,"version":"3.50.1"},"reference-count":38,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2023,8,23]],"date-time":"2023-08-23T00:00:00Z","timestamp":1692748800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Since the release of the contrastive language-image pre-training (CLIP) model designed by the OpenAI team, it has been applied in several fields owing to its high accuracy. Sleep staging is an important method of diagnosing sleep disorders, and the completion of sleep staging tasks with high accuracy has always remained the main goal of sleep staging algorithm designers. This study is aimed at designing a multimodal model based on the CLIP model that is more suitable for sleep staging tasks using sleep signals and labels. The pre-training efforts of the model involve five different training sets. Finally, the proposed method is tested on two training sets (EDF-39 and EDF-153), with accuracies of 87.3 and 85.4%, respectively.<\/jats:p>","DOI":"10.3390\/s23177341","type":"journal-article","created":{"date-parts":[[2023,8,23]],"date-time":"2023-08-23T08:20:30Z","timestamp":1692778830000},"page":"7341","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Sleep CLIP: A Multimodal Sleep Staging Model Based on Sleep Signals and Sleep Staging Labels"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-7561-5095","authenticated-orcid":false,"given":"Weijia","family":"Yang","sequence":"first","affiliation":[{"name":"School of Applied Mathematics, Chengdu University of Information Science and Technology, Chengdu 610051, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuxian","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Applied Mathematics, Chengdu University of Information Science and Technology, Chengdu 610051, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiancheng","family":"Hu","sequence":"additional","affiliation":[{"name":"School of Applied Mathematics, Chengdu University of Information Science and Technology, Chengdu 610051, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tuming","family":"Yuan","sequence":"additional","affiliation":[{"name":"School of Applied Mathematics, Chengdu University of Information Science and Technology, Chengdu 610051, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,8,23]]},"reference":[{"key":"ref_1","unstructured":"Chokroverty, S. (2009). Sleep Disorders Medicine E-Book: Basic Science, Technical Considerations, and Clinical Aspects, Elsevier Health Sciences."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Thorpy, M. (2017). International classification of sleep disorders. Sleep Disord. Med. Basic Sci. Tech. Consid. Clin. Asp., 475\u2013484.","DOI":"10.1007\/978-1-4939-6578-6_27"},{"key":"ref_3","first-page":"126","article-title":"Overview of sleep & sleep disorders","volume":"131","author":"Chokroverty","year":"2010","journal-title":"Indian J. Med. Res."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"zsac154","DOI":"10.1093\/sleep\/zsac154","article-title":"Scoring sleep with artificial intelligence enables quantification of sleep stage ambiguity: Hypnodensity based on multiple expert scorers and auto-scoring","volume":"46","author":"Bakker","year":"2023","journal-title":"Sleep"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1350012","DOI":"10.1142\/S0129065713500123","article-title":"Combination of heterogeneous EEG feature extraction methods and stacked sequential learning for sleep stage classification","volume":"23","author":"Herrera","year":"2013","journal-title":"Int. J. Neural Syst."},{"key":"ref_6","first-page":"280","article-title":"Automatic sleep staging based on power spectral density and random forest","volume":"40","author":"Gao","year":"2023","journal-title":"Sheng Wu Yi Xue Gong Cheng Xue Za Zhi = J. Biomed. Eng. = Shengwu Yixue Gongchengxue Zazhi"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1258","DOI":"10.1109\/TIM.2018.2799059","article-title":"Ensemble SVM method for automatic sleep stage classification","volume":"67","author":"Alickovic","year":"2018","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/j.neures.2022.09.009","article-title":"Sleep stage classification in EEG signals using the clustering approach based probability distribution features coupled with classification algorithms","volume":"188","author":"Li","year":"2023","journal-title":"Neurosci. Res."},{"key":"ref_9","unstructured":"Tsinalis, O., Matthews, P.M., and Guo, Y. (2016). Automatic sleep stage scoring with single-channel EEG using convolutional neural networks. arXiv."},{"key":"ref_10","first-page":"1","article-title":"Automated sleep stage classification based on EEG signals using convolutional neural network and residual network","volume":"169","author":"Phan","year":"2019","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_11","first-page":"366","article-title":"Sleep stage classification from EEG signals using transfer learning of convolutional neural networks","volume":"49","author":"Biswal","year":"2019","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_12","first-page":"2305","article-title":"Interpretable sleep stage classification using attention-based convolutional neural network","volume":"24","author":"Phan","year":"2020","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"104299","DOI":"10.1016\/j.bspc.2022.104299","article-title":"CoSleepNet: Automated sleep staging using a hybrid CNN-LSTM network on imbalanced EEG-EOG datasets","volume":"80","author":"Efe","year":"2023","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1998","DOI":"10.1109\/TNSRE.2017.2721116","article-title":"DeepSleepNet: A model for automatic sleep stage scoring based on raw single-channel EEG","volume":"25","author":"Supratak","year":"2017","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"758","DOI":"10.1109\/TNSRE.2018.2813138","article-title":"A deep learning architecture for temporal sleep stage classification using multivariate and multimodal time series","volume":"26","author":"Chambon","year":"2018","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_16","first-page":"102460","article-title":"Hybrid deep learning models for sleep stage classification using EEG signals","volume":"67","author":"Belouchrani","year":"2021","journal-title":"Biomed. Signal Process. Control"},{"key":"ref_17","first-page":"1216","article-title":"Transformer networks for sleep stage classification using EEG signals","volume":"29","author":"Phan","year":"2021","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1075","DOI":"10.1109\/TNSRE.2023.3238764","article-title":"EEG-Based Sleep Stage Classification via Neural Architecture Search","volume":"31","author":"Kong","year":"2023","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_19","first-page":"763","article-title":"Adversarial learning for sleep stage classification using EEG signals","volume":"28","author":"Li","year":"2020","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_20","first-page":"103515","article-title":"A domain adaptation approach for generalizability of sleep stage classifiers","volume":"109","author":"Ghassemi","year":"2020","journal-title":"J. Biomed. Inform."},{"key":"ref_21","unstructured":"Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18\u201324). Learning transferable visual models from natural language supervision. Proceedings of the 38th International Conference on Machine Learning, PMLR, Virtual."},{"key":"ref_22","unstructured":"Su, Y., Lan, T., Liu, Y., Liu, F., Yogatama, D., Wang, Y., Kong, L., and Collier, N. (2022). Language models can see: Plugging visual controls in text generation. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D. (2021, January 11\u201317). Styleclip: Text-driven manipulation of stylegan imagery. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.00209"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Khattak, M.U., Rasheed, H., Maaz, M., Khan, S., and KhanMaple, F.S. (2023, January 18\u201322). Multi-modal prompt learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.","DOI":"10.1109\/CVPR52729.2023.01832"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1171467","DOI":"10.3389\/fphys.2023.1171467","article-title":"DynamicSleepNet: A multi-exit neural network with adaptive inference time for sleep stage classification","volume":"14","author":"Wenjian","year":"2023","journal-title":"Front. Physiol."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Tataraidze, A., Korostovtseva, L., Anishchenko, L., Bochkarev, M., Sviryaev, Y., and Ivashov, S. (2016, January 16\u201320). Bioradiolocation-based sleep stage classification. Proceedings of the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Orlando, FL, USA.","DOI":"10.1109\/EMBC.2016.7591321"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1016\/S1389-9457(01)00149-6","article-title":"Atlas, rules, and recording techniques for the scoring of cyclic alternating pattern (CAP) in human sleep","volume":"2","author":"Terzano","year":"2001","journal-title":"Sleep Med."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1046\/j.1440-1819.1999.00527.x","article-title":"Development of the polysomnographic database on CD-ROM","volume":"53","author":"Ichimaru","year":"1999","journal-title":"Psychiatry Clin. Neurosci."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"e215","DOI":"10.1161\/01.CIR.101.23.e215","article-title":"PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals","volume":"101","author":"Goldberger","year":"2000","journal-title":"Circulation"},{"key":"ref_30","unstructured":"Alvarez-Estevez, D., and Rijsman, R. (2022). Haaglanden Medisch Centrum sleep staging database (version 1.1). PhysioNet."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1185","DOI":"10.1109\/10.867928","article-title":"Analysis of a sleep-dependent neuronal feedback loop: The slow-wave microcontinuity of the EEG","volume":"47","author":"Kemp","year":"2000","journal-title":"IEEE-BME"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/5254.708428","article-title":"Support vector machines","volume":"13","author":"Hearst","year":"1998","journal-title":"IEEE Intell. Syst. Their Appl."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1010933404324","article-title":"Random forests","volume":"45","author":"Breiman","year":"2001","journal-title":"Mach. Learn."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"400","DOI":"10.1109\/TNSRE.2019.2896659","article-title":"SeqSleepNet: End-to-end hierarchical recurrent neural network for sequence-to-sequence automatic sleep staging","volume":"27","author":"Phan","year":"2019","journal-title":"IEEE Trans. Neural Syst. Rehabil. Eng."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Mousavi, S., Fatemeh, A., and Rajendra Acharya, U. (2019). SleepEEGNet: Automated sleep stage scoring with sequence to sequence deep learning approach. PLoS ONE, 14.","DOI":"10.1371\/journal.pone.0216456"},{"key":"ref_36","first-page":"4415","article-title":"U-time: A fully convolutional network for time series segmentation applied to sleep staging","volume":"32","author":"Perslev","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Supratak, A., and Yike, G. (2020, January 20\u201324). TinySleepNet: An efficient deep learning model for sleep stage scoring based on raw single-channel EEG. Proceedings of the 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Montreal, QC, Canada.","DOI":"10.1109\/EMBC44109.2020.9176741"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Jia, Z., Lin, Y., Wang, J., Wang, X., Xie, P., and Zhang, Y. (2021). SalientSleepNet: Multimodal salient wave detection network for sleep staging. arXiv.","DOI":"10.24963\/ijcai.2021\/360"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/17\/7341\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:40:29Z","timestamp":1760128829000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/17\/7341"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,23]]},"references-count":38,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2023,9]]}},"alternative-id":["s23177341"],"URL":"https:\/\/doi.org\/10.3390\/s23177341","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,23]]}}}