{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T15:05:10Z","timestamp":1781535910272,"version":"3.54.5"},"reference-count":51,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2020,7,31]],"date-time":"2020-07-31T00:00:00Z","timestamp":1596153600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["16SV7223K"],"award-info":[{"award-number":["16SV7223K"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["16SV7512"],"award-info":[{"award-number":["16SV7512"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The scarcity of labelled time-series data can hinder a proper training of deep learning models. This is especially relevant for the growing field of ubiquitous computing, where data coming from wearable devices have to be analysed using pattern recognition techniques to provide meaningful applications. To address this problem, we propose a transfer learning method based on attributing sensor modality labels to a large amount of time-series data collected from various application fields. Using these data, our method firstly trains a Deep Neural Network (DNN) that can learn general characteristics of time-series data, then transfers it to another DNN designed to solve a specific target problem. In addition, we propose a general architecture that can adapt the transferred DNN regardless of the sensors used in the target field making our approach in particular suitable for multichannel data. We test our method for two ubiquitous computing problems\u2014Human Activity Recognition (HAR) and Emotion Recognition (ER)\u2014and compare it a baseline training the DNN without using transfer learning. For HAR, we also introduce a new dataset, Cognitive Village-MSBand (CogAge), which contains data for 61 atomic activities acquired from three wearable devices (smartphone, smartwatch, and smartglasses). Our results show that our transfer learning approach outperforms the baseline for both HAR and ER.<\/jats:p>","DOI":"10.3390\/s20154271","type":"journal-article","created":{"date-parts":[[2020,8,3]],"date-time":"2020-08-03T06:16:47Z","timestamp":1596435407000},"page":"4271","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":63,"title":["Deep Transfer Learning for Time Series Data Based on Sensor Modality Classification"],"prefix":"10.3390","volume":"20","author":[{"given":"Fr\u00e9d\u00e9ric","family":"Li","sequence":"first","affiliation":[{"name":"Institute of Medical Informatics, University of L\u00fcbeck, Ratzeburger Allee 160, 23538 L\u00fcbeck, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kimiaki","family":"Shirahama","sequence":"additional","affiliation":[{"name":"Department of Informatics, Kindai University, 3-4-1 Kowakae, Higashiosaka City, Osaka 577-8502, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3288-750X","authenticated-orcid":false,"given":"Muhammad Adeel","family":"Nisar","sequence":"additional","affiliation":[{"name":"Institute of Medical Informatics, University of L\u00fcbeck, Ratzeburger Allee 160, 23538 L\u00fcbeck, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3210-3891","authenticated-orcid":false,"given":"Xinyu","family":"Huang","sequence":"additional","affiliation":[{"name":"Institute of Medical Informatics, University of L\u00fcbeck, Ratzeburger Allee 160, 23538 L\u00fcbeck, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4877-8287","authenticated-orcid":false,"given":"Marcin","family":"Grzegorzek","sequence":"additional","affiliation":[{"name":"Institute of Medical Informatics, University of L\u00fcbeck, Ratzeburger Allee 160, 23538 L\u00fcbeck, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2020,7,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.patrec.2018.02.010","article-title":"Deep Learning for Sensor-based Activity Recognition: A Survey","volume":"119","author":"Wang","year":"2018","journal-title":"Pattern Recognit. Lett."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Li, F., Shirahama, K., Nisar, A.M., K\u00f6ping, L., and Grzegorzek, M. (2018). Comparison of Feature Learning Methods for Human Activity Recognition Using Wearable Sensors. Sensors, 18.","DOI":"10.3390\/s18020679"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D., and Fergus, R. (2014, January 6\u201312). Vizualizing and Understanding Convolutional Networks. Proceedings of the European conference on computer vision, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ordonez, F.J., and Roggen, D. (2016). Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors, 16.","DOI":"10.3390\/s16010115"},{"key":"ref_5","unstructured":"Hammerla, N.Y., Halloran, S., and Pl\u00f6tz, T. (2016). Deep, Convolutional and Recurrent Models for Human Activity Recognition using Wearables. arXiv."},{"key":"ref_6","unstructured":"Li, X., Zhang, P., Song, D., Yu, G., Hou, Y., and Hu, B. (2015, January 13). EEG-based Emotion Identification Using Unsupervised Deep Feature Learning. Proceedings of the NeuroIR, Santiago, Chile."},{"key":"ref_7","unstructured":"Liu, W., Zheng, W.L., and Lu, B.L. (2015, January 13). Multimodal Emotion Recognition Using Multimodal Deep-Learning. Proceedings of the SIGIR2015 Workshop on Neuro-Physiological Methods in IR Research, Santiago, Chile."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A Survey on Transfer Learning","volume":"22","author":"Pan","year":"2009","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_9","unstructured":"Glorot, X., and Bengio, Y. (2010, January 13\u201315). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Chia Laguna Resort, Italy."},{"key":"ref_10","unstructured":"Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. (2014, January 8\u201313). How Transferable are Features in Deep Neural Networks?. Proceedings of the NIPS, Montr\u00e9al, QC, Canada."},{"key":"ref_11","unstructured":"Huh, M., Agrawal, P., and Efros, A.A. (July, January 26). What Makes ImageNet Good for Transfer Learning?. Proceedings of the CVPR, Las Vegas, NV, USA."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"917","DOI":"10.1007\/s10618-019-00619-1","article-title":"Deep Learning for Time Series Classification: A Review","volume":"33","author":"Fawaz","year":"2019","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_13","unstructured":"Fawaz, H.I., Forestier, G., Weber, J., Idoumghar, L., and Muller, P.A. (2018, January 10\u201313). Transfer Learning for Time-Series Classification. Proceedings of the IEEE International Conference on Big Data, Seattle, WA, USA."},{"key":"ref_14","unstructured":"Malhotra, P., TV, V., Vig, L., Agarwal, P., and Shroff, G. (2017, January 26\u201328). TimeNet: Pre-trained deep recurrent neural networks for time series classification. Proceedings of the ESANN, Bruges, Belgium."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1109\/T-AFFC.2011.15","article-title":"DEAP: A Database for Emotion Analysis Using Physiological Signals","volume":"3","author":"Koelstra","year":"2012","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_16","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20138). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the NIPS, Lake Tahoe, CA, USA."},{"key":"ref_17","unstructured":"Simonyan, K., and Zisserman, A. (2015, January 7\u20139). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the ICLR, San Diego, CA, USA."},{"key":"ref_18","unstructured":"He, K., Zhang, X., Ren, S., and Sen, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the CVPR, Las Vegas, NV, USA."},{"key":"ref_19","unstructured":"Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. (2014, January 22\u201324). DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. Proceedings of the ICML, Beijing, China."},{"key":"ref_20","unstructured":"Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y. (2014, January 14\u201316). OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks. Proceedings of the ICLR, Banff, AB, Canada."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 24\u201327). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the CVPR, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.81"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Oquab, M., Bottou, L., Laptev, I., and Sivic, J. (2014, January 24\u201327). Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks. Proceedings of the CVPR, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.222"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/j.patcog.2017.07.019","article-title":"On Automated Source Selection for Transfer Learning in Convolutional Neural Networks","volume":"73","author":"Afridi","year":"2018","journal-title":"Pattern Recognit."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Guo, Y., Shi, H., Kumar, A., Grauman, K., Rosing, T., and Feris, R. (2019, January 16\u201320). SpotTune: Transfer Learning Through Adaptive Fine-Tuning. Proceedings of the CVPR, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00494"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"107049","DOI":"10.1016\/j.patcog.2019.107049","article-title":"A Baseline Regularisation Scheme for Transfer Learning with Convolutional Neural Networks","volume":"98","author":"Li","year":"2019","journal-title":"Pattern Recognit."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Pathak, D., Krahenbuhl, P., Donahue, J., and Efros, A.A. (2016, January 26\u20131). Context Encoders: Feature Learning by Inpainting. Proceedings of the CVPR, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.278"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1007\/s10115-013-0665-3","article-title":"Transfer Learning for Activity Recognition: A Survey","volume":"36","author":"Cook","year":"2013","journal-title":"Knowl. Inf. Syst."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Morales, F.J.O., and Roggen, D. (2016, January 17\u201321). Deep Convolutional Feature Transfer Across Mobile Activity Recognition Domains, Sensor Modalities and Locations. Proceedings of the ISWC, Kobe, Japan.","DOI":"10.1145\/2971763.2971764"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Khan, A.H., Roy, N., and Misra, A. (2018, January 19\u201323). Scaling Human Activity Recognition via Deep Learning-based Domain Adaptation. Proceedings of the PERCOM, Athens, Greece.","DOI":"10.1109\/PERCOM.2018.8444585"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Wang, J., Chen, Y., Hu, L., Peng, X., and Yu, P.S. (2018, January 19\u201323). Stratified Transfer Learning for Cross-Domain Activity Recognition. Proceedings of the PERCOM, Athens, Greece.","DOI":"10.1109\/PERCOM.2018.8444572"},{"key":"ref_31","unstructured":"Chen, Y., Keogh, E., Hu, B., Begum, N., Bagnall, A., Mueen, A., and Batista, G. (2020, July 30). The UCR Time Series Classification Archive. Available online: www.cs.ucr.edu\/~eamonn\/time_series_data\/."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1109\/SURV.2012.110112.00192","article-title":"A Survey on Human Activity Recognition using Wearable Sensors","volume":"15","author":"Lara","year":"2013","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_33","unstructured":"Picard, R.W. (2000). Affective Computing, The MIT Press."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1109\/79.911197","article-title":"Emotion Recognition in Human-Computer Interaction","volume":"18","author":"Cowie","year":"2001","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Gr\u00fcnewald, A., Kr\u00f6nert, D., Li, F., Kampling, H., P\u00f6hler, J., Br\u00fcck, R., Littau, J., Schnieber, K., Piet, A., and Grzegorzek, M. (2018, January 29\u201331). Biomedical Data Acquisition and Processing to Recognize Emotions for Affective Learning. Proceedings of the IEEE International Conference on Bioinformatics and Bioengineering, Taichung, Taiwan.","DOI":"10.1109\/BIBE.2018.00031"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Rozgic, V., Vitaladevni, S.V., and Prasad, R. (2013, January 26\u201330). Robust EEG emotion classification using segment level decision fusion. Proceedings of the ICASSP, Vancouver, BC, Canada.","DOI":"10.1109\/ICASSP.2013.6637858"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1155\/2014\/627892","article-title":"EEG-based Emotion Recognition using Deep Learning Network with Principal Component Based Covariate Shift Adaptation","volume":"14","author":"Jirayucharoensak","year":"2014","journal-title":"Sci. World J."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Ma, J., Tang, H., Zheng, W.L., and Lu, B.L. (2019, January 21\u201325). Emotion Recognition using Multimodal Residual LSTM Network. Proceedings of the ACMMM, Nice, France.","DOI":"10.1145\/3343031.3350871"},{"key":"ref_39","unstructured":"Smirnov, D., and Nguifo, E.M. (2018, January 10\u201314). Time-series Classification with Recurrent Neural Networks. Proceedings of the ECML\/PKDD Workshop on Advanced Analytics and Learning on Temporal Data, Dublin, Ireland."},{"key":"ref_40","unstructured":"Dheeru, D., and Taniskidou, E.K. (2020, July 30). UCI Machine Learning Repository. Available online: http:\/\/archive.ics.uci.edu\/ml."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"2033","DOI":"10.1016\/j.patrec.2012.12.014","article-title":"The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition","volume":"34","author":"Chavarriaga","year":"2013","journal-title":"Pattern Recognit. Lett."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"618","DOI":"10.1016\/j.snb.2015.03.028","article-title":"Reservoir computing compensates slow response of chemosensor arrays exposed to fast varying gas concentrations in continuous monitoring","volume":"215","author":"Fonollosa","year":"2015","journal-title":"Sens. Actuators B Chem."},{"key":"ref_43","unstructured":"Roesler, O. (2020, July 30). The EEG Eye State Dataset. Available online: https:\/\/archive.ics.uci.edu\/ml\/datasets\/EEG+Eye+State#."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1016\/j.enbuild.2017.01.083","article-title":"Data driven prediction models of energy use of appliances in a low-energy house","volume":"140","author":"Candanedo","year":"2017","journal-title":"Energy Build."},{"key":"ref_45","unstructured":"Kingma, D.P., and Welling, M. (2014, January 22\u201324). Auto-Encoding Variational Bayes. Proceedings of the ICML, Beijing, China."},{"key":"ref_46","unstructured":"Zeiler, M.D. (2012). ADADELTA: An Adaptive Learning Rate Method. arXiv."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"162","DOI":"10.3389\/fnins.2018.00162","article-title":"Exploring EEG Features in Cross-subject Emotion Recognition","volume":"12","author":"Li","year":"2019","journal-title":"Front. Neurosci."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Yu, R., Li, A., Chen, C., Morariu, J.L.V.I., Han, X., Gao, M., and Lin, C. (2018, January 18\u201323). NISP: Pruning Networks using Neuron Importance Score Propagation. Proceedings of the CVPR, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00958"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Roffo, G., Melzi, S., and Cristani, M. (2015, January 11\u201318). Infinite Feature Selection. Proceedings of the ICCV, Araucano Park, Chile.","DOI":"10.1109\/ICCV.2015.478"},{"key":"ref_50","unstructured":"Simonyan, K., Vedaldi, A., and Zisserman, A. (2014, January 14\u201316). Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. Proceedings of the ICLR Workshop, Banff, AB, Canada."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/1500000033","article-title":"LifeLogging: Personal Big Data","volume":"8","author":"Gurrin","year":"2014","journal-title":"Found. Trends Inf. Retr."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/15\/4271\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:53:15Z","timestamp":1760176395000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/15\/4271"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,31]]},"references-count":51,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2020,8]]}},"alternative-id":["s20154271"],"URL":"https:\/\/doi.org\/10.3390\/s20154271","relation":{"is-referenced-by":[{"id-type":"doi","id":"10.1007\/s40747-025-02120-3","asserted-by":"object"}]},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,31]]}}}