{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,29]],"date-time":"2026-07-29T07:16:36Z","timestamp":1785309396098,"version":"3.55.0"},"reference-count":49,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2016,1,18]],"date-time":"2016-01-18T00:00:00Z","timestamp":1453075200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Google Faculty Research Award","award":["Is deep learning useful for wearable activity recognition?"],"award-info":[{"award-number":["Is deep learning useful for wearable activity recognition?"]}]},{"DOI":"10.13039\/501100000266","name":"EPSRC","doi-asserted-by":"publisher","award":["EP\/N007816\/1"],"award-info":[{"award-number":["EP\/N007816\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Human activity recognition (HAR) tasks have traditionally been solved using engineered features obtained by heuristic processes. Current research suggests that deep convolutional neural networks are suited to automate feature extraction from raw sensor inputs. However, human activities are made of complex sequences of motor movements, and capturing this temporal dynamics is fundamental for successful HAR. Based on the recent success of recurrent neural networks for time series domains, we propose a generic deep framework for activity recognition based on convolutional and LSTM recurrent units, which: (i) is suitable for multimodal wearable sensors; (ii) can perform sensor fusion naturally; (iii) does not require expert knowledge in designing features; and (iv) explicitly models the temporal dynamics of feature activations. We evaluate our framework on two datasets, one of which has been used in a public activity recognition challenge. Our results show that our framework outperforms competing deep non-recurrent networks on the challenge dataset by 4% on average; outperforming some of the previous reported results by up to 9%. Our results show that the framework can be applied to homogeneous sensor modalities, but can also fuse multimodal sensors to improve performance. We characterise key architectural hyperparameters\u2019 influence on performance to provide insights about their optimisation.<\/jats:p>","DOI":"10.3390\/s16010115","type":"journal-article","created":{"date-parts":[[2016,1,18]],"date-time":"2016-01-18T10:58:33Z","timestamp":1453114713000},"page":"115","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2308,"title":["Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition"],"prefix":"10.3390","volume":"16","author":[{"given":"Francisco","family":"Ord\u00f3\u00f1ez","sequence":"first","affiliation":[{"name":"Wearable Technologies, Sensor Technology Research Centre, University of Sussex, Brighton BN1 9RH, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Roggen","sequence":"additional","affiliation":[{"name":"Wearable Technologies, Sensor Technology Research Centre, University of Sussex, Brighton BN1 9RH, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2016,1,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"949","DOI":"10.1109\/TSMCA.2009.2025137","article-title":"The resident in the loop: Adapting the smart home to the user","volume":"39","author":"Rashidi","year":"2009","journal-title":"IEEE Trans. Syst. Man. Cybern. J. Part A"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Patel, S., Park, H., Bonato, P., Chan, L., and Rodgers, M. (2012). A review of wearable sensors and systems with application in rehabilitation. J. NeuroEng. Rehabil., 9.","DOI":"10.1186\/1743-0003-9-21"},{"key":"ref_3","unstructured":"Avci, A., Bosch, S., Marin-Perianu, M., Marin-Perianu, R., and Havinga, P. (2010, January 22\u201323). Activity Recognition Using Inertial Sensing for Healthcare, Wellbeing and Sports Applications: A Survey. Proceedings of the 23rd International Conference on Architecture of Computing Systems (ARCS), Hannover, Germany."},{"key":"ref_4","unstructured":"Mazilu, S., Blanke, U., Hardegger, M., Tr\u00f6ster, G., Gazit, E., and Hausdorff, J.M. (May, January 26). GaitAssist: A Daily-Life Support and Training System for Parkinson\u2019s Disease Patients with Freezing of Gait. Proceedings of the ACM Conference on Human Factors in Computing Systems (SIGCHI), Toronto, ON, Canada."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"203","DOI":"10.1016\/j.pmcj.2012.06.002","article-title":"The mobile fitness coach: Towards individualized skill assessment using personalized mobile devices","volume":"9","author":"Kranz","year":"2013","journal-title":"Perv. Mob. Comput."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/MPRV.2008.40","article-title":"Wearable Activity Tracking in Car Manufacturing","volume":"7","author":"Stiefmeier","year":"2008","journal-title":"IEEE Perv. Comput. Mag."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2033","DOI":"10.1016\/j.patrec.2012.12.014","article-title":"The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition","volume":"34","author":"Chavarriaga","year":"2013","journal-title":"Pattern Recognit. Lett."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2499621","article-title":"A Tutorial on Human Activity Recognition Using Body-worn Inertial Sensors","volume":"46","author":"Bulling","year":"2014","journal-title":"ACM Comput. Surv."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Roggen, D., Cuspinera, L.P., Pombo, G., Ali, F., and Nguyen-Dinh, L. (2015, January 9\u201311). Limited-Memory Warping LCSS for Real-Time Low-Power Pattern Recognition in Wireless Nodes. Proceedings of the 12th European Conference Wireless Sensor Networks (EWSN), Porto, Portugal.","DOI":"10.1007\/978-3-319-15582-1_10"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1109\/MPRV.2014.52","article-title":"In-Home Activity Recognition: Bayesian Inference for Hidden Markov Models","volume":"13","author":"Ordonez","year":"2014","journal-title":"Perv. Comput. IEEE"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1088\/0967-3334\/30\/4\/R01","article-title":"Activity identification using body-mounted sensors: A review of classification techniques","volume":"30","author":"Preece","year":"2009","journal-title":"Physiol. Meas."},{"key":"ref_12","first-page":"645","article-title":"Preprocessing techniques for context recognition from accelerometer data","volume":"14","author":"Figo","year":"2010","journal-title":"Perv. Mob. Comput."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Lee, H., Grosse, R., Ranganath, R., and Ng, A.Y. (2009, January 14\u201318). Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations. Proceedings of the 26th Annual International Conference on Machine Learning (ICML), Montreal, QC, Canada.","DOI":"10.1145\/1553374.1553453"},{"key":"ref_14","unstructured":"Lee, H., Pham, P., Largman, Y., and Ng, A. (2008, January 8\u201310). Unsupervised feature learning for audio classification using convolutional deep belief networks. Proceedings of the 22th Annual Conference on Advances in Neural Information Processing Systems (NIPS), Vancouver, BC, Canada."},{"key":"ref_15","unstructured":"LeCun, Y., and Bengio, Y. (1998). The Handbook of Brain Theory and Neural Networks, MIT Press."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Sainath, T., Vinyals, O., Senior, A., and Sak, H. (2015, January 19\u201324). Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks. Proceedings of the 40th International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, Australia.","DOI":"10.1109\/ICASSP.2015.7178838"},{"key":"ref_17","unstructured":"Yang, J.B., Nguyen, M.N., San, P.P., Li, X.L., and Krishnaswamy, S. (2015, January 25\u201331). Deep Convolutional Neural Networks On Multichannel Time Series For Human Activity Recognition. Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI), Buenos Aires, Argentina."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1016\/0893-9659(91)90080-F","article-title":"Turing computability with neural nets","volume":"4","author":"Siegelmann","year":"1991","journal-title":"Appl. Math. Lett."},{"key":"ref_19","first-page":"115","article-title":"Learning precise timing with LSTM recurrent networks","volume":"3","author":"Gers","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Graves, A., Mohamed, A.R., and Hinton, G. (2013, January 26\u201331). Speech recognition with deep recurrent neural networks. Proceeedings of the 38th International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, USA.","DOI":"10.1109\/ICASSP.2013.6638947"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Palaz, D., Magimai.-Doss, M., and Collobert, R. (2015, January 6\u201310). Analysis of CNN-based Speech Recognition System using Raw Speech as Input. Proceedings of the 16th Annual Conference of International Speech Communication Association (Interspeech), Dresden, Germany.","DOI":"10.21437\/Interspeech.2015-3"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Pigou, L., Oord, A.V.D., Dieleman, S., van Herreweghe, M., and Dambre, J. (2015). Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video. arXiv Preprint, arXiv:1506.01911.","DOI":"10.1007\/s11263-016-0957-7"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Zeng, M., Nguyen, L.T., Yu, B., Mengshoel, O.J., Zhu, J., Wu, P., and Zhang, J. (2014, January 6\u20137). Convolutional Neural Networks for human activity recognition using mobile sensors. Proceedings of the 6th IEEE International Conference on Mobile Computing, Applications and Services (MobiCASE), Austin, TX, USA.","DOI":"10.4108\/icst.mobicase.2014.257786"},{"key":"ref_24","unstructured":"Oord, A.V.D., Dieleman, S., and Schrauwen, B. (2013, January 5\u201310). Deep content-based music recommendation. Proeedings of the Neural Information Processing Systems, Lake Tahoe, NE, USA."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1016\/j.neunet.2014.08.005","article-title":"Deep convolutional neural networks for large-scale speech tasks","volume":"64","author":"Sainath","year":"2015","journal-title":"Neural Netw."},{"key":"ref_26","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20136). Imagenet classification with deep convolutional neural networks. Proceedings of the 25th Conference on Advances in Neural Information Processing Systems (NIPS), Lake Tahoe, NV, USA."},{"key":"ref_27","unstructured":"Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y. (2013). Overfeat: Integrated recognition, localization and detection using convolutional networks. Cornell Univ. Lib., arXiv:1312.6229."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Toshev, A., and Szegedy, C. (2014, January 6\u201312). Deeppose: Human pose estimation via deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Zurich, Switzerland.","DOI":"10.1109\/CVPR.2014.214"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Deng, L., and Platt, J.C. (2014, January 14\u201318). Ensemble deep learning for speech recognition. Proceedings of the 15th Annual Conference of International Speech Communication Association (Interspeech), Singapore.","DOI":"10.21437\/Interspeech.2014-433"},{"key":"ref_30","unstructured":"Ng, J.Y.H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., and Toderici, G. (2015). Beyond short snippets: Deep networks for video classification. Cornell Univ. Lab., arXiv:1503.08909."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks","volume":"313","author":"Hinton","year":"2006","journal-title":"Science"},{"key":"ref_32","unstructured":"Pl\u00f6tz, T., Hammerla, N.Y., and Olivier, P. (2011, January 16\u201322). Feature Learning for Activity Recognition in Ubiquitous Computing. Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI), Barcelona, Spain."},{"key":"ref_33","unstructured":"Karpathy, A., Johnson, J., and Li, F.F. (2015). Visualizing and understanding recurrent networks. Cornell Univ. Lab., arXiv:1506.02078."},{"key":"ref_34","unstructured":"Dieleman, S., Schl\u00fcter, J., Raffel, C., Olson, E., S\u00f8nderby, S.K., Nouri, D., Maturana, D., Thoma, M., Battenberg, E., and Kelly, J. (2015). Lasagne: First Release, Zenodo."},{"key":"ref_35","unstructured":"Dauphin, Y.N., de Vries, H., Chung, J., and Bengio, Y. (2015). RMSProp and equilibrated adaptive learning rates for non-convex optimization. arXiv, arXiv:1502.04390."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., F\u00f6rster, K., Tr\u00f6ster, G., Lukowicz, P., Bannach, D., Pirkl, G., and Ferscha, A. (2010, January 15\u201318). Collecting complex activity data sets in highly rich networked sensor environments. Proceedings of the 7th IEEE International Conference on Networked Sensing Systems (INSS), Kassel, Germany.","DOI":"10.1109\/INSS.2010.5573462"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Reiss, A., and Stricker, D. (2012, January 18\u201322). Introducing a New Benchmarked Dataset for Activity Monitoring. Proceedings of the 16th International Symposium on Wearable Computers (ISWC), Newcastle, UK.","DOI":"10.1109\/ISWC.2012.13"},{"key":"ref_38","unstructured":"Zappi, P., Lombriser, C., Farella, E., Roggen, D., Benini, L., and Tr\u00f6ster, G. (February, January 30). Activity recognition from on-body sensors: accuracy-power trade-off by dynamic sensor selection. Proceedings of the 5th European Conference on Wireless Sensor Networks (EWSN), Bologna, Italy."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Banos, O., Garcia, R., Holgado, J.A., Damas, M., Pomares, H., Rojas, I., Saez, A., and Villalonga, C. (2014, January 2\u20135). mHealthDroid: a novel framework for agile development of mobile health applications. Proceedings of the 6th International Work-conference on Ambient Assisted Living an Active Ageing, Belfast, UK.","DOI":"10.1007\/978-3-319-13105-4_14"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1007\/s00779-013-0638-2","article-title":"Activity recognition for creatures of habit","volume":"18","author":"Gordon","year":"2014","journal-title":"Pers. Ubiquitous Comput."},{"key":"ref_41","unstructured":"Opportunity Dataset. Available online: https:\/\/archive.ics.uci.edu\/ml\/datasets\/OPPORTUNITY+Activity+Recognition."},{"key":"ref_42","unstructured":"Skoda Dataset. Available online: http:\/\/www.ife.ee.ethz.ch\/research\/groups\/Dataset."},{"key":"ref_43","unstructured":"Alsheikh, M.A., Selim, A., Niyato, D., Doyle, L., Lin, S., and Tan, H.P. (2015). Deep Activity Recognition Models with Triaxial Accelerometers. arXiv preprint, arXiv:1511.04664."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"429","DOI":"10.3233\/IDA-2002-6504","article-title":"The class imbalance problem: A systematic study","volume":"6","author":"Japkowicz","year":"2002","journal-title":"Intell. Data Anal."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going Deeper With Convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Berchtold, M., Budde, M., Gordon, D., Schmidtke, H.R., and Beigl, M. (2010, January 10\u201313). Actiserv: Activity recognition service for mobile phones. Proceedings of the International Symposium on Wearable Computers (ISWC), Seoul, Korea.","DOI":"10.1109\/ISWC.2010.5665868"},{"key":"ref_47","unstructured":"Cheng, K.T., and Wang, Y.C. (2011, January 25\u201328). Using mobile GPU for general-purpose computing: A case study of face recognition on smartphones. Proceedings of the International Symposium on VLSI Design, Automation and Test (VLSI-DAT), Hsinchu, Taiwan."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Welbourne, E., and Tapia, E.M. (2014, January 13\u201317). CrowdSignals: A call to crowdfund the community\u2019s largest mobile dataset. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, Seattle, WA, USA.","DOI":"10.1145\/2638728.2641309"},{"key":"ref_49","unstructured":"Ordonez, F.J., and Roggen, D. DeepConvLSTM. Available online: https:\/\/github.com\/sussexwearlab\/DeepConvLSTM."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/16\/1\/115\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T19:17:52Z","timestamp":1760210272000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/16\/1\/115"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,1,18]]},"references-count":49,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2016,1]]}},"alternative-id":["s16010115"],"URL":"https:\/\/doi.org\/10.3390\/s16010115","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,1,18]]}}}