{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T00:19:10Z","timestamp":1790641150575,"version":"4.1.0"},"reference-count":43,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2021,9,30]],"date-time":"2021-09-30T00:00:00Z","timestamp":1632960000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"European Union\u2019s Horizon 2020 research and innovation programme under the Marie Sk\u0142odowska-Curie","award":["813162"],"award-info":[{"award-number":["813162"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Wearable sensors are widely used in activity recognition (AR) tasks with broad applicability in health and well-being, sports, geriatric care, etc. Deep learning (DL) has been at the forefront of progress in activity classification with wearable sensors. However, most state-of-the-art DL models used for AR are trained to discriminate different activity classes at high accuracy, not considering the confidence calibration of predictive output of those models. This results in probabilistic estimates that might not capture the true likelihood and is thus unreliable. In practice, it tends to produce overconfident estimates. In this paper, the problem is addressed by proposing deep time ensembles, a novel ensembling method capable of producing calibrated confidence estimates from neural network architectures. In particular, the method trains an ensemble of network models with temporal sequences extracted by varying the window size over the input time series and averaging the predictive output. The method is evaluated on four different benchmark HAR datasets and three different neural network architectures. Across all the datasets and architectures, our method shows an improvement in calibration by reducing the expected calibration error (ECE)by at least 40%, thereby providing superior likelihood estimates. In addition to providing reliable predictions our method also outperforms the state-of-the-art classification results in the WISDM, UCI HAR, and PAMAP2 datasets and performs as good as the state-of-the-art in the Skoda dataset.<\/jats:p>","DOI":"10.3390\/s21196566","type":"journal-article","created":{"date-parts":[[2021,10,10]],"date-time":"2021-10-10T21:37:49Z","timestamp":1633901869000},"page":"6566","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Confidence-Calibrated Human Activity Recognition"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6780-7755","authenticated-orcid":false,"given":"Debaditya","family":"Roy","sequence":"first","affiliation":[{"name":"School of Electrical Engineering and Computer Science (EECS), KTH Royal Institute of Technology, 114 28 Stockholm, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4516-7317","authenticated-orcid":false,"given":"Sarunas","family":"Girdzijauskas","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Computer Science (EECS), KTH Royal Institute of Technology, 114 28 Stockholm, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Serghei","family":"Socolovschi","sequence":"additional","affiliation":[{"name":"School of Electrical Engineering and Computer Science (EECS), KTH Royal Institute of Technology, 114 28 Stockholm, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,30]]},"reference":[{"key":"ref_1","unstructured":"Rassem, A., El-Beltagy, M., and Saleh, M. (2017). Cross-country skiing gears classification using deep learning. arXiv."},{"key":"ref_2","unstructured":"Avci, A., Bosch, S., Marin-Perianu, M., Marin-Perianu, R., and Havinga, P. (2010, January 22\u201323). Activity recognition using inertial sensing for healthcare, wellbeing and sports applications: A survey. Proceedings of the 23th International Conference on Architecture of Computing Systems 2010, VDE, Hannover, Germany."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Zhang, M., and Sawchuk, A.A. (2012, January 5\u20138). USC-HAD: A daily activity dataset for ubiquitous activity recognition using wearable sensors. Proceedings of the 2012 ACM Conference on Ubiquitous Computing, Pittsburgh, PA, USA.","DOI":"10.1145\/2370216.2370438"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Hong, Y.J., Kim, I.J., Ahn, S.C., and Kim, H.G. (2008, January 13\u201315). Activity recognition using wearable sensors for elder care. Proceedings of the 2008 Second International Conference on Future Generation Communication and Networking, Hainan, China.","DOI":"10.1109\/FGCN.2008.165"},{"key":"ref_5","unstructured":"Gal, Y., and Ghahramani, Z. (2016, January 20\u201322). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the International Conference on Machine Learning, New York, NY, USA."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3090076","article-title":"Ensembles of deep lstm learners for activity recognition using wearables","volume":"1","author":"Guan","year":"2017","journal-title":"ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_7","unstructured":"Kumar, A., Liang, P., and Ma, T. (2019). Verified uncertainty calibration. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Dietterich, T.G. (2000). Ensemble methods in machine learning. Lecture Notes in Computer Science, Proceedings of the International Workshop on Multiple Classifier Systems, Cagliari, Italy, 21\u201323 June 2000, Springer.","DOI":"10.1007\/3-540-45014-9_1"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Seo, S., Seo, P.H., and Han, B. (2019, January 16\u201320). Learning for single-shot confidence calibration in deep neural networks through stochastic inferences. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00924"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"133190","DOI":"10.1109\/ACCESS.2019.2940729","article-title":"Smartphone and smartwatch-based biometrics using activities of daily living","volume":"7","author":"Weiss","year":"2019","journal-title":"IEEE Access"},{"key":"ref_11","first-page":"3","article-title":"A public domain dataset for human activity recognition using smartphones","volume":"3","author":"Anguita","year":"2013","journal-title":"Esann"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Reiss, A., and Stricker, D. (2012, January 18\u201322). Introducing a new benchmarked dataset for activity monitoring. Proceedings of the 2012 16th International Symposium on Wearable Computers, Newcastle, UK.","DOI":"10.1109\/ISWC.2012.13"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/MPRV.2008.40","article-title":"Wearable activity tracking in car manufacturing","volume":"7","author":"Stiefmeier","year":"2008","journal-title":"IEEE Pervasive Comput."},{"key":"ref_14","unstructured":"Guo, C., Pleiss, G., Sun, Y., and Weinberger, K.Q. (2017, January 6\u201311). On calibration of modern neural networks. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Bao, L., and Intille, S.S. (2004). Activity recognition from user-annotated acceleration data. Lecture Notes in Computer Science, Proceedings of the International Conference on Pervasive Computing, Linz\/Vienna, Austria, 21\u201323 April 2004, Springer.","DOI":"10.1007\/978-3-540-24646-6_1"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Casale, P., Pujol, O., and Radeva, P. (2011). Human activity recognition from accelerometer data using a wearable device. Lecture Notes in Computer Science, Proceedings of the Iberian Conference on Pattern Recognition and Image Analysis, Las Palmas de Gran Canaria, Spain, 8\u201310 June 2011, Springer.","DOI":"10.1007\/978-3-642-21257-4_36"},{"key":"ref_17","unstructured":"Feng, Z., Mo, L., and Li, M. (2015, January 25\u201329). A Random Forest-based ensemble method for activity recognition. Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"He, Z., and Jin, L. (2009, January 11\u201314). Activity recognition from acceleration data based on discrete consine transform and SVM. Proceedings of the 2009 IEEE International Conference on Systems, Man and Cybernetics, San Antonio, TX, USA.","DOI":"10.1109\/ICSMC.2009.5346042"},{"key":"ref_19","unstructured":"He, Z.Y., and Jin, L.W. (2008, January 12\u201315). Activity recognition from acceleration data using AR model representation and SVM. Proceedings of the 2008 International Conference on Machine Learning and Cybernetics, Kunming, China."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"915","DOI":"10.1016\/j.asoc.2017.09.027","article-title":"Real-time human activity recognition from accelerometer data using Convolutional Neural Networks","volume":"62","author":"Ignatov","year":"2018","journal-title":"Appl. Soft Comput."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ord\u00f3\u00f1ez, F.J., and Roggen, D. (2016). Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors, 16.","DOI":"10.3390\/s16010115"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Yao, S., Hu, S., Zhao, Y., Zhang, A., and Abdelzaher, T. (2017, January 3\u20137). Deepsense: A unified deep learning framework for time-series mobile sensing data processing. Proceedings of the 26th International Conference on World Wide Web, Perth, Australia.","DOI":"10.1145\/3038912.3052577"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Ishimaru, S., Hoshika, K., Kunze, K., Kise, K., and Dengel, A. (2017, January 11\u201315). Towards reading trackers in the wild: Detecting reading activities by EOG glasses and deep neural networks. Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers, Maui, HI, USA.","DOI":"10.1145\/3123024.3129271"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Jiang, W., and Yin, Z. (2015, January 26\u201330). Human activity recognition using wearable sensors by deep convolutional neural networks. Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia.","DOI":"10.1145\/2733373.2806333"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"Lecun","year":"1998","journal-title":"Proc. IEEE"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long Short-Term Memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Murad, A., and Pyun, J.Y. (2017). Deep recurrent neural networks for human activity recognition. Sensors, 17.","DOI":"10.3390\/s17112556"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Wang, L. (2016). Recognition of human activities using continuous autoencoders with wearable sensors. Sensors, 16.","DOI":"10.3390\/s16020189"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"2364","DOI":"10.1016\/j.procs.2020.03.289","article-title":"A lightweight deep learning model for human activity recognition on edge devices","volume":"167","author":"Agarwal","year":"2020","journal-title":"Procedia Comput. Sci."},{"key":"ref_30","unstructured":"Hammerla, N.Y., Halloran, S., and Pl\u00f6tz, T. (2016). Deep, convolutional, and recurrent models for human activity recognition using wearables. arXiv."},{"key":"ref_31","first-page":"61","article-title":"Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods","volume":"10","author":"Platt","year":"1999","journal-title":"Adv. Large Margin Classif."},{"key":"ref_32","first-page":"609","article-title":"Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers","volume":"1","author":"Zadrozny","year":"2001","journal-title":"ICML Citeseer"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Zadrozny, B., and Elkan, C. (2002, January 23\u201326). Transforming classifier scores into accurate multiclass probability estimates. Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Edmonton, AB, Canada.","DOI":"10.1145\/775047.775151"},{"key":"ref_34","unstructured":"Naeini, M.P., Cooper, G., and Hauskrecht, M. (2015, January 25\u201330). Obtaining well calibrated probabilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence, Austin, TX, USA."},{"key":"ref_35","unstructured":"Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017, January 4\u20139). Simple and scalable predictive uncertainty estimation using deep ensembles. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_36","unstructured":"Song, Y., Viventi, J., and Wang, Y. (2016). Diversity encouraged learning of unsupervised LSTM ensemble for neural activity video prediction. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Beluch, W.H., Genewein, T., N\u00fcrnberger, A., and K\u00f6hler, J.M. (2018, January 18\u201322). The power of ensembles for active learning in image classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00976"},{"key":"ref_38","first-page":"12","article-title":"The comparison and evaluation of forecasters","volume":"32","author":"DeGroot","year":"1983","journal-title":"J. R. Stat. Soc. Ser. D (The Stat.)"},{"key":"ref_39","unstructured":"(2021, April 30). Reliability Diagrams. Available online: https:\/\/github.com\/hollance\/reliability-diagrams."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1088\/0954-898X_8_3_004","article-title":"Optimal ensemble averaging of neural networks","volume":"8","author":"Naftaly","year":"1997","journal-title":"Netw. Comput. Neural Syst."},{"key":"ref_41","unstructured":"Malinin, A., Mlodozeniec, B., and Gales, M. (2019). Ensemble distribution distillation. arXiv."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 11\u201318). Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.510"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., and Paluri, M. (2018, January 18\u201322). A closer look at spatiotemporal convolutions for action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00675"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/19\/6566\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:08:14Z","timestamp":1760166494000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/19\/6566"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,30]]},"references-count":43,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2021,10]]}},"alternative-id":["s21196566"],"URL":"https:\/\/doi.org\/10.3390\/s21196566","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,30]]}}}