{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T06:50:33Z","timestamp":1778309433546,"version":"3.51.4"},"reference-count":64,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2022,3,18]],"date-time":"2022-03-18T00:00:00Z","timestamp":1647561600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Samsung Electronics of Amazonia Ltda","award":["003\/2019"],"award-info":[{"award-number":["003\/2019"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In this article, we introduce explainable methods to understand how Human Activity Recognition (HAR) mobile systems perform based on the chosen validation strategies. Our results introduce a new way to discover potential bias problems that overestimate the prediction accuracy of an algorithm because of the inappropriate choice of validation methodology. We show how the SHAP (Shapley additive explanations) framework, used in literature to explain the predictions of any machine learning model, presents itself as a tool that can provide graphical insights into how human activity recognition models achieve their results. Now it is possible to analyze which features are important to a HAR system in each validation methodology in a simplified way. We not only demonstrate that the validation procedure k-folds cross-validation (k-CV), used in most works to evaluate the expected error in a HAR system, can overestimate by about 13% the prediction accuracy in three public datasets but also choose a different feature set when compared with the universal model. Combining explainable methods with machine learning algorithms has the potential to help new researchers look inside the decisions of the machine learning algorithms, avoiding most times the overestimation of prediction accuracy, understanding relations between features, and finding bias before deploying the system in real-world scenarios.<\/jats:p>","DOI":"10.3390\/s22062360","type":"journal-article","created":{"date-parts":[[2022,3,20]],"date-time":"2022-03-20T21:37:17Z","timestamp":1647812237000},"page":"2360","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":42,"title":["How Validation Methodology Influences Human Activity Recognition Mobile Systems"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1479-1707","authenticated-orcid":false,"given":"Hendrio","family":"Bragan\u00e7a","sequence":"first","affiliation":[{"name":"Institute of Computing, Federal University of Amazonas, Manaus 69067-005, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1740-2618","authenticated-orcid":false,"given":"Juan G.","family":"Colonna","sequence":"additional","affiliation":[{"name":"Institute of Computing, Federal University of Amazonas, Manaus 69067-005, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9777-3947","authenticated-orcid":false,"given":"Hor\u00e1cio A. B. F.","family":"Oliveira","sequence":"additional","affiliation":[{"name":"Institute of Computing, Federal University of Amazonas, Manaus 69067-005, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0003-908X","authenticated-orcid":false,"given":"Eduardo","family":"Souto","sequence":"additional","affiliation":[{"name":"Institute of Computing, Federal University of Amazonas, Manaus 69067-005, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1109\/SURV.2012.110112.00192","article-title":"A survey on human activity recognition using wearable sensors","volume":"15","author":"Lara","year":"2012","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Shoaib, M., Scholten, H., and Havinga, P.J. (2013, January 8\u201321). Towards physical activity recognition using smartphone sensors. Proceedings of the 2013 IEEE 10th International Conference on Ubiquitous Intelligence and Computing and 2013 IEEE 10th International Conference on Autonomic and Trusted Computing, Vietri sul Mere, Italy.","DOI":"10.1109\/UIC-ATC.2013.43"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Lathia, N., Sandstrom, G.M., Mascolo, C., and Rentfrow, P.J. (2017). Happier people live more active lives: Using smartphones to link happiness and physical activity. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0160589"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Singh, D., Merdivan, E., Psychoula, I., Kropf, J., Hanke, S., Geist, M., and Holzinger, A. (2017, January 15). Human activity recognition using recurrent neural networks. Proceedings of the International Cross-Domain Conference for Machine Learning and Knowledge Extraction, Reggio, Italy.","DOI":"10.1007\/978-3-319-66808-6_18"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"3652","DOI":"10.1109\/JIOT.2018.2889966","article-title":"IoT structured long-term wearable social sensing for mental wellbeing","volume":"6","author":"Yang","year":"2018","journal-title":"IEEE Internet Things J."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1016\/j.eswa.2018.03.056","article-title":"Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges","volume":"105","author":"Nweke","year":"2018","journal-title":"Expert Syst. Appl."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1007\/s40860-021-00147-0","article-title":"Trends in human activity recognition using smartphones","volume":"7","author":"Ferrari","year":"2021","journal-title":"J. Reliab. Intell. Environ."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ferrari, A., Micucci, D., Mobilio, M., and Napoletano, P. (2022). Deep learning and model personalization in sensor-based human activity recognition. J. Reliab. Intell. Environ., 1\u201313.","DOI":"10.1007\/s40860-021-00167-w"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-021-95947-y","article-title":"Human activity recognition using wearable sensors, discriminant analysis, and long short-term memory-based neural structured learning","volume":"11","author":"Uddin","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"838","DOI":"10.1177\/1745691616650285","article-title":"Using smartphones to collect behavioral data in psychological science: Opportunities, practical considerations, and challenges","volume":"11","author":"Harari","year":"2016","journal-title":"Perspect. Psychol. Sci."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Bragan\u00e7a, H., Colonna, J.G., Lima, W.S., and Souto, E. (2020). A Smartphone Lightweight Method for Human Activity Recognition Based on Information Theory. Sensors, 20.","DOI":"10.3390\/s20071856"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"645","DOI":"10.1007\/s00779-010-0293-9","article-title":"Preprocessing techniques for context recognition from accelerometer data","volume":"14","author":"Figo","year":"2010","journal-title":"Pers. Ubiquitous Comput."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2059","DOI":"10.3390\/s150102059","article-title":"A survey of online activity recognition using mobile phones","volume":"15","author":"Shoaib","year":"2015","journal-title":"Sensors"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Li, F., Shirahama, K., Nisar, M.A., K\u00f6ping, L., and Grzegorzek, M. (2018). Comparison of feature learning methods for human activity recognition using wearable sensors. Sensors, 18.","DOI":"10.3390\/s18020679"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2499621","article-title":"A tutorial on human activity recognition using body-worn inertial sensors","volume":"46","author":"Bulling","year":"2014","journal-title":"Acm Comput. Surv. (CSUR)"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1214\/09-SS054","article-title":"A survey of cross-validation procedures for model selection","volume":"4","author":"Arlot","year":"2010","journal-title":"Stat. Surv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Varma, S., and Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinform., 7.","DOI":"10.1186\/1471-2105-7-91"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2382577.2382579","article-title":"Leakage in data mining: Formulation, detection, and avoidance","volume":"6","author":"Kaufman","year":"2012","journal-title":"Acm Trans. Knowl. Discov. Data (Tkdd)"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Colonna, J.G., Gama, J., and Nakamura, E.F. (2016, January 14\u201316). How to correctly evaluate an automatic bioacoustics classification method. Proceedings of the 17th Conference of the Spanish Association for Artificial Intelligence, Salamanca, Spain.","DOI":"10.1007\/978-3-319-44636-3_4"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"gix019","DOI":"10.1093\/gigascience\/gix019","article-title":"The need to approximate the use-case in clinical machine learning","volume":"6","author":"Saeb","year":"2017","journal-title":"GigaScience"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1093\/gigascience\/gix020","article-title":"Using and understanding cross-validation strategies. Perspectives on Saeb et al","volume":"6","author":"Little","year":"2017","journal-title":"GigaScience"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Widhalm, P., Leodolter, M., and Br\u00e4ndle, N. (2018, January 8\u201312). Top in the lab, flop in the field? Evaluation of a sensor-based travel activity classifier with the SHL dataset. Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, Singapore.","DOI":"10.1145\/3267305.3267514"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Bissoto, A., Fornaciali, M., Valle, E., and Avila, S. (2019, January 16\u201320). (De) Constructing Bias on Skin Lesion Datasets. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00335"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2053951717743530","DOI":"10.1177\/2053951717743530","article-title":"Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data","volume":"4","author":"Veale","year":"2017","journal-title":"Big Data Soc."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"74","DOI":"10.1145\/1964897.1964918","article-title":"Activity recognition using cell phone accelerometers","volume":"12","author":"Kwapisz","year":"2011","journal-title":"ACM Sigkdd Explor. Newsl."},{"key":"ref_26","unstructured":"Anguita, D., Ghio, A., Oneto, L., Parra, X., and Reyes-Ortiz, J.L. (2013, January 24\u201326). A Public Domain Dataset for Human Activity Recognition Using Smartphones. Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, Belgium."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Lima, W.S., Bragan\u00e7a, H.L.S., Quispe, K.G.M., and Souto, J.P. (2018). Human Activity Recognition based on Symbolic Representation Algorithms for Inertial Sensors. Sensors, 18.","DOI":"10.3390\/s18114045"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.patrec.2018.02.010","article-title":"Deep learning for sensor-based activity recognition: A survey","volume":"119","author":"Wang","year":"2019","journal-title":"Pattern Recognit. Lett."},{"key":"ref_29","unstructured":"Dehghani, A., Glatard, T., and Shihab, E. (2019). Subject cross validation in human activity recognition. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"107561","DOI":"10.1016\/j.patcog.2020.107561","article-title":"Sensor-based and vision-based human activity recognition: A comprehensive survey","volume":"108","author":"Dang","year":"2020","journal-title":"Pattern Recognit."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Liu, H., Hartmann, Y., and Schultz, T. (2022, January 9\u201311). A practical wearable sensor-based human activity recognition research pipeline. Proceedings of the 5th International Conference on Health Informatics (HEALTHINF 2022), Vienna, Austria.","DOI":"10.5220\/0010937000003123"},{"key":"ref_32","first-page":"1","article-title":"Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities","volume":"54","author":"Chen","year":"2021","journal-title":"ACM Comput. Surv. (CSUR)"},{"key":"ref_33","unstructured":"Das, D., Nishimura, Y., Vivek, R.P., Takeda, N., Fish, S.T., Ploetz, T., and Chernova, S. (2021). Explainable Activity Recognition for Smart Home Systems. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Bao, L., and Intille, S.S. (2004, January 14\u201317). Activity recognition from user-annotated acceleration data. Proceedings of the International Conference on Pervasive Computing, Orlando, FL, USA.","DOI":"10.1007\/978-3-540-24646-6_1"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"10146","DOI":"10.3390\/s140610146","article-title":"Fusion of smartphone motion sensors for physical activity recognition","volume":"14","author":"Shoaib","year":"2014","journal-title":"Sensors"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Ravi, D., Wong, C., Lo, B., and Yang, G.Z. (2016, January 14\u201317). Deep learning for human activity recognition: A resource efficient implementation on low-power devices. Proceedings of the 2016 IEEE 13th International Conference on Wearable and Implantable Body Sensor Networks (BSN), San Francisco, CA, USA.","DOI":"10.1109\/BSN.2016.7516235"},{"key":"ref_37","unstructured":"Duda, R.O., Hart, P.E., and Stork, D.G. (2000). Pattern Classification, John Wiley & Sons."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1016\/j.eswa.2016.04.032","article-title":"Human activity recognition with smartphone sensors using deep learning neural networks","volume":"59","author":"Ronao","year":"2016","journal-title":"Expert Syst. Appl."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"915","DOI":"10.1016\/j.asoc.2017.09.027","article-title":"Real-time human activity recognition from accelerometer data using Convolutional Neural Networks","volume":"62","author":"Ignatov","year":"2018","journal-title":"Appl. Soft Comput."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"2839","DOI":"10.1016\/j.patcog.2015.03.009","article-title":"Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation","volume":"48","author":"Wong","year":"2015","journal-title":"Pattern Recognit."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Sousa Lima, W., Souto, E., El-Khatib, K., Jalali, R., and Gama, J. (2019). Human activity recognition using inertial sensors in a smartphone: An overview. Sensors, 19.","DOI":"10.3390\/s19143213"},{"key":"ref_42","first-page":"1137","article-title":"A study of cross-validation and bootstrap for accuracy estimation and model selection","volume":"14","author":"Kohavi","year":"1995","journal-title":"Ijcai. Montr. Can."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Hammerla, N.Y., and Pl\u00f6tz, T. (2015, January 7\u201311). Let\u2019s (not) stick together: Pairwise similarity biases cross-validation in activity recognition. Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Osaka, Japan.","DOI":"10.1145\/2750858.2807551"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"6474","DOI":"10.3390\/s140406474","article-title":"Window size impact in human activity recognition","volume":"14","author":"Banos","year":"2014","journal-title":"Sensors"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"133982","DOI":"10.1109\/ACCESS.2020.3010715","article-title":"Deep neural networks for human activity recognition with wearable sensors: Leave-one-subject-out cross-validation for model selection","volume":"8","author":"Gholamiangonabadi","year":"2020","journal-title":"IEEE Access"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Bettini, C., Civitarese, G., and Fiori, M. (2021, January 22\u201326). Explainable Activity Recognition over Interpretable Models. Proceedings of the 2021 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops), Kassel, Germany.","DOI":"10.1109\/PerComWorkshops51409.2021.9430955"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"e59","DOI":"10.1002\/ail2.59","article-title":"Explainable activity recognition in videos: Lessons learned","volume":"2","author":"Roy","year":"2021","journal-title":"Appl. AI Lett."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"388","DOI":"10.1016\/j.bbe.2017.04.004","article-title":"Physical activity recognition by smartphones, a survey","volume":"37","author":"Morales","year":"2017","journal-title":"Biocybern. Biomed. Eng."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Lockhart, J.W., and Weiss, G.M. (2014, January 13\u201317). Limitations with activity recognition methodology & data sets. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Seattle, WA, USA.","DOI":"10.1145\/2638728.2641306"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"32066","DOI":"10.1109\/ACCESS.2020.2973425","article-title":"On the personalization of classification models for human activity recognition","volume":"8","author":"Ferrari","year":"2020","journal-title":"IEEE Access"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"1659","DOI":"10.1016\/j.patrec.2011.06.025","article-title":"Improving the classification accuracy of streaming data using SAX similarity features","volume":"32","author":"Siirtola","year":"2011","journal-title":"Pattern Recognit. Lett."},{"key":"ref_52","unstructured":"Weiss, G.M., and Lockhart, J. (2012, January 22\u201326). The impact of personalization on smartphone-based activity recognition. Proceedings of the Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence, Toronto, ON, Canada."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2021). Explainable ai: A review of machine learning interpretability methods. Entropy, 23.","DOI":"10.3390\/e23010018"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Yin, M., Wortman Vaughan, J., and Wallach, H. (2019, January 4\u20139). Understanding the effect of accuracy on trust in machine learning models. Proceedings of the 2019 chi Conference on Human Factors in Computing Systems, Glasgow, UK.","DOI":"10.1145\/3290605.3300509"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Toreini, E., Aitken, M., Coopamootoo, K., Elliott, K., Zelaya, C.G., and Van Moorsel, A. (2020, January 27\u201330). The relationship between trust in AI and trustworthy machine learning technologies. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, Barcelona, Spain.","DOI":"10.1145\/3351095.3372834"},{"key":"ref_56","unstructured":"Alikhademi, K., Richardson, B., Drobina, E., and Gilbert, J.E. (2021). Can Explainable AI Explain Unfairness? A Framework for Evaluating Explainable AI. arXiv."},{"key":"ref_57","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016). Model-agnostic interpretability of machine learning. arXiv."},{"key":"ref_58","unstructured":"Shrikumar, A., Greenside, P., and Kundaje, A. (2017, January 6\u201311). Learning important features through propagating activation differences. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia."},{"key":"ref_59","unstructured":"Lundberg, S.M., and Lee, S.I. (2017, January 4\u20139). A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_60","unstructured":"Molnar, C. (2020). Interpretable Machine Learning, Lulu."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Ibrahim, M., Louie, M., Modarres, C., and Paisley, J. (2019, January 27\u201328). Global explanations of neural networks: Mapping the landscape of predictions. Proceedings of the 2019 AAAI\/ACM Conference on AI, Ethics, and Society, Honolulu, HI, USA.","DOI":"10.1145\/3306618.3314230"},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"2522","DOI":"10.1038\/s42256-019-0138-9","article-title":"From local explanations to global understanding with explainable AI for trees","volume":"2","author":"Lundberg","year":"2020","journal-title":"Nat. Mach. Intell."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1145\/507338.507355","article-title":"Data mining: Practical machine learning tools and techniques with Java implementations","volume":"31","author":"Witten","year":"2002","journal-title":"Acm Sigmod Rec."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"749","DOI":"10.1038\/s41551-018-0304-0","article-title":"Explainable machine-learning predictions for the prevention of hypoxaemia during surgery","volume":"2","author":"Lundberg","year":"2018","journal-title":"Nat. Biomed. Eng."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/6\/2360\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:38:57Z","timestamp":1760135937000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/6\/2360"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,18]]},"references-count":64,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["s22062360"],"URL":"https:\/\/doi.org\/10.3390\/s22062360","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,18]]}}}