{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,30]],"date-time":"2025-12-30T17:46:23Z","timestamp":1767116783876,"version":"build-2065373602"},"reference-count":58,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2022,7,13]],"date-time":"2022-07-13T00:00:00Z","timestamp":1657670400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Workplace Safety and Insurance Board of Ontario","award":["RICH2018","CHRP-538866"],"award-info":[{"award-number":["RICH2018","CHRP-538866"]}]},{"name":"Natural Sciences and Engineering Research Council of Canada and Canadian Institutes of Health Research","award":["RICH2018","CHRP-538866"],"award-info":[{"award-number":["RICH2018","CHRP-538866"]}]},{"name":"Susanne and William Holland Surgeon Scientist Award","award":["RICH2018","CHRP-538866"],"award-info":[{"award-number":["RICH2018","CHRP-538866"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>A significant challenge for a supervised learning approach to inertial human activity recognition is the heterogeneity of data generated by individual users, resulting in very poor performance for some subjects. We present an approach to personalized activity recognition based on deep feature representation derived from a convolutional neural network (CNN). We experiment with both categorical cross-entropy loss and triplet loss for training, and describe a novel loss function based on subject triplets. We evaluate these methods on three publicly available inertial human activity recognition datasets (MHEALTH, WISDM, and SPAR) comparing classification accuracy, out-of-distribution activity detection, and generalization to new activity classes. The proposed triplet algorithm achieved an average 96.7% classification accuracy across tested datasets versus the 87.5% achieved by the baseline CNN algorithm. We demonstrate that personalized algorithms, and, in particular, the proposed novel triplet loss algorithms, are more robust to inter-subject variability and thus exhibit better performance on classification and out-of-distribution detection tasks.<\/jats:p>","DOI":"10.3390\/s22145222","type":"journal-article","created":{"date-parts":[[2022,7,14]],"date-time":"2022-07-14T00:12:40Z","timestamp":1657757560000},"page":"5222","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":19,"title":["Personalized Activity Recognition with Deep Triplet Embeddings"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1617-596X","authenticated-orcid":false,"given":"David","family":"Burns","sequence":"first","affiliation":[{"name":"Orthopaedic Biomechanics Laboratory, Holland Bone and Joint Program, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada"},{"name":"Division of Orthopaedic Surgery, Department of Surgery, University of Toronto, Toronto, ON M5S 2E8, Canada"},{"name":"Halterix Corporation, Toronto, ON M5E 1L4, Canada"},{"name":"Institute of Biomedical Engineering, University of Toronto, Toronto, ON M5S 2E8, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9768-8037","authenticated-orcid":false,"given":"Philip","family":"Boyer","sequence":"additional","affiliation":[{"name":"Orthopaedic Biomechanics Laboratory, Holland Bone and Joint Program, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada"},{"name":"Institute of Biomedical Engineering, University of Toronto, Toronto, ON M5S 2E8, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1938-0210","authenticated-orcid":false,"given":"Colin","family":"Arrowsmith","sequence":"additional","affiliation":[{"name":"Orthopaedic Biomechanics Laboratory, Holland Bone and Joint Program, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada"},{"name":"Halterix Corporation, Toronto, ON M5E 1L4, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6822-8314","authenticated-orcid":false,"given":"Cari","family":"Whyne","sequence":"additional","affiliation":[{"name":"Orthopaedic Biomechanics Laboratory, Holland Bone and Joint Program, Sunnybrook Research Institute, Toronto, ON M4N 3M5, Canada"},{"name":"Division of Orthopaedic Surgery, Department of Surgery, University of Toronto, Toronto, ON M5S 2E8, Canada"},{"name":"Institute of Biomedical Engineering, University of Toronto, Toronto, ON M5S 2E8, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,7,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Sousa Lima, W., Souto, E., El-Khatib, K., Jalali, R., and Gama, J. (2019). Human Activity Recognition Using Inertial Sensors in a Smartphone: An Overview. Sensors, 19.","DOI":"10.3390\/s19143213"},{"key":"ref_2","first-page":"13","article-title":"Convergence Revolution Comes to Wearables: Multiple Advances are Taking Biosensor Networks to the Next Level in Health Care","volume":"7","author":"Mertz","year":"2016","journal-title":"IEEE Pulse"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1109\/MPUL.2016.2592260","article-title":"Wearables and the Internet of Things for Health: Wearable, Interconnected Devices Promise More Efficient and Comprehensive Health Care","volume":"7","author":"Metcalf","year":"2016","journal-title":"IEEE Pulse"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Piwek, L., Ellis, D.A., Andrews, S., and Joinson, A. (2016). The Rise of Consumer Health Wearables: Promises and Barriers. PLoS Med., 13.","DOI":"10.1371\/journal.pmed.1001953"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"465","DOI":"10.1109\/JBHI.2020.2988360","article-title":"Sleeve for Knee Angle Monitoring: An IMU-POF Sensor Fusion System","volume":"25","author":"Schneider","year":"2021","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1221","DOI":"10.1007\/s40279-018-0878-4","article-title":"Wearable Inertial Sensor Systems for Lower Limb Exercise Detection and Evaluation: A Systematic Review","volume":"48","author":"Caulfield","year":"2018","journal-title":"Sport. Med."},{"key":"ref_7","first-page":"S220","article-title":"Wearable Movement Sensors for Rehabilitation: A Focused Review of Technological and Clinical Advances","volume":"10","author":"Porciuncula","year":"2018","journal-title":"PMR"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"e21374","DOI":"10.2196\/21374","article-title":"Adherence Patterns and Dose Response of Physiotherapy for Rotator Cuff Pathology: Longitudinal Cohort Study","volume":"8","author":"Burns","year":"2021","journal-title":"JMIR Rehabil. Assist. Technol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"e88","DOI":"10.2196\/mhealth.7870","article-title":"Mobile Device Accuracy for Step Counting Across Age Groups","volume":"5","author":"Modave","year":"2017","journal-title":"JMIR mHealth uHealth"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/j.apergo.2017.04.011","article-title":"Effect of local magnetic field disturbances on inertial measurement units accuracy","volume":"63","author":"Mecheri","year":"2017","journal-title":"Appl. Ergon."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Schroff, F., Kalenichenko, D., and Philbin, J. (2015, January 7\u201312). FaceNet: A Unified Embedding for Face Recognition and Clustering. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"ref_12","unstructured":"Hoffer, E., and Ailon, N. (2018). Deep metric learning using Triplet network. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1192","DOI":"10.1109\/SURV.2012.110112.00192","article-title":"A Survey on Human Activity Recognition using Wearable Sensors","volume":"15","author":"Lara","year":"2013","journal-title":"IEEE Commun. Surv. Tut."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2499621","article-title":"A tutorial on human activity recognition using body-worn inertial sensors","volume":"46","author":"Bulling","year":"2014","journal-title":"ACM Comput. Surv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.neucom.2015.01.082","article-title":"Features and models for human activity recognition","volume":"167","author":"Gonzalez","year":"2015","journal-title":"Neurocomputing"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1016\/j.sigpro.2015.09.029","article-title":"Feature extraction from smartphone inertial signals for human activity segmentation","volume":"120","author":"Montero","year":"2016","journal-title":"Signal Process."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wang, Z., Yan, W., and Oates, T. (2016). Time Series Classification from Scratch with Deep Neural Networks: A Strong Baseline. arXiv.","DOI":"10.1109\/IJCNN.2017.7966039"},{"key":"ref_18","first-page":"11:1","article-title":"Ensembles of Deep LSTM Learners for Activity Recognition using Wearables","volume":"1","author":"Guan","year":"2017","journal-title":"Proc. ACM Interact. Mobile Wear. Ubiquit. Technol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.patrec.2018.02.010","article-title":"Deep learning for sensor-based activity recognition: A survey","volume":"119","author":"Wang","year":"2019","journal-title":"Pattern Recognit. Lett."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"6703","DOI":"10.1109\/TII.2020.2968920","article-title":"PSDRNN: An Efficient and Effective HAR Scheme Based on Feature Extraction and Deep Learning","volume":"16","author":"Li","year":"2020","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1461","DOI":"10.1007\/s00607-021-00928-8","article-title":"Multi-input CNN-GRU based human activity recognition using wearable sensors","volume":"103","author":"Dua","year":"2021","journal-title":"Computing"},{"key":"ref_22","unstructured":"Weiss, G.M., and Lockhart, J.W. (2012, January 22\u201326). The Impact of Personalization on Smartphone-Based Activity Recognition. Proceedings of the AAAI 2012, Toronto, ON, USA."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Sztyler, T., and Stuckenschmidt, H. (2017, January 13\u201317). Online personalization of cross-subjects based activity recognition models on wearable devices. Proceedings of the 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom), Kona, HI, USA.","DOI":"10.1109\/PERCOM.2017.7917864"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Zhao, S., Li, W., and Cao, J. (2018). A User-Adaptive Algorithm for Activity Recognition Based on K-Means Clustering, Local Outlier Factor, and Multivariate Gaussian Distribution. Sensors, 18.","DOI":"10.3390\/s18061850"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"10779","DOI":"10.1007\/s11042-016-3267-8","article-title":"Towards online and personalized daily activity recognition, habit modeling, and anomaly detection for the solitary elderly through unobtrusive sensing","volume":"76","author":"Meng","year":"2017","journal-title":"Multimed. Tools Appl."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Rokni, S.A., Nourollahi, M., and Ghasemzadeh, H. (2018). Personalized Human Activity Recognition Using Convolutional Neural Networks. arXiv.","DOI":"10.1609\/aaai.v32i1.12185"},{"key":"ref_27","unstructured":"Cvetkovic, B., Lustrek, M., Kaluza, B., and Gams, M. (2011, January 16). Semi-supervised Learning for Adaptation of Human Activity Recognition Classifier to the User. Proceedings of the STAMI 2011: Space, Time and Ambient Intelligence, Barcelona, Spain."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1109\/THMS.2015.2489688","article-title":"Toward Personalized Activity Recognition Systems With a Semipopulation Approach","volume":"46","author":"Hong","year":"2016","journal-title":"IEEE Trans.-Hum.-Mach. Syst."},{"key":"ref_29","unstructured":"Bettini, C., Civitarese, G., and Presotto, R. (2021). Personalized Semi-Supervised Federated Learning for Human Activity Recognition. arXiv."},{"key":"ref_30","unstructured":"Li, X., Wu, M., Chen, Z., and Zhang, L. (2021). Personalization Models for Human Activity Recognition with Distribution Matching-Based Metrics, Proceedings of the Deep Learning for Human Activity Recognition, Kyoto, Japan, 8 January 2021, Springer Singapore."},{"key":"ref_31","unstructured":"US Food and Drug Administration (2019). Proposed Regulatory Framework for Mondifications to Artificial Intelligence \/ Machine Learning - Based Software as a Medical Device, US Food and Drug Administration."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Bendale, A., and Boult, T. (2015). Towards Open Set Deep Networks. arXiv.","DOI":"10.1109\/CVPR.2016.173"},{"key":"ref_33","unstructured":"Geng, C., Huang, S.j., and Chen, S. (2019). Recent Advances in Open Set Recognition: A Survey. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Boyer, P., Burns, D., and Whyne, C. (2021). Out-of-Distribution Detection of Human Activity Recognition with Smartwatch Inertial Sensors. Sensors, 21.","DOI":"10.3390\/s21051669"},{"key":"ref_35","unstructured":"Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y. (2014). OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks. arXiv."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Li, G., Ge, Y., Zhang, Z., Jin, Z., and Blumenstein, M. (2017). Learning Deep and Shallow Features for Human Activity Recognition, Proceedings of the Knowledge Science, Engineering and Management, Melbourne, VIC, Australia, 19\u201320 August 2017, Springer International Publishing. Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-319-63558-3"},{"key":"ref_37","unstructured":"Bromley, J., Guyon, I., LeCun, Y., S\u00e4ckinger, E., and Shah, R. (1993, January 6\u201314). Signature Verification Using a \u201cSiamese\u201d Time Delay Neural Network. Proceedings of the 6th International Conference on Neural Information Processing Systems, San Francisco, CA, USA."},{"key":"ref_38","first-page":"1","article-title":"Triple Cross-Domain Attention on Human Activity Recognition Using Wearable Sensors","volume":"30","author":"Tang","year":"2022","journal-title":"IEEE Trans. Emerg. Top. Comput. Intell."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Khaertdinov, B., Ghaleb, E., and Asteriadis, S. (2021, January 22\u201326). Deep Triplet Networks with Attention for Sensor-based Human Activity Recognition. Proceedings of the 2021 IEEE International Conference on Pervasive Computing and Communications (PerCom), Kassel, Germany.","DOI":"10.1109\/PERCOM50583.2021.9439116"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"107889","DOI":"10.1016\/j.patcog.2021.107889","article-title":"A hierarchical sampling based triplet network for fine-grained image classification","volume":"115","author":"He","year":"2021","journal-title":"Pattern Recognit."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Hao, Y., Wang, B., and Zheng, R. (2020). Invariant Feature Learning for Sensor-based Human Activity Recognition. arXiv.","DOI":"10.1109\/TMC.2021.3064252"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"S6","DOI":"10.1186\/1475-925X-14-S2-S6","article-title":"Design, implementation and validation of a novel open framework for agile development of mobile health applications","volume":"14","author":"Banos","year":"2015","journal-title":"Biomed. Eng. Online"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"133190","DOI":"10.1109\/ACCESS.2019.2940729","article-title":"Smartphone and Smartwatch-Based Biometrics Using Activities of Daily Living","volume":"7","author":"Weiss","year":"2019","journal-title":"IEEE Access"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"075007","DOI":"10.1088\/1361-6579\/aacfd9","article-title":"Shoulder physiotherapy exercise recognition: Machine learning the inertial signals from a smartwatch","volume":"39","author":"Burns","year":"2018","journal-title":"Physiol. Meas."},{"key":"ref_45","unstructured":"Jordao, A., Nazare Jr., A.C., Sena, J., and Schwartz, W.R. (2019). Human Activity Recognition Based on Wearable Sensor Data: A Standardization of the State-of-the-Art. arXiv."},{"key":"ref_46","unstructured":"(2022, May 29). Chollet, Fran\u00e7ois Keras. GitHub Repository. Available online: https:\/\/github.com\/fchollet\/keras."},{"key":"ref_47","first-page":"1","article-title":"Seglearn: A Python Package for Learning Sequences and Time Series","volume":"19","author":"Burns","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"ref_48","unstructured":"Louppe, G., Wehenkel, L., Sutera, A., and Geurts, P. (2013, January 5\u201310). Understanding variable importances in forests of randomized trees. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/s10994-006-6226-1","article-title":"Extremely randomized trees","volume":"63","author":"Geurts","year":"2006","journal-title":"Mach. Learn."},{"key":"ref_50","unstructured":"Hendrycks, D., and Gimpel, K. (2016). A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. arXiv."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, J., Philbin, J., Chen, B., and Wu, Y. (2014). Learning Fine-grained Image Similarity with Deep Ranking. arXiv.","DOI":"10.1109\/CVPR.2014.180"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Wang, L., Li, Y., and Lazebnik, S. (2016). Learning Deep Structure-Preserving Image-Text Embeddings. arXiv.","DOI":"10.1109\/CVPR.2016.541"},{"key":"ref_53","unstructured":"Ferrari, V., Hebert, M., Sminchisescu, C., and Weiss, Y. (2018). Correcting the Triplet Selection Bias for Triplet Loss, Proceedings of the Computer Vision\u2014ECCV 2018, Munich, Germany, 8\u201314 September2018, Springer International Publishing. Lecture Notes in Computer Science."},{"key":"ref_54","unstructured":"Guo, C., Pleiss, G., Sun, Y., and Weinberger, K.Q. (, January 6\u201311). On Calibration of Modern Neural Networks. Proceedings of the International Conference on Machine Learning, Sydney, NSW, Australia."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2016). Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. arXiv.","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. arXiv.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_57","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv."},{"key":"ref_58","first-page":"9507938","article-title":"A Real-Time Patient Monitoring Framework for Fall Detection","volume":"2019","author":"Ajerla","year":"2019","journal-title":"Wirel. Commun. Mob. Netw."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/14\/5222\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:49:16Z","timestamp":1760140156000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/14\/5222"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,13]]},"references-count":58,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2022,7]]}},"alternative-id":["s22145222"],"URL":"https:\/\/doi.org\/10.3390\/s22145222","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,7,13]]}}}