{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T05:28:07Z","timestamp":1773984487908,"version":"3.50.1"},"reference-count":46,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2021,8,29]],"date-time":"2021-08-29T00:00:00Z","timestamp":1630195200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100007567","name":"City University of Hong Kong","doi-asserted-by":"publisher","award":["9610450"],"award-info":[{"award-number":["9610450"]}],"id":[{"id":"10.13039\/100007567","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>With the recent advances in deep learning, wearable sensors have increasingly been used in automated animal activity recognition. However, there are two major challenges in improving recognition performance\u2014multi-modal feature fusion and imbalanced data modeling. In this study, to improve classification performance for equine activities while tackling these two challenges, we developed a cross-modality interaction network (CMI-Net) involving a dual convolution neural network architecture and a cross-modality interaction module (CMIM). The CMIM adaptively recalibrated the temporal- and axis-wise features in each modality by leveraging multi-modal information to achieve deep intermodality interaction. A class-balanced (CB) focal loss was adopted to supervise the training of CMI-Net to alleviate the class imbalance problem. Motion data was acquired from six neck-attached inertial measurement units from six horses. The CMI-Net was trained and verified with leave-one-out cross-validation. The results demonstrated that our CMI-Net outperformed the existing algorithms with high precision (79.74%), recall (79.57%), F1-score (79.02%), and accuracy (93.37%). The adoption of CB focal loss improved the performance of CMI-Net, with increases of 2.76%, 4.16%, and 3.92% in precision, recall, and F1-score, respectively. In conclusion, CMI-Net and CB focal loss effectively enhanced the equine activity classification performance using imbalanced multi-modal sensor data.<\/jats:p>","DOI":"10.3390\/s21175818","type":"journal-article","created":{"date-parts":[[2021,8,31]],"date-time":"2021-08-31T22:58:15Z","timestamp":1630450695000},"page":"5818","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["Cross-Modality Interaction Network for Equine Activity Recognition Using Imbalanced Multi-Modal Data"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7223-7926","authenticated-orcid":false,"given":"Axiu","family":"Mao","sequence":"first","affiliation":[{"name":"Department of Infectious Diseases and Public Health, Jockey Club College of Veterinary Medicine and Life Sciences, City University of Hong Kong, Hong Kong, China"}]},{"given":"Endai","family":"Huang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, City University of Hong Kong, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4239-6483","authenticated-orcid":false,"given":"Haiming","family":"Gan","sequence":"additional","affiliation":[{"name":"Department of Infectious Diseases and Public Health, Jockey Club College of Veterinary Medicine and Life Sciences, City University of Hong Kong, Hong Kong, China"},{"name":"College of Electronic Engineering, South China Agricultural University, Guangzhou 510642, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5465-8578","authenticated-orcid":false,"given":"Rebecca S. V.","family":"Parkes","sequence":"additional","affiliation":[{"name":"Department of Veterinary Clinical Sciences, Jockey Club College of Veterinary Medicine and Life Sciences, City University of Hong Kong, Hong Kong, China"},{"name":"Centre for Companion Animal Health, Jockey Club College of Veterinary Medicine and Life Sciences, City University of Hong Kong, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9741-5912","authenticated-orcid":false,"given":"Weitao","family":"Xu","sequence":"additional","affiliation":[{"name":"Department of Computer Science, City University of Hong Kong, Hong Kong, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1556-1068","authenticated-orcid":false,"given":"Kai","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Infectious Diseases and Public Health, Jockey Club College of Veterinary Medicine and Life Sciences, City University of Hong Kong, Hong Kong, China"},{"name":"Animal Health Research Centre, Chengdu Research Institute, City University of Hong Kong, Chengdu 610000, China"}]}],"member":"1968","published-online":{"date-parts":[[2021,8,29]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"106020","DOI":"10.1016\/j.compag.2021.106020","article-title":"A framework for energy-efficient equine activity recognition with leg accelerometers","volume":"183","author":"Eerdekens","year":"2021","journal-title":"Comput. Electron. Agric."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Parkes, R.S.V., Weller, R., Pfau, T., and Witte, T.H. (2019). The effect of training on stride duration in a cohort of two-year-old and three-year-old thoroughbred racehorses. Animals, 9.","DOI":"10.3390\/ani9070466"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"567","DOI":"10.1111\/evj.12715","article-title":"Do we have to redefine lameness in the era of quantitative gait analysis?","volume":"49","author":"Pfau","year":"2017","journal-title":"Equine Vet. J."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Bosch, S., Serra Bragan\u00e7a, F., Marin-Perianu, M., Marin-Perianu, R., van der Zwaag, B.J., Voskamp, J., Back, W., Van Weeren, R., and Havinga, P. (2018). Equimoves: A wireless networked inertial measurement system for objective examination of horse gait. Sensors, 18.","DOI":"10.3390\/s18030850"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"105291","DOI":"10.1016\/j.compag.2020.105291","article-title":"Smart poultry management: Smart sensors, big data, and the internet of things","volume":"170","author":"Astill","year":"2020","journal-title":"Comput. Electron. Agric."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Rue\u00df, D., Rue\u00df, J., H\u00fcmmer, C., Deckers, N., Migal, V., Kienapfel, K., Wieckert, A., Barnewitz, D., and Reulke, R. (2019, January 18\u201322). Equine Welfare Assessment: Horse Motion Evaluation and Comparison to Manual Pain Measurements. Proceedings of the Pacific-Rim Symposium on Image and Video Technology, PSIVT 2019, Sydney, Australia.","DOI":"10.1007\/978-3-030-34879-3_13"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Kamminga, J.W., Meratnia, N., and Havinga, P.J.M. (2019, January 26\u201328). Dataset: Horse Movement Data and Analysis of its Potential for Activity Recognition. Proceedings of the 2nd Workshop on Data Acquisition to Analysis, DATA 2019, Prague, Czech Republic.","DOI":"10.1145\/3359427.3361908"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"105393","DOI":"10.1016\/j.applanim.2021.105393","article-title":"Dog behaviour classification with movement sensors placed on the harness and the collar","volume":"241","author":"Kumpulainen","year":"2021","journal-title":"Appl. Anim. Behav. Sci."},{"key":"ref_9","unstructured":"Tran, D.N., Nguyen, T.N., Khanh, P.C.P., and Trana, D.T. (2021). An IoT-based Design Using Accelerometers in Animal Behavior Recognition Systems. IEEE Sens. J."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"840","DOI":"10.1111\/evj.13130","article-title":"Accelerometer activity tracking in horses and the effect of pasture management on time budget","volume":"51","author":"Maisonpierre","year":"2019","journal-title":"Equine Vet. J."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1016\/j.eswa.2018.03.056","article-title":"Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges","volume":"105","author":"Nweke","year":"2018","journal-title":"Expert Syst. Appl."},{"key":"ref_12","first-page":"106241","article-title":"Deep Learning-based Cattle Activity Classification Using Joint Time-frequency Data Representation","volume":"187","author":"Noorbin","year":"2020","journal-title":"Comput. Electron. Agric."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"105178","DOI":"10.1016\/j.compag.2019.105178","article-title":"Dam behavior patterns in Japanese black beef cattle prior to calving: Automated detection using LSTM-RNN","volume":"169","author":"Peng","year":"2020","journal-title":"Comput. Electron. Agric."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Bocaj, E., Uzunidis, D., Kasnesis, P., and Patrikakis, C.Z. (2020, January 14\u201316). On the Benefits of Deep Convolutional Neural Networks on Animal Activity Recognition. Proceedings of the 2020 International Conference on Smart Systems and Technologies (SST), Osijek, Croatia.","DOI":"10.1109\/SST49455.2020.9263702"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Eerdekens, A., Deruyck, M., Fontaine, J., Martens, L., de Poorter, E., Plets, D., and Joseph, W. (September, January 31). Resampling and Data Augmentation for Equines\u2019 Behaviour Classification Based on Wearable Sensor Accelerometer Data Using a Convolutional Neural Network. Proceedings of the 2020 International Conference on Omni-layer Intelligent Systems (COINS), Barcelona, Spain.","DOI":"10.1109\/COINS49042.2020.9191639"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Chambers, R.D., Yoder, N.C., Carson, A.B., Junge, C., Allen, D.E., Prescott, L.M., Bradley, S., Wymore, G., Lloyd, K., and Lyle, S. (2021). Deep learning classification of canine behavior using a single collar-mounted accelerometer: Real-world validation. Animals, 11.","DOI":"10.3390\/ani11061549"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Liu, N., Zhang, N., and Han, J. (2020, January 14\u201319). Learning Selective Self-Mutual Attention for RGB-D Saliency Detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Available online: http:\/\/cvpr2020.thecvf.com\/.","DOI":"10.1109\/CVPR42600.2020.01377"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ha, S., and Choi, S. (2016, January 24\u201329). Convolutional neural networks for human activity recognition using multiple accelerometer and gyroscope sensors. Proceedings of the 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada.","DOI":"10.1109\/IJCNN.2016.7727224"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"114177","DOI":"10.1016\/j.eswa.2020.114177","article-title":"MLT-DNet: Speech emotion recognition using 1D dilated CNN based on multi-learning trick approach","volume":"167","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"5116","DOI":"10.1002\/int.22505","article-title":"Optimal feature selection based speech emotion recognition using two-stream deep convolutional neural network","volume":"36","author":"Mustaqeem","year":"2021","journal-title":"Int. J. Intell. Syst."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"105878","DOI":"10.1016\/j.compag.2020.105878","article-title":"Transfer learning and SE-ResNet152 networks-based for small-scale unbalanced fish species identification","volume":"180","author":"Xu","year":"2021","journal-title":"Comput. Electron. Agric."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zhang, S., Li, Z., Yan, S., He, X., and Sun, J. (2021, January 19\u201325). Distribution Alignment: A Unified Framework for Long-tail Visual Recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2021, Available online: http:\/\/cvpr2021.thecvf.com\/.","DOI":"10.1109\/CVPR46437.2021.00239"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Tan, J., Wang, C., Li, B., Li, Q., Ouyang, W., Yin, C., and Yan, J. (2020, January 14\u201319). Equalization loss for long-tailed object recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Available online: http:\/\/cvpr2020.thecvf.com\/.","DOI":"10.1109\/CVPR42600.2020.01168"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"3573","DOI":"10.1109\/TNNLS.2017.2732482","article-title":"Cost-sensitive learning of deep feature representations from imbalanced data","volume":"29","author":"Khan","year":"2017","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Cui, Y., Jia, M., Lin, T.Y., Song, Y., and Belongie, S. (2019, January 16\u201320). Class-balanced loss based on effective number of samples. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00949"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1109\/TPAMI.2018.2858826","article-title":"Focal Loss for Dense Object Detection","volume":"42","author":"Lin","year":"2018","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Wang, T., Zhu, Y., Zhao, C., Zeng, W., Wang, J., and Tang, M. (2021, January 19\u201325). Adaptive Class Suppression Loss for Long-Tail Object Detection. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2021, Available online: http:\/\/cvpr2020.thecvf.com\/.","DOI":"10.1109\/CVPR46437.2021.00312"},{"key":"ref_28","unstructured":"Mao, A.X., Huang, E.D., Xu, W.T., and Liu, K. (2021, January 20\u201323). Cross-modality Interaction Network for Equine Activity Recognition Using Time-Series Motion Data. Proceedings of the 2021 International Symposium on Animal Environment and Welfare (ISAEW), Chongqing, China. in press."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1949","DOI":"10.1109\/TIP.2021.3049959","article-title":"Bilateral Attention Network for RGB-D Salient Object Detection","volume":"30","author":"Zhang","year":"2021","journal-title":"IEEE Trans. Image Process."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Woo, S., Park, J., Lee, J.Y., and Kweon, I.S. (2018, January 8\u201314). CBAM: Convolutional block attention module. Proceedings of the European Conference on Computer Vision, ECCV 2018, Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_1"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"107101","DOI":"10.1016\/j.asoc.2021.107101","article-title":"Att-Net: Enhanced emotion recognition system using lightweight self-attention module","volume":"102","year":"2021","journal-title":"Appl. Soft Comput."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Kamminga, J.W., Jan\u00dfen, L.M., Meratnia, N., and Havinga, P.J.M. (2019). Horsing around\u2014A dataset comprising horse movement. Data, 4.","DOI":"10.3390\/data4040131"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_34","unstructured":"Kamminga, J.W., Le, D.V., and Havinga, P.J.M. (2020, January 24). Towards deep unsupervised representation learning from accelerometer time series for animal activity recognition. Proceedings of the 6th Workshop on Mining and Learning from Time Series, MiLeTS 2020, San Diego, CA, USA."},{"key":"ref_35","unstructured":"Nair, V., and Hinton, G.E. (2010, January 21\u201324). Rectified Linear Units Improve Restricted Boltzmann Machines Vinod. Proceedings of the 27th International Conference on Machine Learning, ICML 2010, Haifa, Israel."},{"key":"ref_36","unstructured":"Joze, H.R.V., Shaban, A., Iuzzolino, M.L., and Koishida, K. (2020, January 14\u201319). MMTM: Multimodal transfer module for CNN fusion. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Available online: http:\/\/cvpr2020.thecvf.com\/."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"101213","DOI":"10.1016\/j.pmcj.2020.101213","article-title":"A framework for the recognition of horse gaits through wearable devices","volume":"67","author":"Casella","year":"2020","journal-title":"Pervasive Mob. Comput."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Zeng, M., Nguyen, L.T., Yu, B., Mengshoel, O.J., Zhu, J., Wu, P., and Zhang, J. (2014, January 6\u20137). Convolutional Neural Networks for human activity recognition using mobile sensors. Proceedings of the 6th international conference on mobile computing, applications and services, MobiCASE 2014, Austin, TX, USA.","DOI":"10.4108\/icst.mobicase.2014.257786"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1109\/4235.585893","article-title":"No free lunch theorems for optimization","volume":"1","author":"Wolpert","year":"1997","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Wei, J., Wang, Q., Li, Z., Wang, S., Zhou, S.K., and Cui, S. (2021, January 19\u201325). Shallow Feature Matters for Weakly Supervised Object Localization. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2021, Available online: http:\/\/cvpr2021.thecvf.com\/.","DOI":"10.1109\/CVPR46437.2021.00593"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"Lecun","year":"2015","journal-title":"Nature"},{"key":"ref_42","first-page":"2579","article-title":"Visualizing Data using t-SNE","volume":"9","author":"Hinton","year":"2008","journal-title":"J. Mach. Learn. Res."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"758","DOI":"10.2746\/0425164044848000","article-title":"Effects of girth, saddle and weight on movements of the horse","volume":"36","author":"Back","year":"2004","journal-title":"Equine Vet. J."},{"key":"ref_44","first-page":"1","article-title":"Recent Advances in Open Set Recognition: A Survey","volume":"14","author":"Geng","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Yoshihashi, R., You, S., Shao, W., Iida, M., Kawakami, R., and Naemura, T. (2019, January 16\u201320). Classification-Reconstruction Learning for Open-Set Recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00414"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1547","DOI":"10.1007\/s10994-017-5646-4","article-title":"Weightless neural networks for open set recognition","volume":"106","author":"Cardoso","year":"2017","journal-title":"Mach. Learn."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/17\/5818\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:55:08Z","timestamp":1760165708000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/17\/5818"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,29]]},"references-count":46,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2021,9]]}},"alternative-id":["s21175818"],"URL":"https:\/\/doi.org\/10.3390\/s21175818","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,29]]}}}