{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T21:27:53Z","timestamp":1791322073464,"version":"4.3.1"},"reference-count":51,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T00:00:00Z","timestamp":1775088000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Automated recognition of equine\u2013human interaction behaviors from video represents a significant challenge in computational ethology, with critical applications spanning animal welfare assessment, equine-assisted services evaluation, and safety monitoring in equestrian environments. Existing approaches to animal behavior recognition typically focus on single species in isolation, rely solely on facial expression analysis while ignoring full-body posture, or employ flat classification architectures that fail under the severe class imbalances characteristic of naturalistic behavioral datasets. Furthermore, no prior framework integrates simultaneous analysis of both human and equine body language for cross-species interaction classification. This paper presents a novel hierarchical classification framework integrating multi-modal computer vision features to distinguish behavioral states during horse\u2013human encounters. Our methodology employs three complementary feature extraction pipelines: YOLOv8 for spatial relationship modeling, MediaPipe for human postural analysis, and AP-10K for equine body language interpretation. From 28 annotated interaction videos comprising 50,270 temporal samples across five horse breeds, we extract 35 discriminative features capturing proximity dynamics, body orientation, and species-specific behavioral indicators. To address severe class imbalance (18.3:1 ratio between affiliative and avoidant categories), we implement cost-sensitive gradient boosting with automatic class weight optimization within a two-stage hierarchical architecture. The first stage classifies interactions into three parent categories (affiliative, neutral, avoidant) achieving 73.2% balanced accuracy, while stage two discriminates six fine-grained sub-behaviors achieving 88.5% balanced accuracy (under oracle parent-category routing; cascaded end-to-end performance is 62.9% balanced accuracy due to Stage 1 error propagation, identifying parent classification as the primary bottleneck). Notably, our system achieves 85.0% recall on safety-critical avoidant behaviors despite their representation of only 3.8% of the dataset. Extensive ablation studies demonstrate that equine pose features contribute most critically to classification performance, while comprehensive cross-validation analysis confirms model robustness across diverse interaction contexts. The proposed framework establishes the first systematic multimodal cross-species behavioral assessment pipeline in human\u2013animal interaction research, with direct implications for improving equine welfare monitoring and rider safety protocols.<\/jats:p>","DOI":"10.3390\/s26072202","type":"journal-article","created":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T11:51:03Z","timestamp":1775130663000},"page":"2202","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Multi-Modal Feature Fusion and Hierarchical Classification for Automated Equine\u2013Human Interaction Behavior Recognition"],"prefix":"10.3390","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-6172-9292","authenticated-orcid":false,"given":"Samierra","family":"Arora","sequence":"first","affiliation":[{"name":"System Design & Management, Massachusetts Institute of Technology, Cambridge, MA 02142, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0858-1343","authenticated-orcid":false,"given":"Emily","family":"Kieson","sequence":"additional","affiliation":[{"name":"Equine International, Boston, MA 02115, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9732-9679","authenticated-orcid":false,"given":"Christine","family":"Rudd","sequence":"additional","affiliation":[{"name":"Equine International, Boston, MA 02115, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7271-3224","authenticated-orcid":false,"given":"Peter A.","family":"Gloor","sequence":"additional","affiliation":[{"name":"System Design & Management, Massachusetts Institute of Technology, Cambridge, MA 02142, USA"},{"name":"Cologne Institute for Information Sciences, University of Cologne, 50923 Cologne, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,4,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.applanim.2007.04.015","article-title":"A review of the human\u2013horse relationship","volume":"109","author":"Hausberger","year":"2008","journal-title":"Appl. Anim. Behav. Sci."},{"key":"ref_2","unstructured":"McGreevy, P.D. (2012). Equine Behavior: A Guide for Veterinarians and Equine Scientists, Saunders. [2nd ed.]."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Waran, N. (2007). The Welfare of Horses, Springer.","DOI":"10.1007\/978-0-306-48215-1"},{"key":"ref_4","unstructured":"Food and Agriculture Organization of the United Nations (2022). FAOSTAT: Live Animals, FAO. Available online: https:\/\/www.fao.org\/home\/en."},{"key":"ref_5","first-page":"813","article-title":"Randomized controlled trial of therapeutic horseback riding in children with autism spectrum disorder","volume":"51","author":"Gabriels","year":"2012","journal-title":"J. Am. Acad. Child Adolesc. Psychiatry"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1261","DOI":"10.1007\/s10803-009-0734-3","article-title":"The effect of therapeutic horseback riding on social functioning in children with autism","volume":"39","author":"Bass","year":"2009","journal-title":"J. Autism Dev. Disord."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"684","DOI":"10.1111\/j.1469-8749.2011.03951.x","article-title":"Effects of hippotherapy and therapeutic horseback riding on postural control or balance in children with cerebral palsy","volume":"53","author":"Zadnikar","year":"2011","journal-title":"Dev. Med. Child Neurol."},{"key":"ref_8","first-page":"3","article-title":"Effects of therapeutic horseback riding on post-traumatic stress disorder in military veterans","volume":"5","author":"Johnson","year":"2018","journal-title":"Mil. Med. Res."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/0020-1383(95)00176-X","article-title":"Accidents with horses: What has changed in 20 years?","volume":"27","author":"Chitnavis","year":"1996","journal-title":"Injury"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"813","DOI":"10.1111\/evj.13076","article-title":"Research trends in equine movement analysis, future opportunities and potential barriers in the digital age: A scoping review from 1978 to 2018","volume":"51","author":"Egan","year":"2019","journal-title":"Equine Vet. J."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1111\/vaa.12212","article-title":"An equine pain face","volume":"42","author":"Gleerup","year":"2015","journal-title":"Vet. Anaesth. Analg."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Dalla Costa, E., Minero, M., Lebelt, D., Stucke, D., Canali, E., and Leach, M.C. (2014). Development of the Horse Grimace Scale (HGS) as a pain assessment tool in horses undergoing routine castration. PLoS ONE, 9.","DOI":"10.1371\/journal.pone.0092281"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Sigurj\u00f3nsd\u00f3ttir, H., and Haraldsson, H. (2019). Significance of group composition for the welfare of pastured horses. Animals, 9.","DOI":"10.3390\/ani9010014"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Merkies, K., Ready, C., Farkas, L., and Hodder, A. (2019). Eye blink rates and eyelid twitches as a non-invasive measure of stress in the domestic horse. Animals, 9.","DOI":"10.3390\/ani9080562"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Fureix, C., Jego, P., Henry, S., Lansade, L., and Hausberger, M. (2012). Towards an ethological animal model of depression? A study on horses. PLoS ONE, 7.","DOI":"10.1371\/journal.pone.0039280"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1016\/j.applanim.2011.10.004","article-title":"Keeping horses in groups: A review","volume":"136","author":"Hartmann","year":"2012","journal-title":"Appl. Anim. Behav. Sci."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1007\/s11263-022-01716-3","article-title":"Going deeper than tracking: A survey of computer-vision based recognition of animal pain and emotions","volume":"131","author":"Feighelstein","year":"2023","journal-title":"Int. J. Comput. Vis."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Feighelstein, M., Riccie-Bonot, C., Hasan, H., Weinberg, H., Rettig, T., Segal, M., Distelfeld, T., Shimshoni, I., Mills, D.S., and Zamansky, A. (2024). Automated recognition of emotional states of horses from facial expressions. PLoS ONE, 19.","DOI":"10.1371\/journal.pone.0302893"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Bhave, A., Kieson, E., Hafner, A., and Gloor, P.A. (2025). Identifying Novel Emotions and Wellbeing of Horses from Videos Through Unsupervised Learning. Sensors, 25.","DOI":"10.20944\/preprints202501.0526.v1"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1281","DOI":"10.1038\/s41593-018-0209-y","article-title":"DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning","volume":"21","author":"Mathis","year":"2018","journal-title":"Nat. Neurosci."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"486","DOI":"10.1038\/s41592-022-01426-1","article-title":"SLEAP: A deep learning system for multi-animal pose tracking","volume":"19","author":"Pereira","year":"2022","journal-title":"Nat. Methods"},{"key":"ref_22","unstructured":"Yu, H., Xu, Y., Zhang, J., Zhao, W., Guan, Z., and Tao, D. (2021). AP-10K: A benchmark for animal pose estimation in the wild. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Arnab, A., Doersch, C., and Zisserman, A. (2019, January 15\u201320). Exploiting temporal context for 3D human pose estimation. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00351"},{"key":"ref_24","unstructured":"Elkan, C. (2001). The foundations of cost-sensitive learning. International Joint Conference on Artificial Intelligence, Lawrence Erlbaum Associates Ltd."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic minority over-sampling technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Intell. Res."},{"key":"ref_26","unstructured":"He, H., Bai, Y., Garcia, E.A., and Li, S. (2008, January 1\u20138). ADASYN: Adaptive synthetic sampling approach for imbalanced learning. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Fern\u00e1ndez, A., Garc\u00eda, S., Galar, M., Prati, R.C., Krawczyk, B., and Herrera, F. (2018). Learning from Imbalanced Data Sets, Springer.","DOI":"10.1007\/978-3-319-98074-4"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1007\/s00265-002-0556-1","article-title":"Social grouping and maternal behaviour in feral horses (Equus caballus): The influence of males on maternal protectiveness","volume":"53","author":"Cameron","year":"2003","journal-title":"Behav. Ecol. Sociobiol."},{"key":"ref_29","unstructured":"Mills, D.S., and McDonnell, S.M. (2005). Relationships and communication in socially natural horse herds. The Domestic Horse, Cambridge University Press."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1016\/S0168-1591(02)00112-0","article-title":"Equid play ethogram","volume":"78","author":"McDonnell","year":"2002","journal-title":"Appl. Anim. Behav. Sci."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"R677","DOI":"10.1016\/j.cub.2014.06.023","article-title":"The eyes and ears are visual indicators of attention in domestic horses","volume":"24","author":"Wathan","year":"2014","journal-title":"Curr. Biol."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1016\/j.applanim.2014.09.015","article-title":"Can ear postures reliably measure the positive emotional state of cows?","volume":"161","author":"Proctor","year":"2014","journal-title":"Appl. Anim. Behav. Sci."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"106236","DOI":"10.1016\/j.applanim.2024.106236","article-title":"Which affiliative behaviour can be used as a valid, reliable and feasible indicator of positive welfare in horse husbandry?","volume":"273","author":"Hartmann","year":"2024","journal-title":"Appl. Anim. Behav. Sci."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1016\/j.applanim.2015.09.005","article-title":"Detecting horses\u2019 sickness: In search of visible signs","volume":"175","author":"Hausberger","year":"2016","journal-title":"Appl. Anim. Behav. Sci."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"423","DOI":"10.1109\/TPAMI.2018.2798607","article-title":"Multimodal machine learning: A survey and taxonomy","volume":"41","author":"Ahuja","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1109\/TSMCC.2011.2161285","article-title":"A Review on Ensembles for the Class Imbalance Problem: Bagging-, Boosting-, and Hybrid-Based Approaches","volume":"42","author":"Galar","year":"2012","journal-title":"IEEE Trans. Syst. Man Cybern. Part C"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"105179","DOI":"10.1016\/j.compag.2019.105179","article-title":"Development of a methodological framework for a robust prediction of the main behaviours of dairy cows using a combination of machine learning algorithms on accelerometer data","volume":"169","author":"Riaboff","year":"2020","journal-title":"Comput. Electron. Agric."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"519","DOI":"10.1006\/anbe.2003.2078","article-title":"Anatomical constraints generate honesty: Acoustic cues to age and weight in the roars of red deer stags","volume":"65","author":"Reby","year":"2003","journal-title":"Anim. Behav."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1007\/s13748-016-0094-0","article-title":"Learning from imbalanced data: Open challenges and future directions","volume":"5","author":"Krawczyk","year":"2016","journal-title":"Prog. Artif. Intell."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1007\/s10618-012-0295-5","article-title":"Training and assessing classification with imbalanced data","volume":"28","author":"Menardi","year":"2014","journal-title":"Data Min. Knowl. Discov."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1325","DOI":"10.1111\/2041-210X.12584","article-title":"BORIS: A free, versatile open-source event-logging software for video\/audio coding and live observations","volume":"7","author":"Friard","year":"2016","journal-title":"Methods Ecol. Evol."},{"key":"ref_42","unstructured":"Jocher, G., Chaurasia, A., and Qiu, J. (2025, January 15). Ultralytics YOLO (Version 8.0.0) [Software]. Available online: https:\/\/github.com\/ultralytics\/ultralytics."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27\u201330). You only look once: Unified, real-time object detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_44","unstructured":"Redmon, J., and Farhadi, A. (2018). YOLOv3: An incremental improvement. arXiv."},{"key":"ref_45","unstructured":"Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.L., Yong, M.G., and Lee, J. (2019). MediaPipe: A framework for building perception pipelines. arXiv."},{"key":"ref_46","first-page":"6639","article-title":"CatBoost: Unbiased boosting with categorical features","volume":"31","author":"Prokhorenkova","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Chen, T., and Guestrin, C. (2016). XGBoost: A scalable tree boosting system. KDD \u201916: The 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Association for Computing Machinery.","DOI":"10.1145\/2939672.2939785"},{"key":"ref_48","first-page":"3149","article-title":"LightGBM: A highly efficient gradient boosting decision tree","volume":"30","author":"Ke","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"9575","DOI":"10.1038\/s41598-022-13348-1","article-title":"Automated recognition of pain in cats","volume":"12","author":"Feighelstein","year":"2022","journal-title":"Sci. Rep."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Corujo, L.A., Kieson, E., Schloesser, T., and Gloor, P.A. (2021). Emotion Recognition in Horses with Convolutional Neural Networks. Future Internet, 13.","DOI":"10.3390\/fi13100250"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/7\/2202\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T11:54:26Z","timestamp":1775130866000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/26\/7\/2202"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,2]]},"references-count":51,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2026,4]]}},"alternative-id":["s26072202"],"URL":"https:\/\/doi.org\/10.3390\/s26072202","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,2]]}}}