{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T18:21:13Z","timestamp":1774462873748,"version":"3.50.1"},"reference-count":48,"publisher":"MDPI AG","issue":"23","license":[{"start":{"date-parts":[[2023,11,30]],"date-time":"2023-11-30T00:00:00Z","timestamp":1701302400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000849","name":"National Centre for the Replacement, Refinement and Reduction of Animals in Research","doi-asserted-by":"publisher","award":["NC\/T002050\/1"],"award-info":[{"award-number":["NC\/T002050\/1"]}],"id":[{"id":"10.13039\/501100000849","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper presents a spatiotemporal deep learning approach for mouse behavioral classification in the home-cage. Using a series of dual-stream architectures with assorted modifications for optimal performance, we introduce a novel feature sharing approach that jointly processes the streams at regular intervals throughout the network. The dataset in focus is an annotated, publicly available dataset of a singly-housed mouse. We achieved even better classification accuracy by ensembling the best performing models; an Inception-based network and an attention-based network, both of which utilize this feature sharing attribute. Furthermore, we demonstrate through ablation studies that for all models, the feature sharing architectures consistently outperform the conventional dual-stream having standalone streams. In particular, the inception-based architectures showed higher feature sharing gains with their increase in accuracy anywhere between 6.59% and 15.19%. The best-performing models were also further evaluated on other mouse behavioral datasets.<\/jats:p>","DOI":"10.3390\/s23239532","type":"journal-article","created":{"date-parts":[[2023,11,30]],"date-time":"2023-11-30T09:39:12Z","timestamp":1701337152000},"page":"9532","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Dual-Stream Spatiotemporal Networks with Feature Sharing for Monitoring Animals in the Home Cage"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7393-4600","authenticated-orcid":false,"given":"Ezechukwu Israel","family":"Nwokedi","sequence":"first","affiliation":[{"name":"School of Computer Science, College of Science, University of Lincoln, Brayford Pool, Lincoln LN6 7TS, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0443-7704","authenticated-orcid":false,"given":"Rasneer Sonia","family":"Bains","sequence":"additional","affiliation":[{"name":"Mary Lyon Centre at MRC Harwell, Oxfordshire OX11 0RD, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8253-2606","authenticated-orcid":false,"given":"Luc","family":"Bidaut","sequence":"additional","affiliation":[{"name":"Independent Researcher, Lincoln LN6 7TS, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0115-0724","authenticated-orcid":false,"given":"Xujiong","family":"Ye","sequence":"additional","affiliation":[{"name":"School of Computer Science, College of Science, University of Lincoln, Brayford Pool, Lincoln LN6 7TS, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0572-0600","authenticated-orcid":false,"given":"Sara","family":"Wells","sequence":"additional","affiliation":[{"name":"Mary Lyon Centre at MRC Harwell, Oxfordshire OX11 0RD, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7636-4554","authenticated-orcid":false,"given":"James M.","family":"Brown","sequence":"additional","affiliation":[{"name":"School of Computer Science, College of Science, University of Lincoln, Brayford Pool, Lincoln LN6 7TS, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,11,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1017\/S0963180115000079","article-title":"The flaws and human harms of animal experimentation","volume":"24","author":"Akhtar","year":"2015","journal-title":"Camb. Q. Healthc. Ethics"},{"key":"ref_2","unstructured":"NC3Rs (2023, November 23). How Many Animals Are Used in Research?. Available online: https:\/\/nc3rs.org.uk\/how-many-animals-are-used-research#:~:text=In%20Great%20Britain%20in%202020,and%20monkeys%2C%20are%20also%20used."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"425","DOI":"10.1038\/nrg.2017.19","article-title":"Comparative transcriptomics in human and mouse","volume":"18","author":"Breschi","year":"2017","journal-title":"Nat. Rev. Genet."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1002\/9780470942390.mo140195","article-title":"Aging research using mouse models","volume":"5","author":"Anderson","year":"2015","journal-title":"Curr. Protoc. Mouse Biol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"697621","DOI":"10.3389\/fnagi.2021.697621","article-title":"Functional aging in male C57BL\/6J mice across the life-span: A systematic behavioral analysis of motor, emotional, and memory function to define an aging phenotype","volume":"13","author":"Yanai","year":"2021","journal-title":"Front. Aging Neurosci."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1078\/0940-2993-00301","article-title":"Behavioral phenotyping of mice in pharmacological and toxicological research","volume":"55","author":"Karl","year":"2003","journal-title":"Exp. Toxicol. Pathol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/ncomms1064","article-title":"Automated home-cage behavioural phenotyping of mice","volume":"1","author":"Jhuang","year":"2010","journal-title":"Nat. Commun."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"575434","DOI":"10.3389\/fnbeh.2020.575434","article-title":"Three pillars of automated home-cage phenotyping of mice: Novel findings, refinement, and reproducibility based on literature and experience","volume":"14","author":"Voikar","year":"2020","journal-title":"Front. Behav. Neurosci."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"e01454","DOI":"10.1016\/j.heliyon.2019.e01454","article-title":"Non-intrusive high throughput automated data collection from the home cage","volume":"5","author":"Iannello","year":"2019","journal-title":"Heliyon"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"235","DOI":"10.3758\/s13428-014-0451-5","article-title":"SCORHE: A novel and practical approach to video monitoring of laboratory mice housed in vivarium cage racks","volume":"47","author":"Salem","year":"2015","journal-title":"Behav. Res. Methods"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"112620","DOI":"10.1016\/j.bbr.2020.112620","article-title":"IntelliCage as a tool for measuring mouse behavior\u201320 years perspective","volume":"388","author":"Kiryk","year":"2020","journal-title":"Behav. Brain Res."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Liu, H., Liu, T., Chen, Y., Zhang, Z., and Li, Y.F. (2022). EHPE: Skeleton cues-based gaussian coordinate encoding for efficient human pose estimation. IEEE Trans. Multimed.","DOI":"10.1109\/TMM.2022.3197364"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1016\/j.neucom.2020.12.090","article-title":"NGDNet: Nonuniform Gaussian-label distribution learning for infrared head pose estimation and on-task behavior understanding in the classroom","volume":"436","author":"Liu","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_14","first-page":"413","article-title":"Tracking of Individual Mice in a Social Setting Using Video Tracking Combined with RFID tags","volume":"10","author":"Armstrong","year":"2016","journal-title":"Proc. Meas. Behav."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., and Fei-Fei, L. (2014, January 23\u201328). Large-scale video classification with convolutional neural networks. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.223"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Han, X. (2017). Automatic liver lesion segmentation using a deep convolutional neural network method. arXiv.","DOI":"10.1002\/mp.12155"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1269","DOI":"10.1109\/TNNLS.2020.3041646","article-title":"A local-global dual-stream network for building extraction from very-high-resolution remote sensing images","volume":"33","author":"Zhang","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"16439","DOI":"10.1007\/s00521-021-06239-5","article-title":"Local-aware spatio-temporal attention network with multi-stage feature fusion for human action recognition","volume":"33","author":"Hou","year":"2021","journal-title":"Neural Comput. Appl."},{"key":"ref_19","unstructured":"Drozdzal, M., Vorontsov, E., Chartrand, G., Kadoury, S., and Pal, C. (2016). International Workshop on Deep Learning in Medical Image Analysis, International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, DLMIA 2016, LABELS 2016: Deep Learning and Data Labeling for Medical Applications, Springer."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"126582","DOI":"10.1016\/j.neucom.2023.126582","article-title":"TSDTVOS: Target-guided spatiotemporal dual-stream transformers for video object segmentation","volume":"555","author":"Zhou","year":"2023","journal-title":"Neurocomputing"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_22","unstructured":"Glorot, X., and Bengio, Y. (2010, January 13\u201315). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Weng, O., Marcano, G., Loncar, V., Khodamoradi, A., Sheybani, N., Meza, A., Koushanfar, F., Denolf, K., Duarte, J.M., and Kastner, R. (2023). Tailor: Altering Skip Connections for Resource-Efficient Inference. arXiv.","DOI":"10.1145\/3624990"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"383","DOI":"10.5194\/isprs-archives-XLIII-B2-2020-383-2020","article-title":"Long-short skip connections in deep neural networks for dsm refinement","volume":"43","author":"Bittner","year":"2020","journal-title":"Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Carreira, J., and Zisserman, A. (2017, January 21\u201326). Quo vadis, action recognition? A new model and the kinetics dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.502"},{"key":"ref_26","unstructured":"Simonyan, K., and Zisserman, A. (2014). Two-stream convolutional networks for action recognition in videos. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"5455","DOI":"10.1109\/JSTARS.2020.3021074","article-title":"A two-stream multiscale deep learning architecture for pan-sharpening","volume":"13","author":"Wei","year":"2020","journal-title":"IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7\u201312). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"183","DOI":"10.4236\/jbise.2019.122012","article-title":"Applying deep learning models to mouse behavior recognition","volume":"12","author":"Nguyen","year":"2019","journal-title":"J. Biomed. Sci. Eng."},{"key":"ref_30","unstructured":"Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., and Serre, T. (2011, January 6\u201313). HMDB: A large video database for human motion recognition. Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain.","DOI":"10.1109\/ICCV.2011.6126543"},{"key":"ref_32","unstructured":"Doll\u00e1r, P., Rabaud, V., Cottrell, G., and Belongie, S. (2005, January 15\u201316). Behavior recognition via sparse spatio-temporal features. Proceedings of the 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, Beijing, China."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"426","DOI":"10.1016\/j.bbr.2011.07.052","article-title":"Towards high-throughput phenotyping of complex patterned behaviors in rodents: Focus on mouse self-grooming and its sequencing","volume":"225","author":"Kyzar","year":"2011","journal-title":"Behav. Brain Res."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/j.ejphar.2004.11.054","article-title":"Mouse grooming microstructure is a reliable anxiety marker bidirectionally sensitive to GABAergic drugs","volume":"508","author":"Kalueff","year":"2005","journal-title":"Eur. J. Pharmacol."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Liu, H., Huang, X., Xu, J., Mao, H., Li, Y., Ren, K., Ma, G., Xue, Q., Tao, H., and Wu, S. (2021). Dissection of the relationship between anxiety and stereotyped self-grooming using the Shank3B mutant autistic model, acute stress model and chronic pain model. Neurobiol. Stress, 15.","DOI":"10.1016\/j.ynstr.2021.100417"},{"key":"ref_36","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv."},{"key":"ref_37","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv."},{"key":"ref_38","unstructured":"Bertasius, G., Wang, H., and Torresani, L. (2021). Is space-time attention all you need for video understanding. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lu\u010di\u0107, M., and Schmid, C. (2021, January 11\u201317). Vivit: A video vision transformer. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, BC, Canada.","DOI":"10.1109\/ICCV48922.2021.00676"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"796","DOI":"10.1177\/0020294020902788","article-title":"A novel multi-stream method for violent interaction detection using deep learning","volume":"53","author":"Li","year":"2020","journal-title":"Meas. Control"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Gharagozloo, M., Amrani, A., Wittingstall, K., Hamilton-Wright, A., and Gris, D. (2021). Machine Learning in Modeling of Mouse Behavior. Front. Neurosci., 15.","DOI":"10.3389\/fnins.2021.700253"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Graves, A., Fern\u00e1ndez, S., and Schmidhuber, J. (2005, January 10\u201315). Bidirectional LSTM networks for improved phoneme classification and recognition. Proceedings of the International Conference on Artificial Neural Networks, Warsaw, Poland.","DOI":"10.1007\/11550907_126"},{"key":"ref_44","unstructured":"Suzuki, S., Iseki, Y., Shiino, H., Zhang, H., Iwamoto, A., and Takahashi, F. (2018, January 12). Convolutional Neural Network and Bidirectional LSTM Based Taxonomy Classification Using External Dataset at SIGIR eCom Data Challenge. Proceedings of the eCOM@ SIGIR, Ann Arbor, MI, USA."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"188","DOI":"10.1016\/j.isprsjprs.2019.01.015","article-title":"Recurrently exploring class-wise attention in a hybrid convolutional and bidirectional LSTM network for multi-label aerial image classification","volume":"149","author":"Hua","year":"2019","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Farneb\u00e4ck, G. (2003, January 18\u201321). Two-frame motion estimation based on polynomial expansion. Proceedings of the Scandinavian conference on Image Analysis, SCIA 2023, Sirkka, Finland.","DOI":"10.1007\/3-540-45103-X_50"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1093\/oxfordjournals.pan.a004868","article-title":"Logistic regression in rare events data","volume":"9","author":"King","year":"2001","journal-title":"Political Anal."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27\u201330). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.308"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/23\/9532\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:35:03Z","timestamp":1760132103000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/23\/9532"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,30]]},"references-count":48,"journal-issue":{"issue":"23","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["s23239532"],"URL":"https:\/\/doi.org\/10.3390\/s23239532","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,30]]}}}