{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,28]],"date-time":"2026-10-28T03:35:18Z","timestamp":1793158518430,"version":"4.5.0"},"reference-count":27,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2021,9,23]],"date-time":"2021-09-23T00:00:00Z","timestamp":1632355200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Shandong Key R&amp;D Program","award":["2019JZZY010703"],"award-info":[{"award-number":["2019JZZY010703"]}]},{"name":"Research and Demonstration of Real-time Accurate Monitoring System for Early-stage Fish in Recirculating Aquaculture System","award":["2019YFE0103700"],"award-info":[{"award-number":["2019YFE0103700"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Animals"],"abstract":"<jats:p>The rapid and precise recognition of fish behavior is critical in perceiving health and welfare by allowing farmers to make informed management decisions on recirculating aquaculture systems while reducing labor. The conventional recognition methods are to obtain movement information by implanting sensors on the skin or in the body of the fish, which can affect the normal behavior and welfare of the fish. We present a novel nondestructive method with spatiotemporal and motion information based on deep learning for real-time recognition of fish schools\u2019 behavior. In this work, a dual-stream 3D convolutional neural network (DSC3D) was proposed for the recognition of five behavior states of fish schools, including feeding, hypoxia, hypothermia, frightening and normal behavior. This DSC3D combines spatiotemporal features and motion features by using FlowNet2 and 3D convolutional neural networks and shows significant results suitable for industrial applications in automatic monitoring of fish behavior, with an average accuracy rate of 95.79%. The model evaluation results on the test dataset further demonstrated that our proposed method could be used as an effective tool for the intelligent perception of fish health status.<\/jats:p>","DOI":"10.3390\/ani11102774","type":"journal-article","created":{"date-parts":[[2021,9,24]],"date-time":"2021-09-24T00:30:33Z","timestamp":1632443433000},"page":"2774","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":77,"title":["Automatic Recognition of Fish Behavior with a Fusion of RGB and Optical Flow Data Based on Deep Learning"],"prefix":"10.3390","volume":"11","author":[{"given":"Guangxu","family":"Wang","sequence":"first","affiliation":[{"name":"National Innovation Center for Digital Fishery, China Agricultural University, Beijing 100083, China"},{"name":"College of Information and Electrical Engineering, China Agricultural University, Beijing 100083, China"},{"name":"Beijing Engineering and Technology Research Centre for Internet of Things in Agriculture, Beijing 100083, China"},{"name":"Key Laboratory of Agricultural Information Acquisition Technology, Ministry of Agriculture, Beijing 100083, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akhter","family":"Muhammad","sequence":"additional","affiliation":[{"name":"National Innovation Center for Digital Fishery, China Agricultural University, Beijing 100083, China"},{"name":"College of Information and Electrical Engineering, China Agricultural University, Beijing 100083, China"},{"name":"Beijing Engineering and Technology Research Centre for Internet of Things in Agriculture, Beijing 100083, China"},{"name":"Key Laboratory of Agricultural Information Acquisition Technology, Ministry of Agriculture, Beijing 100083, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chang","family":"Liu","sequence":"additional","affiliation":[{"name":"National Innovation Center for Digital Fishery, China Agricultural University, Beijing 100083, China"},{"name":"College of Information and Electrical Engineering, China Agricultural University, Beijing 100083, China"},{"name":"Beijing Engineering and Technology Research Centre for Internet of Things in Agriculture, Beijing 100083, China"},{"name":"Key Laboratory of Agricultural Information Acquisition Technology, Ministry of Agriculture, Beijing 100083, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ling","family":"Du","sequence":"additional","affiliation":[{"name":"National Innovation Center for Digital Fishery, China Agricultural University, Beijing 100083, China"},{"name":"College of Information and Electrical Engineering, China Agricultural University, Beijing 100083, China"},{"name":"Beijing Engineering and Technology Research Centre for Internet of Things in Agriculture, Beijing 100083, China"},{"name":"Key Laboratory of Agricultural Information Acquisition Technology, Ministry of Agriculture, Beijing 100083, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1842-419X","authenticated-orcid":false,"given":"Daoliang","family":"Li","sequence":"additional","affiliation":[{"name":"National Innovation Center for Digital Fishery, China Agricultural University, Beijing 100083, China"},{"name":"College of Information and Electrical Engineering, China Agricultural University, Beijing 100083, China"},{"name":"Beijing Engineering and Technology Research Centre for Internet of Things in Agriculture, Beijing 100083, China"},{"name":"Key Laboratory of Agricultural Information Acquisition Technology, Ministry of Agriculture, Beijing 100083, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,23]]},"reference":[{"key":"ref_1","unstructured":"FAO (2020). FAO yearbook of fishery and aquaculture statistics. FAO Yearb. Fish. Stat."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1111\/raq.12464","article-title":"Deep learning for smart fish farming: Applications, opportunities and challenges","volume":"13","author":"Yang","year":"2021","journal-title":"Rev. Aquac."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"635","DOI":"10.1016\/j.scitotenv.2013.07.075","article-title":"Biological early warning system based on the responses of aquatic organisms to disturbances: A review","volume":"466","author":"Bae","year":"2014","journal-title":"Sci. Total Environ."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"735508","DOI":"10.1016\/j.aquaculture.2020.735508","article-title":"Automatic recognition methods of fish feeding behavior in aquaculture: A review","volume":"528","author":"Li","year":"2020","journal-title":"Aquaculture"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1390","DOI":"10.1111\/raq.12388","article-title":"Nonintrusive methods for biomass estimation in aquaculture with emphasis on fish: A review","volume":"12","author":"Li","year":"2019","journal-title":"Rev. Aquac."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1016\/j.compag.2017.02.013","article-title":"Near-infrared imaging to quantify the feeding behavior of fish in aquaculture","volume":"135","author":"Zhou","year":"2017","journal-title":"Comput. Electron. Agric."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"2506","DOI":"10.1093\/icesjms\/fsv104","article-title":"Natural swimming speed of Dascyllus reticulatus increases with water temperature","volume":"72","author":"Beyan","year":"2015","journal-title":"ICES J. Mar. Sci."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"145049","DOI":"10.1109\/ACCESS.2019.2945606","article-title":"3-D Video Tracking of Multiple Fish in a Water Tank","volume":"7","author":"Liu","year":"2019","journal-title":"IEEE Access"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Cheng, S., Zhao, K., and Zhang, D. (2019). Abnormal Water Quality Monitoring Based on Visual Sensing of Three-Dimensional Motion Behavior of Fish. Symmetry, 11.","DOI":"10.3390\/sym11091179"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/j.ecoenv.2014.09.028","article-title":"Water quality monitoring using abnormal tail-beat frequency of crucian carp","volume":"111","author":"Xiao","year":"2015","journal-title":"Ecotoxicol. Environ. Saf."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"102117","DOI":"10.1016\/j.aquaeng.2020.102117","article-title":"A modified YOLOv3 model for fish detection based on MobileNetv1 as backbone","volume":"91","author":"Cai","year":"2020","journal-title":"Aquac. Eng."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"105075","DOI":"10.1016\/j.compag.2019.105075","article-title":"Visual features based automated identification of fish species using deep convolutional neural networks","volume":"167","author":"Rauf","year":"2019","journal-title":"Comput. Electron. Agric."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"115051","DOI":"10.1016\/j.eswa.2021.115051","article-title":"Real-time nondestructive fish behavior detecting in mixed polyculture system using deep-learning and low-cost devices","volume":"178","author":"Hu","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"102085","DOI":"10.1016\/j.aquaeng.2020.102085","article-title":"Research of dissolved oxygen prediction in recirculating aquaculture systems based on deep belief network","volume":"90","author":"Ren","year":"2020","journal-title":"Aquac. Eng."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1016\/j.aquaculture.2019.04.056","article-title":"Evaluation of fish feeding intensity in aquaculture using a convolutional neural network and machine vision","volume":"507","author":"Zhou","year":"2019","journal-title":"Aquaculture"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Maloy, H., Aamodt, A., and Misimi, E. (2019). A spatio-temporal recurrent network for salmon feeding action recognition from underwater videos in aquaculture. Comput. Electron. Agric., 167.","DOI":"10.1016\/j.compag.2019.105087"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1016\/j.aquaculture.2018.04.064","article-title":"Modified motion influence map and recurrent neural network-based monitoring of the local unusual behaviors for fish school in intensive aquaculture","volume":"493","author":"Zhao","year":"2018","journal-title":"Aquaculture"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1109\/TPAMI.2012.59","article-title":"3D Convolutional Neural Networks for Human Action Recognition","volume":"35","author":"Ji","year":"2013","journal-title":"IEEE Trans. Pattern Anal."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 11\u201318). Learning Spatiotemporal Features with 3D Convolutional Networks. Proceedings of the IEEE International Conference on Computer Vision, Las Condes, Chile.","DOI":"10.1109\/ICCV.2015.510"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1023\/B:VISI.0000011205.11775.fd","article-title":"Lucas-Kanade 20 years on: A unifying framework","volume":"56","author":"Baker","year":"2004","journal-title":"Int. J. Comput. Vis."},{"key":"ref_22","unstructured":"Ng, J.Y., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., and Toderici, G. (2015, January 7\u201312). Beyond Short Snippets: Deep Networks for Video Classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Dosovitskiy, A., Fischer, P., Ilg, E., Haeusser, P., Hazirbas, C., Golkov, V., van der Smagt, P., Cremers, D., and Brox, T. (2015, January 7\u201313). FlowNet: Learning Optical Flow with Convolutional Networks. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.316"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1016\/j.inffus.2016.05.004","article-title":"Pixel-level image fusion: A survey of the state of the art","volume":"33","author":"Li","year":"2017","journal-title":"Inf. Fusion"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"101591","DOI":"10.1016\/j.isci.2020.101591","article-title":"Automatic Classification of Cichlid Behaviors Using 3D Convolutional Residual Networks","volume":"23","author":"Long","year":"2020","journal-title":"Iscience"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Carreira, J., and Zisserman, A. (2017, January 21\u201326). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.502"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"114","DOI":"10.1016\/j.compag.2018.02.006","article-title":"Near infrared computer vision and neuro-fuzzy model-based feeding decision system for fish in aquaculture","volume":"146","author":"Zhou","year":"2018","journal-title":"Comput. Electron. Agric."}],"container-title":["Animals"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-2615\/11\/10\/2774\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:03:56Z","timestamp":1760166236000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-2615\/11\/10\/2774"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,23]]},"references-count":27,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2021,10]]}},"alternative-id":["ani11102774"],"URL":"https:\/\/doi.org\/10.3390\/ani11102774","relation":{},"ISSN":["2076-2615"],"issn-type":[{"value":"2076-2615","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,23]]}}}