{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T06:57:32Z","timestamp":1789109852324,"version":"build-2803163510"},"reference-count":27,"publisher":"Public Library of Science (PLoS)","issue":"2","license":[{"start":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T00:00:00Z","timestamp":1771545600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000270","name":"Natural Environment Research Council","doi-asserted-by":"publisher","award":["NE\/W006359\/1"],"award-info":[{"award-number":["NE\/W006359\/1"]}],"id":[{"id":"10.13039\/501100000270","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000270","name":"Natural Environment Research Council","doi-asserted-by":"crossref","award":["NE\/Z000114\/1"],"award-info":[{"award-number":["NE\/Z000114\/1"]}],"id":[{"id":"10.13039\/501100000270","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100000270","name":"Natural Environment Research Council","doi-asserted-by":"publisher","award":["NE\/W006359\/1"],"award-info":[{"award-number":["NE\/W006359\/1"]}],"id":[{"id":"10.13039\/501100000270","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000270","name":"Natural Environment Research Council","doi-asserted-by":"publisher","award":["NE\/W006359\/1"],"award-info":[{"award-number":["NE\/W006359\/1"]}],"id":[{"id":"10.13039\/501100000270","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000270","name":"Natural Environment Research Council","doi-asserted-by":"publisher","award":["NE\/Z000114\/1"],"award-info":[{"award-number":["NE\/Z000114\/1"]}],"id":[{"id":"10.13039\/501100000270","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000270","name":"Natural Environment Research Council","doi-asserted-by":"publisher","award":["NE\/Z000114\/1"],"award-info":[{"award-number":["NE\/Z000114\/1"]}],"id":[{"id":"10.13039\/501100000270","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosbiology.org"],"crossmark-restriction":false},"short-container-title":["PLoS Biol"],"abstract":"<jats:p>\n                    Here we introduce BehaveAI, a biologically inspired video analysis framework that integrates static and motion information through a novel color-from-motion encoding strategy. This method translates object movement\u2014direction, speed, and acceleration\u2014into color gradients, enabling both human annotators and pre-trained convolutional neural networks (CNNs) to infer motion patterns while retaining high-resolution spatial detail. Using a range of case studies, we demonstrate how the increased salience of motion information allows for the robust detection of objects that are challenging or impossible to identify reliably from static frames alone, particularly in complex natural scenes. We further demonstrate the reliable classification of different behaviors in animals and single-celled organisms. Additionally, the framework supports flexible hierarchical model structures that can separate the tasks of detection and classification for optimal efficiency, and provide individual tracking data that specifies\n                    <jats:italic>what<\/jats:italic>\n                    is present\n                    <jats:italic>where<\/jats:italic>\n                    and what it is\n                    <jats:italic>doing<\/jats:italic>\n                    in each frame. The framework makes use of the latest deep learning architecture (YOLO11), combined with a semi-supervised annotation workflow. Together with salient motion information, these features can dramatically reduce the effort required for dataset annotation such that reliable models can often be made within an hour. Moreover, smaller annotation datasets mean that model training can be achieved on conventional computers without dedicated hardware, thereby improving accessibility. The motion encoding approach is also computationally lightweight, and can run in real-time on low-end edge devices such as a Raspberry Pi. We release the framework as a free, open source, and user-friendly package.\n                  <\/jats:p>","DOI":"10.1371\/journal.pbio.3003632","type":"journal-article","created":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T18:23:23Z","timestamp":1771611803000},"page":"e3003632","update-policy":"https:\/\/doi.org\/10.1371\/journal.pbio.corrections_policy","source":"Crossref","is-referenced-by-count":4,"title":["BehaveAI enables rapid detection and classification of objects and behavior from motion"],"prefix":"10.1371","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9071-2594","authenticated-orcid":true,"given":"Jolyon","family":"Troscianko","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas A.","family":"O\u2019Shea-Wheller","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4824-3728","authenticated-orcid":true,"given":"James A. M.","family":"Galloway","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kevin J.","family":"Gaston","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,2,20]]},"reference":[{"key":"pbio.3003632.ref001","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1016\/j.cosrev.2014.04.001","article-title":"Traditional and recent approaches in background modeling for foreground detection: an overview","author":"T Bouwmans","year":"2014","journal-title":"Comput Sci Rev"},{"key":"pbio.3003632.ref002","article-title":"Imagenet classification with deep convolutional neural networks","volume":"25","author":"A Krizhevsky","year":"2012","journal-title":"Adv Neural Inform Process Syst"},{"issue":"6","key":"pbio.3003632.ref003","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster R-CNN: towards real-time object detection with region proposal networks","volume":"39","author":"S Ren","year":"2017","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"issue":"4","key":"pbio.3003632.ref004","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1111\/2041-210X.14502","article-title":"YOLO\u2010behaviour: a simple, flexible framework to automatically quantify animal behaviours from videos","volume":"16","author":"AHH Chan","year":"2025","journal-title":"Methods Ecol Evol"},{"key":"pbio.3003632.ref005","doi-asserted-by":"crossref","first-page":"11779322211037770","DOI":"10.1177\/11779322211037770","article-title":"Deep learning approach for quantification of fluorescently labeled blood cells in Danio rerio (Zebrafish)","volume":"15","author":"S Thapa","year":"2021","journal-title":"Bioinform Biol Insights"},{"issue":"1516","key":"pbio.3003632.ref006","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1098\/rstb.2008.0218","article-title":"Camouflage and visual perception","volume":"364","author":"T Troscianko","year":"2009","journal-title":"Philos Trans R Soc Lond B Biol Sci"},{"issue":"2","key":"pbio.3003632.ref007","doi-asserted-by":"crossref","first-page":"201","DOI":"10.3758\/BF03212378","article-title":"Visual perception of biological motion and a model for its analysis","volume":"14","author":"G Johansson","year":"1973","journal-title":"Percept Psychophys"},{"key":"pbio.3003632.ref008","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.63377","article-title":"DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels","volume":"10","author":"JP Bohnslav","year":"2021","journal-title":"Elife"},{"issue":"3","key":"pbio.3003632.ref009","doi-asserted-by":"crossref","first-page":"100415","DOI":"10.1016\/j.crmeth.2023.100415","article-title":"LabGym: quantification of user-defined animal behaviors using learning-based holistic assessment","volume":"3","author":"Y Hu","year":"2023","journal-title":"Cell Rep Methods"},{"key":"pbio.3003632.ref010","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.47994","article-title":"DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning","volume":"8","author":"JM Graving","year":"2019","journal-title":"Elife"},{"issue":"9","key":"pbio.3003632.ref011","doi-asserted-by":"crossref","first-page":"1281","DOI":"10.1038\/s41593-018-0209-y","article-title":"DeepLabCut: markerless pose estimation of user-defined body parts with deep learning","volume":"21","author":"A Mathis","year":"2018","journal-title":"Nat Neurosci"},{"issue":"4","key":"pbio.3003632.ref012","doi-asserted-by":"crossref","first-page":"486","DOI":"10.1038\/s41592-022-01426-1","article-title":"SLEAP: a deep learning system for multi-animal pose tracking","volume":"19","author":"TD Pereira","year":"2022","journal-title":"Nat Methods"},{"issue":"3","key":"pbio.3003632.ref013","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1007\/s10211-022-00397-y","article-title":"Using machine learning and DeepLabCut in animal behavior","volume":"25","author":"A Hardin","year":"2022","journal-title":"acta ethol"},{"issue":"7","key":"pbio.3003632.ref014","doi-asserted-by":"crossref","first-page":"1329","DOI":"10.1038\/s41592-024-02318-2","article-title":"Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics","volume":"21","author":"C Weinreb","year":"2024","journal-title":"Nat Methods"},{"key":"pbio.3003632.ref015","doi-asserted-by":"crossref","unstructured":"Carreira J, Zisserman A. Quo vadis, action recognition? A new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 6299\u2013308.","DOI":"10.1109\/CVPR.2017.502"},{"key":"pbio.3003632.ref016","article-title":"Two-stream convolutional networks for action recognition in videos","volume":"27","author":"K Simonyan","year":"2014","journal-title":"Adv Neural Inform Process Syst"},{"key":"pbio.3003632.ref017","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780198524724.001.0001","volume-title":"The visual brain in action","author":"D Milner","year":"2006"},{"issue":"1","key":"pbio.3003632.ref018","doi-asserted-by":"crossref","DOI":"10.1007\/s12283-023-00437-1","article-title":"Time-weighted motion history image for human activity classification in sports","volume":"26","author":"H Komori","year":"2023","journal-title":"Sports Eng"},{"key":"pbio.3003632.ref019","unstructured":"Raji ID, Bender EM, Paullada A, Denton E, Hanna A. AI and the everything in the whole wide world benchmark. arXiv:2111.15366. arXiv. 2021. https:\/\/doi.org\/10.48550\/arXiv.2111.15366"},{"key":"pbio.3003632.ref020","unstructured":"Jocher G, Qiu J, Chaurasia A. Ultralytics YOLO (Version 8.0.0) [Computer software]. 2023. Available from: https:\/\/ultralytics.com"},{"issue":"1","key":"pbio.3003632.ref021","doi-asserted-by":"crossref","first-page":"260","DOI":"10.1038\/s41597-023-02173-4","article-title":"VISEM-Tracking, a human spermatozoa tracking dataset","volume":"10","author":"V Thambawita","year":"2023","journal-title":"Sci Data"},{"key":"pbio.3003632.ref022","unstructured":"Chan AHH, Brookes O, Waldmann U, Naik H, Couzin ID, Mirmehdi M, et al. Towards application-specific evaluation of vision models: Case studies in ecology and biology. In: arXiv, 2025. https:\/\/doi.org\/10.48550\/arXiv.2505.02825"},{"key":"pbio.3003632.ref023","doi-asserted-by":"crossref","unstructured":"Beery S, Horn G van, Perona P. Recognition in terra incognita. arXiv, 2018. https:\/\/doi.org\/10.48550\/arXiv.1807.04975","DOI":"10.1007\/978-3-030-01270-0_28"},{"key":"pbio.3003632.ref024","doi-asserted-by":"crossref","unstructured":"Naik H, Hang Chan AH, Yang J, Delacoux M, Couzin ID, Kano F, et al. 3D-POP: an automated annotation approach to facilitate markerless 2D-3D tracking of freely moving birds with marker-based motion capture. In: 2023 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 21274\u201384. https:\/\/doi.org\/10.1109\/cvpr52729.2023.02038","DOI":"10.1109\/CVPR52729.2023.02038"},{"key":"pbio.3003632.ref025","doi-asserted-by":"crossref","unstructured":"Haugen TB, Hicks SA, Andersen JM, Witczak O, Hammer HL, Borgli R, et al. VISEM. In: Proceedings of the 10th ACM Multimedia Systems Conference, 2019. 261\u20136. https:\/\/doi.org\/10.1145\/3304109.3325814","DOI":"10.1145\/3304109.3325814"},{"issue":"2000","key":"pbio.3003632.ref026","first-page":"20230725","article-title":"Artificial light at night causes conflicting behavioural and morphological defence responses in a marine isopod","volume":"290","author":"K Bullough","year":"2023","journal-title":"Proc Biol Sci"},{"key":"pbio.3003632.ref027","unstructured":"Welch G, Bishop G. An introduction to the Kalman filter. 1995."}],"container-title":["PLOS Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pbio.3003632","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T13:40:15Z","timestamp":1772113215000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pbio.3003632"}},"subtitle":[],"editor":[{"given":"Gail L","family":"Patricelli","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,2,20]]},"references-count":27,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2026,2,20]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pbio.3003632","relation":{},"ISSN":["1545-7885"],"issn-type":[{"value":"1545-7885","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,20]]}}}