{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,8]],"date-time":"2026-06-08T12:34:27Z","timestamp":1780922067279,"version":"3.54.1"},"reference-count":47,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,4,13]],"date-time":"2022-04-13T00:00:00Z","timestamp":1649808000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Hungarian Ministry of Innovation and Technology NRDI Office","award":["NKFIH-1527-3\/2021, NKFIH-870-5\/2020"],"award-info":[{"award-number":["NKFIH-1527-3\/2021, NKFIH-870-5\/2020"]}]},{"name":"National Research - Development and Innovation Office, ELTE Thematic Excellence Programme","award":["TKP2020-IKA-05"],"award-info":[{"award-number":["TKP2020-IKA-05"]}]},{"name":"National Research, Development and Innovation Fund of Hungary, Thematic Excellence Programme","award":["2020-4.1.1.-TKP2020"],"award-info":[{"award-number":["2020-4.1.1.-TKP2020"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J. Imaging"],"abstract":"<jats:p>Identity tracking and instance segmentation are crucial in several areas of biological research. Behavior analysis of individuals in groups of similar animals is a task that emerges frequently in agriculture or pharmaceutical studies, among others. Automated annotation of many hours of surveillance videos can facilitate a large number of biological studies\/experiments, which otherwise would not be feasible. Solutions based on machine learning generally perform well in tracking and instance segmentation; however, in the case of identical, unmarked instances (e.g., white rats or mice), even state-of-the-art approaches can frequently fail. We propose a pipeline of deep generative models for identity tracking and instance segmentation of highly similar instances, which, in contrast to most region-based approaches, exploits edge information and consequently helps to resolve ambiguity in heavily occluded cases. Our method is trained by synthetic data generation techniques, not requiring prior human annotation. We show that our approach greatly outperforms other state-of-the-art unsupervised methods in identity tracking and instance segmentation of unmarked rats in real-world laboratory video recordings.<\/jats:p>","DOI":"10.3390\/jimaging8040109","type":"journal-article","created":{"date-parts":[[2022,4,13]],"date-time":"2022-04-13T21:33:42Z","timestamp":1649885622000},"page":"109","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Tracking Highly Similar Rat Instances under Heavy Occlusions: An Unsupervised Deep Generative Pipeline"],"prefix":"10.3390","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8223-960X","authenticated-orcid":false,"given":"Anna","family":"Gelencs\u00e9r-Horv\u00e1th","sequence":"first","affiliation":[{"name":"Faculty of Information Technology and Bionics, P\u00e1zm\u00e1ny P\u00e9ter Catholic University, Pr\u00e1ter utca 50\/A, 1083 Budapest, Hungary"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2387-2015","authenticated-orcid":false,"given":"L\u00e1szl\u00f3","family":"Kop\u00e1csi","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence, Faculty of Informatics, E\u00f6tv\u00f6s Lor\u00e1nd University, P\u00e1zm\u00e1ny P\u00e9ter S\u00e9t\u00e1ny 1\/C, 1117 Budapest, Hungary"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1665-969X","authenticated-orcid":false,"given":"Viktor","family":"Varga","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence, Faculty of Informatics, E\u00f6tv\u00f6s Lor\u00e1nd University, P\u00e1zm\u00e1ny P\u00e9ter S\u00e9t\u00e1ny 1\/C, 1117 Budapest, Hungary"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5105-2295","authenticated-orcid":false,"given":"D\u00e1vid","family":"Keller","sequence":"additional","affiliation":[{"name":"Laboratory of Neuromorphology, Department of Anatomy, Histology and Embryology, Semmelweis University, 1094 Budapest, Hungary"},{"name":"ELKH-ELTE Laboratory of Molecular and Systems Neurobiology, E\u00f6tv\u00f6s Lor\u00e1nd Research Network, E\u00f6tv\u00f6s Lor\u00e1nd University, 1000 Brussels, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0397-2991","authenticated-orcid":false,"given":"\u00c1rp\u00e1d","family":"Dobolyi","sequence":"additional","affiliation":[{"name":"ELKH-ELTE Laboratory of Molecular and Systems Neurobiology, E\u00f6tv\u00f6s Lor\u00e1nd Research Network, E\u00f6tv\u00f6s Lor\u00e1nd University, 1000 Brussels, Belgium"},{"name":"Department of Physiology and Neurobiology, E\u00f6tv\u00f6s Lor\u00e1nd University, P\u00e1zm\u00e1ny P\u00e9ter S\u00e9t\u00e1ny 1\/A, 1117 Budapest, Hungary"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Krist\u00f3f","family":"Karacs","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology and Bionics, P\u00e1zm\u00e1ny P\u00e9ter Catholic University, Pr\u00e1ter utca 50\/A, 1083 Budapest, Hungary"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1280-3447","authenticated-orcid":false,"given":"Andr\u00e1s","family":"L\u0151rincz","sequence":"additional","affiliation":[{"name":"Department of Artificial Intelligence, Faculty of Informatics, E\u00f6tv\u00f6s Lor\u00e1nd University, P\u00e1zm\u00e1ny P\u00e9ter S\u00e9t\u00e1ny 1\/C, 1117 Budapest, Hungary"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,4,13]]},"reference":[{"key":"ref_1","first-page":"113","article-title":"Laboratory Mice and Rats","volume":"2","author":"Johnson","year":"2012","journal-title":"Mater. Methods"},{"key":"ref_2","first-page":"207","article-title":"The Mighty Mouse: The Impact of Rodents on Advances in Biomedical Research","volume":"110","author":"Bryda","year":"2013","journal-title":"Mo Med."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1016\/S0278-6915(00)00149-6","article-title":"Strain differences in haematological response to chloroamphenicol succinate in mice: Implications for toxicological research","volume":"39","author":"Festing","year":"2001","journal-title":"Food Chem. Toxicol. Int. J. Publ. Br. Ind. Biol. Res. Assoc."},{"key":"ref_4","unstructured":"Gartner (2021, December 07). The 4 Trends That Prevail on the Gartner Hype Cycle for AI. Available online: https:\/\/www.gartner.com\/en\/articles\/the-4-trends-that-prevail-on-the-gartner-hype-cycle-for-ai-2021."},{"key":"ref_5","unstructured":"Biswas, D. (2021, December 07). Compositional AI: Fusion of AI\/ML Services. Available online: https:\/\/www.researchgate.net\/profile\/Debmalya-Biswas\/publication\/351037326_Compositional_AI_Fusion_of_AIML_Services\/links\/60806670907dcf667bb5a5e3\/Compositional-AI-Fusion-of-AI-ML-Services.pdf."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"He, K., Gkioxari, G., Doll\u00e1r, P., and Girshick, R. (2017, January 22\u201329). Mask R-CNN. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.322"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22\u201329). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkss. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.244"},{"key":"ref_8","unstructured":"Kop\u00e1csi, L., F\u00f3thi, \u00c1., and Lorincz, A. A Self-Supervised Method for Body Part Segmentation and Keypoint Detection of Rat Images. Ann. Univ. Sci. Budapest Sect. Comp., 2021. in press."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Kop\u00e1csi, L., Dobolyi, \u00c1., F\u00f3thi, \u00c1., Keller, D., Varga, V., and Lorincz, A. (2021, January 14\u201317). RATS: Robust Automated Tracking and Segmentation of Similar Instances. Proceedings of the Artificial Neural Networks and Machine Learning\u2014ICANN 2021, Bratislava, Slovakia.","DOI":"10.1007\/978-3-030-86365-4_41"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"e63377","DOI":"10.7554\/eLife.63377","article-title":"DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels","volume":"10","author":"Bohnslav","year":"2021","journal-title":"eLife"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Lauer, J., Zhou, M., Ye, S., Menegas, W., Nath, T., Rahman, M.M., Di Santo, V., Soberanes, D., Feng, G., and Murthy, V.N. (2021). Multi-animal pose estimation and tracking with DeepLabCut. bioRxiv.","DOI":"10.1101\/2021.04.30.442096"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Pereira, T., Tabris, N., Li, J., Ravindranath, S., Papadoyannis, E., Wang, Z., Turner, D., McKenzie-Smith, G., Kocher, S., and Falkner, A. (2020). SLEAP: Multi-animal pose tracking. bioRxiv.","DOI":"10.1101\/2020.08.31.276246"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Nilsson, S.R., Goodwin, N.L., Choong, J.J., Hwang, S., Wright, H.R., Norville, Z.C., Tong, X., Lin, D., Bentzley, B.S., and Eshel, N. (2020). Simple Behavioral Analysis (SimBA)\u2014An open source toolkit for computer classification of complex social behaviors in experimental animals. BioRxiv.","DOI":"10.1101\/2020.04.19.049452"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1942","DOI":"10.1038\/s41386-020-0776-y","article-title":"Deep learning-based behavioral analysis reaches human accuracy and is capable of outperforming commercial solutions","volume":"45","author":"Sturman","year":"2020","journal-title":"Neuropsychopharmacology"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"743","DOI":"10.1038\/nmeth.2994","article-title":"IdTracker: Tracking individuals in a group by automatic identification of unmarked animals","volume":"11","author":"Hinz","year":"2014","journal-title":"Nat. Methods"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"179","DOI":"10.1038\/s41592-018-0295-5","article-title":"Idtracker. ai: Tracking all individuals in small or large collectives of unmarked animals","volume":"16","author":"Bergomi","year":"2019","journal-title":"Nat. Methods"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"14774","DOI":"10.1038\/s41598-017-15104-2","article-title":"ToxId: An efficient algorithm to solve occlusions when tracking multiple animals","volume":"7","author":"Rodriguez","year":"2017","journal-title":"Sci. Rep."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1016\/S0166-4328(89)80102-0","article-title":"Home base behavior of rats (Rattus norvegicus) exploring a novel environment","volume":"34","author":"Eilam","year":"2016","journal-title":"Behav. Brain Res."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1038\/s41684-021-00811-1","article-title":"A review of 28 free animal-tracking software applications: Current features and limitations","volume":"50","author":"Panadeiro","year":"2021","journal-title":"Lab. Anim."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"460","DOI":"10.1111\/2041-210X.12874","article-title":"ToxTrac: A fast and robust software for tracking organisms","volume":"9","author":"Rodriguez","year":"2018","journal-title":"Methods Ecol. Evol."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Lv, X., Dai, C., Chen, L., Lang, Y., Tang, R., Huang, Q., and He, J. (2020). A Robust Real-Time Detecting and Tracking Framework for Multiple Kinds of Unmarked Object. Sensors, 20.","DOI":"10.3390\/s20010002"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"107194","DOI":"10.1016\/j.compeleceng.2021.107194","article-title":"A fast instance segmentation with one-stage multi-task deep neural network for autonomous driving","volume":"93","author":"Tseng","year":"2021","journal-title":"Comput. Electr. Eng."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Long, J., Shelhamer, E., and Darrell, T. (2015, January 7\u201312). Fully convolutional networks for semantic segmentation. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7298965"},{"key":"ref_24","unstructured":"Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A.L. (2016). DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"102795","DOI":"10.1016\/j.cviu.2019.102795","article-title":"Faster training of Mask R-CNN by focusing on instance boundaries","volume":"188","author":"Zimmermann","year":"2019","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1016\/j.media.2016.11.004","article-title":"DCAN: Deep contour-aware networks for object instance segmentation from histology images","volume":"36","author":"Chen","year":"2017","journal-title":"Med. Image Anal."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"264","DOI":"10.1016\/j.biosystemseng.2020.03.008","article-title":"Instance segmentation of apple flowers using the improved mask R\u2013CNN model","volume":"193","author":"Tian","year":"2020","journal-title":"Biosyst. Eng."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1109\/4.996","article-title":"Design of an image edge detection filter using the Sobel operator","volume":"23","author":"Kanopoulos","year":"1988","journal-title":"IEEE J. Solid-State Circ."},{"key":"ref_29","unstructured":"Fischler, M.A., and Firschein, O. (1987). A Computational Approach to Edge Detection. Readings in Computer Vision: Issues, Problems, Principles, and Paradigms, Kaufmann."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Lazarow, J., Lee, K., Shi, K., and Tu, Z. (2020, January 14\u201319). Learning Instance Occlusion for Panoptic Segmentation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.01073"},{"key":"ref_31","unstructured":"Poma, X.S., Riba, E., and Sappa, A. (2020, January 1\u20135). Dense Extreme Inception Network: Towards a Robust CNN Model for Edge Detection. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV), Snowmass Village, CO, USA."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Xie, S., and Tu, Z. (2015, January 7\u201313). Holistically-Nested Edge Detection. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.164"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Nazeri, K., Ng, E., Joseph, T., Qureshi, F., and Ebrahimi, M. (2019, January 27\u201328). EdgeConnect: Structure Guided Image Inpainting using Edge Prediction. Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, Seoul, Korea.","DOI":"10.1109\/ICCVW.2019.00408"},{"key":"ref_34","unstructured":"Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., and Girshick, R. (2021, December 07). Detectron2. Available online: https:\/\/github.com\/facebookresearch\/detectron2."},{"key":"ref_35","unstructured":"Rijsbergen, C.J.V. (1979). Information Retrieval, Butterworth-Heinemann. [2nd ed.]."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"F\u00f3thi, \u00c1., Farag\u00f3, K., Kop\u00e1csi, L., Milacski, Z., Varga, V., and Lorincz, A. (2020, January 23\u201327). Multi Object Tracking for Similar Instances: A Hybrid Architecture. Proceedings of the International Conference on Neural Information Processing, Bangkok, Thailand.","DOI":"10.1007\/978-3-030-63830-6_37"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1080\/10867651.2004.10487596","article-title":"An Image Inpainting Technique Based on the Fast Marching Method","volume":"9","author":"Telea","year":"2004","journal-title":"J. Graph. Tools"},{"key":"ref_38","unstructured":"Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative Adversarial Networks. Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_39","unstructured":"Murphy, K.P. (2013). Machine Learning: A Probabilistic Perspective, MIT Press."},{"key":"ref_40","unstructured":"Million, E. (2021, October 28). The Hadamard Product. Available online: http:\/\/buzzard.ups.edu\/courses\/2007spring\/projects\/million-paper.pdf."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"e453","DOI":"10.7717\/peerj.453","article-title":"Scikit-image: Image processing in Python","volume":"2","author":"Boulogne","year":"2014","journal-title":"PeerJ"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Kornilov, A.S., and Safonov, I.V. (2018). An Overview of Watershed Algorithm Implementations in Open Source Libraries. J. Imaging, 4.","DOI":"10.3390\/jimaging4100123"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Jiang, S., Campbell, D., Lu, Y., Li, H., and Hartley, R. (2021). Learning to Estimate Hidden Motions with Global Motion Aggregation. arXiv.","DOI":"10.1109\/ICCV48922.2021.00963"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Bewley, A., Ge, Z., Ott, L., Ramos, F., and Upcroft, B. (2016, January 25\u201328). Simple online and realtime tracking. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.","DOI":"10.1109\/ICIP.2016.7533003"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1007\/s11263-009-0275-4","article-title":"The PASCAL Visual Object Classes (VOC) challenge","volume":"88","author":"Everingham","year":"2010","journal-title":"Int. J. Comput. Vis."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Perazzi, F., Pont-Tuset, J., McWilliams, B., Gool, L.V., Gross, M., and Sorkine-Hornung, A. (2016, January 27\u201330). A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.85"},{"key":"ref_47","unstructured":"Poma, X.S., Sappa, \u00c1.D., Humanante, P., and Akbarinia, A. (2022, February 01). Dense Extreme Inception Network for Edge Detection. CoRR, Available online: http:\/\/xxx.lanl.gov\/abs\/2112.02250."}],"container-title":["Journal of Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2313-433X\/8\/4\/109\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:53:21Z","timestamp":1760136801000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2313-433X\/8\/4\/109"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,13]]},"references-count":47,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,4]]}},"alternative-id":["jimaging8040109"],"URL":"https:\/\/doi.org\/10.3390\/jimaging8040109","relation":{},"ISSN":["2313-433X"],"issn-type":[{"value":"2313-433X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,13]]}}}