{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:41:54Z","timestamp":1760179314926,"version":"build-2065373602"},"reference-count":56,"publisher":"MDPI AG","issue":"17","license":[{"start":{"date-parts":[[2020,9,2]],"date-time":"2020-09-02T00:00:00Z","timestamp":1599004800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100005632","name":"Narodowe Centrum Bada\u0144 i Rozwoju","doi-asserted-by":"publisher","award":["UOD-DEM-1-183\/001"],"award-info":[{"award-number":["UOD-DEM-1-183\/001"]}],"id":[{"id":"10.13039\/501100005632","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100007835","name":"Politechnika \u015al\u0105ska","doi-asserted-by":"publisher","award":["The research project (RAU-6, 2020) and projects for young scientists"],"award-info":[{"award-number":["The research project (RAU-6, 2020) and projects for young scientists"]}],"id":[{"id":"10.13039\/501100007835","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Tracking and action-recognition algorithms are currently widely used in video surveillance, monitoring urban activities and in many other areas. Their development highly relies on benchmarking scenarios, which enable reliable evaluations\/improvements of their efficiencies. Presently, benchmarking methods for tracking and action-recognition algorithms rely on manual annotation of video databases, prone to human errors, limited in size and time-consuming. Here, using gained experiences, an alternative benchmarking solution is presented, which employs methods and tools obtained from the computer-game domain to create simulated video data with automatic annotations. Presented approach highly outperforms existing solutions in the size of the data and variety of annotations possible to create. With proposed system, a potential user can generate a sequence of random images involving different times of day, weather conditions, and scenes for use in tracking evaluation. In the design of the proposed tool, the concept of crowd simulation is used and developed. The system is validated by comparisons to existing methods.<\/jats:p>","DOI":"10.3390\/s20174960","type":"journal-article","created":{"date-parts":[[2020,9,2]],"date-time":"2020-09-02T09:29:28Z","timestamp":1599038968000},"page":"4960","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Application of Crowd Simulations in the Evaluation of Tracking Algorithms"],"prefix":"10.3390","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9659-7451","authenticated-orcid":false,"given":"Micha\u0142","family":"Staniszewski","sequence":"first","affiliation":[{"name":"Department of Computer Graphics, Vision and Digital Systems, Faculty of Automatic Control, Electronics and Computer Science, Silesian University of Technology, Akademicka 2A, 44-100 Gliwice, Poland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5491-9096","authenticated-orcid":false,"given":"Pawe\u0142","family":"Foszner","sequence":"additional","affiliation":[{"name":"Department of Computer Graphics, Vision and Digital Systems, Faculty of Automatic Control, Electronics and Computer Science, Silesian University of Technology, Akademicka 2A, 44-100 Gliwice, Poland"}]},{"given":"Karol","family":"Kostorz","sequence":"additional","affiliation":[{"name":"Department of Computer Graphics, Vision and Digital Systems, Faculty of Automatic Control, Electronics and Computer Science, Silesian University of Technology, Akademicka 2A, 44-100 Gliwice, Poland"}]},{"given":"Agnieszka","family":"Michalczuk","sequence":"additional","affiliation":[{"name":"Department of Computer Graphics, Vision and Digital Systems, Faculty of Automatic Control, Electronics and Computer Science, Silesian University of Technology, Akademicka 2A, 44-100 Gliwice, Poland"}]},{"given":"Kamil","family":"Wereszczy\u0144ski","sequence":"additional","affiliation":[{"name":"Department of Computer Graphics, Vision and Digital Systems, Faculty of Automatic Control, Electronics and Computer Science, Silesian University of Technology, Akademicka 2A, 44-100 Gliwice, Poland"}]},{"given":"Micha\u0142","family":"Cogiel","sequence":"additional","affiliation":[{"name":"Department of Computer Graphics, Vision and Digital Systems, Faculty of Automatic Control, Electronics and Computer Science, Silesian University of Technology, Akademicka 2A, 44-100 Gliwice, Poland"}]},{"given":"Dominik","family":"Golba","sequence":"additional","affiliation":[{"name":"Department of Computer Graphics, Vision and Digital Systems, Faculty of Automatic Control, Electronics and Computer Science, Silesian University of Technology, Akademicka 2A, 44-100 Gliwice, Poland"}]},{"given":"Konrad","family":"Wojciechowski","sequence":"additional","affiliation":[{"name":"Polish-Japanese Academy of Information Technology, Koszykowa 86, 02-008 Warszawa, Poland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1793-9546","authenticated-orcid":false,"given":"Andrzej","family":"Pola\u0144ski","sequence":"additional","affiliation":[{"name":"Department of Computer Graphics, Vision and Digital Systems, Faculty of Automatic Control, Electronics and Computer Science, Silesian University of Technology, Akademicka 2A, 44-100 Gliwice, Poland"}]}],"member":"1968","published-online":{"date-parts":[[2020,9,2]]},"reference":[{"key":"ref_1","unstructured":"Bashir, F., and Porikli, F. (2006, January 13\u201316). Performance Evaluation of Object Detection and Tracking Systems. Proceedings of the IEEE International Workshop on Performance Evaluation of Tracking and Surveillance (PETS), Hyderabad, India."},{"key":"ref_2","unstructured":"Leal-Taix\u00e9, L., Milan, A., Reid, I., Roth, S., and Schindler, K. (2015). MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking. arXiv."},{"key":"ref_3","unstructured":"Milan, A., Leal-Taix\u00e9, L., Reid, I., Roth, S., and Schindler, K. (2016). MOT16: A Benchmark for Multi-Object Tracking. arXiv."},{"key":"ref_4","unstructured":"Dendorfer, P., Rezatofighi, H., Milan, A., Shi, J., Cremers, D., Reid, I., Roth, S., Schindler, K., and Leal-Taix\u00e9, L. (2019). CVPR19 Tracking and Detection Challenge: How crowded can it get?. arXiv."},{"key":"ref_5","unstructured":"Dendorfer, P., Rezatofighi, H., Milan, A., Shi, J., Cremers, D., Reid, I., Roth, S., Schindler, K., and Leal-Taix\u00e9, L. (2020). MOT20: A benchmark for multi object tracking in crowded scenes. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Geiger, A. (2013). Probabilistic Models for 3D Urban Scene Understanding from Movable Platforms. Pattern Anal. Mach. Intell.","DOI":"10.1109\/TPAMI.2013.185"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Kulbacki, M., Segen, J., Wojciechowski, S., Wereszczy\u0144ski, K., Nowacki, J.P., Drabik, A., and Wojciechowski, K. (2018). Intelligent Video Monitoring System with the Functionality of Online Recognition of People\u2019s Behavior and Interactions Between People. Lect. Notes Comput. Sci., 10752.","DOI":"10.1007\/978-3-319-75420-8_47"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"427","DOI":"10.1007\/978-3-662-49390-8_42","article-title":"Recent developments in tracking objects in a video sequence","volume":"9622","author":"Staniszewski","year":"2016","journal-title":"Lect. Notes Comput. Sci."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1007\/978-3-662-49390-8_41","article-title":"Selected Space-Time Based Methods for Action Recognition","volume":"9622","author":"Wojciechowski","year":"2016","journal-title":"Lect. Notes Comput. Sci."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Kim, S., Bera, A., Best, A., Chabra, R., and Manocha, D. (2016). Interactive and adaptive data-driven crowd simulation. IEEE Virtual Reality (VR), 29\u201338.","DOI":"10.1109\/VR.2016.7504685"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1109\/TVCG.2010.33","article-title":"Directing crowd simulations using navigation fields","volume":"17","author":"Patil","year":"2011","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"135:1","DOI":"10.1145\/2070781.2024169","article-title":"Interactive hybrid simulation of large-scale traffic","volume":"30","author":"Sewall","year":"2011","journal-title":"ACM Trans. Graph."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Bi, H., Mao, T., Wang, Z., and Deng, Z. (2019). A Deep Learning-based Framework for Intersectional Traffic Simulation and Editing. IEEE Trans. Vis. Comput. Graph.","DOI":"10.1109\/TVCG.2018.2889834"},{"key":"ref_14","first-page":"1","article-title":"Interactive Inverse Spatio-Temporal Crowd Motion Design","volume":"7","author":"Mathew","year":"2020","journal-title":"Symp. Interact. Graph. Games Assoc. Comput. Mach."},{"key":"ref_15","unstructured":"Ren, J., Xiang, W., Xiao, Y., Yang, R., Manocha, D., and Jin, X. (2019). Heter-Sim: Heterogeneous Multi-Agent Systems Simulation by Interactive Data-Driven Optimization. IEEE Trans. Vis. Comput. Graph."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1109\/TVCG.2011.133","article-title":"Simulating and Evaluating the Local Behavior of Small Pedestrian Groups","volume":"18","author":"Karamouzas","year":"2012","journal-title":"IEEE Trans. Vis. Comput. Graph."},{"key":"ref_17","first-page":"1757","article-title":"Multi-Agent System Simulation of Indoor Scenarios","volume":"5","author":"Pax","year":"2015","journal-title":"Ann. Comput. Sci. Inf. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"39","DOI":"10.18267\/j.aip.22","article-title":"A Review on Macroscopic Pedestrian Flow Modelling","volume":"2","year":"2013","journal-title":"Acta Inform. Pragensia"},{"key":"ref_19","unstructured":"(2020, September 01). Golaem Crowd Simulation. Available online: http:\/\/golaem.com\/content\/product\/golaem."},{"key":"ref_20","unstructured":"(2020, September 01). Massive Simulating Life. Available online: http:\/\/www.massivesoftware.com\/about.html."},{"key":"ref_21","unstructured":"Van Toll, W., Jaklin, N., and Geraerts, R. (2015, January 23). Towards believable crowds: A generic multi-level framework for agent navigation. Proceedings of the 20th Annual Conference of the Advanced School for Computing and Imaging, Amersfoort, The Netherlands."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"He, Z., Shi, M., and Li, C. (2016, January 26\u201329). Research and application of path-finding algorithm based on unity 3D. Proceedings of the 2016 IEEE\/ACIS 15th International Conference on Computer and Information Science (ICIS), Okayama, Japan.","DOI":"10.1109\/ICIS.2016.7550934"},{"key":"ref_23","unstructured":"Kristinsson, K.V. (2015). Social Navigation in Unity 3D. MSc Project. [Master\u2019s Thesis, School of Computer Science, Reykjavik University]."},{"key":"ref_24","unstructured":"Forbus, F.D., and Wright, W. (2020, September 01). Some Notes on Programming Objects in The SimsTM. Available online: http:\/\/www.qrg.northwestern.edu\/papers\/Files\/Programming_Objects_in_The_Sims.pdf."},{"key":"ref_25","first-page":"83","article-title":"Using computer vision to simulate the motion of virtual agents","volume":"18","author":"Musse","year":"2007","journal-title":"J. Vis. Comput. Animat."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kim, S., Bera, A., and Manocha, D. (2015, January 14\u201316). Interactive crowd content generation and analysis using trajectory-level behavior learning. Proceedings of theIEEE International Symposium on Multimedia (ISM), Miami, FL, USA.","DOI":"10.1109\/ISM.2015.89"},{"key":"ref_27","first-page":"125","article-title":"A*-based pathfinding in modern computer games","volume":"11","author":"Cui","year":"2011","journal-title":"Int. J. Comput. Sci. Netw. Secur."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Bochinski, E., Eiselein, V., and Sikora, T. (September, January 29). High-speed tracking-by- detection without using image information. Proceedings of the 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Lecce, Italy.","DOI":"10.1109\/AVSS.2017.8078516"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Bochinski, E., Senst, T., and Sikora, T. (2018, January 27\u201330). Extending IOU Based Multi-Object Tracking by Visual Information. Proceedings of the IEEE International Conference on Advanced Video and Signals-based Surveillance, Auckland, New Zealand.","DOI":"10.1109\/AVSS.2018.8639144"},{"key":"ref_30","unstructured":"Wen, L., Du, D., Cai, Z., Lei, Z., Chang, M., Qi, H., Lim, J., Yang, M., and Lyu, S. (2015). DETRAC: A new benchmark and protocol for multi-object detection and tracking. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Bae, S.-H., and Yoon, K.-J. (2014, January 23\u201328). Robust Online Multi-Object Tracking based on Tracklet Confidence and Online Discriminative Appearance Learning. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1109\/CVPR.2014.159"},{"key":"ref_32","unstructured":"(2020, September 01). Caviar Dataset. Available online: http:\/\/homepages.inf.ed.ac.uk\/rbf\/caviardata1\/."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Ess, A., Leibe, B., Schindler, K., and Van Gool, L. (2008, January 23\u201328). A mobile vision system for robust multi-person tracking. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.","DOI":"10.1109\/CVPR.2008.4587581"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"McLaughlin, N., Martinez Del Rincon, J., and Miller, P. (2015, January 5\u20139). Enhancing Linear Programming with Motion Modelling for Multi-target Tracking. Proceedings of the 2015 IEEE Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.","DOI":"10.1109\/WACV.2015.17"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Benfold, B., and Reid, I. (2011, January 20\u201325). Stable multi-target tracking in real-time surveillance video. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.","DOI":"10.1109\/CVPR.2011.5995667"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"623","DOI":"10.1109\/TCSVT.2014.2344509","article-title":"Tracking multiple high-density homogeneous targets","volume":"25","author":"Poiesi","year":"2014","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_37","unstructured":"(2020, September 01). TUD Dataset. Available online: https:\/\/www.d2.mpi-inf.mpg.de\/node\/428."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"2054","DOI":"10.1109\/TPAMI.2015.2505309","article-title":"Multi-Target Tracking by Discrete- Continuous Energy Minimization","volume":"38","author":"Milan","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach Intell."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Henschel, R., Zou, Y., and Rosenhahn, B. (2019, January 16\u201317). Multiple People Tracking using Body and Joint Detections. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00105"},{"key":"ref_40","unstructured":"Chu, P., and Ling, H. (November, January 27). FAMNet: Joint Learning of Feature, Affinity and Multi-Dimensional Assignment for Online Multiple Object Tracking. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Seoul, Korea."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"3269","DOI":"10.1109\/TCSVT.2018.2882192","article-title":"Heterogeneous Association Graph Fusion for Target Association in Multiple Object Tracking","volume":"29","author":"Sheng","year":"2018","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"38060","DOI":"10.1109\/ACCESS.2020.2975912","article-title":"OneShotDA: Online Multi-Object Tracker with One-Shot-Learning-Based Data Association","volume":"8","author":"Yoon","year":"2020","journal-title":"IEEE Access"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Henschel, R., Leal-Taixe, L., Cremers, D., and Rosenhahn, B. (2018, January 19\u201321). Fusion of Head and Full-Body Detectors for Multi-Object Tracking. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Salt Lake, UT, USA.","DOI":"10.1109\/CVPRW.2018.00192"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1109\/TPAMI.2018.2876253","article-title":"Motion Segmentation and Multiple Object Tracking by Correlation Co-Clustering","volume":"42","author":"Keuper","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"3660","DOI":"10.1109\/TCSVT.2018.2881123","article-title":"Iterative Multiple Hypothesis Tracking with Tracklet-Level Association","volume":"29","author":"Sheng","year":"2019","journal-title":"IEEE Trans. Circuits Syst. Video Technol."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"2277","DOI":"10.1109\/TMM.2019.2902480","article-title":"Multi-Level Cooperative Fusion of GM-PHD Filters for Online Multiple Human Tracking","volume":"21","author":"Fu","year":"2019","journal-title":"IEEE Trans. Multimed."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Kim, C., Li, F., and Rehg, J.M. (2018, January 8\u201314). Multi-object Tracking with Neural Gating Using Bilinear LSTM. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01237-3_13"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"104423","DOI":"10.1109\/ACCESS.2019.2932301","article-title":"Multiple Object Tracking With Attention to Appearance, Structure, Motion and Size","volume":"7","author":"Karunasekera","year":"2019","journal-title":"IEEE Access"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Bergmann, P., Meinhardt, T., and Leal-Taixe, L. (2019). Tracking without bells and whistles. arXiv.","DOI":"10.1109\/ICCV.2019.00103"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Wang, G., Wang, Y., Zhang, H., Gu, R., and Hwang, J. (2019, January 21\u201325). Exploit the connectivity: Multi-object tracking with trackletnet. Proceedings of the 27th ACM International Conference on Multimedia, Nice, France.","DOI":"10.1145\/3343031.3350853"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Xu, Y., Osep, A., Ban, Y., Horaud, R., Leal-Taixe, L., and Alameda-Pineda, X. (2019). How To Train Your Deep Multi-Object Tracker. arXiv.","DOI":"10.1109\/CVPR42600.2020.00682"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Rezatofighi, S.H., Milan, A., Zhang, Z., Shi, Q., Dick, A., and Reid, I. (2015, January 7\u201313). Joint Probabilistic Data Association Revisited. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.349"},{"key":"ref_53","unstructured":"(2020, September 01). Kitti Dataset. Available online: http:\/\/www.cvlibs.net\/datasets\/kitti\/eval_tracking.php."},{"key":"ref_54","unstructured":"(2020, September 01). Average Frame Rate Video Surveillance 2019 by: IPVM Team. Published on 23 May 2019. Available online: https:\/\/ipvm.com\/reports\/avg-frame-rate-2019."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Keval, H., and Sasse, M.A. (2008, January 27\u201331). To catch a thief\u2014You need at least 8 frames per second: The impact of frame rates on user performance in a CCTV detection task. Proceedings of the 16th ACM International Conference on Multimedia, Vancouver, BC, Canada.","DOI":"10.1145\/1459359.1459527"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Choi, W. (2015, January 7\u201313). Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.347"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/17\/4960\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:05:48Z","timestamp":1760177148000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/20\/17\/4960"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,9,2]]},"references-count":56,"journal-issue":{"issue":"17","published-online":{"date-parts":[[2020,9]]}},"alternative-id":["s20174960"],"URL":"https:\/\/doi.org\/10.3390\/s20174960","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2020,9,2]]}}}