{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T05:18:55Z","timestamp":1769577535712,"version":"3.49.0"},"reference-count":39,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2021,12,16]],"date-time":"2021-12-16T00:00:00Z","timestamp":1639612800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Future Internet"],"abstract":"<jats:p>Machine Learning (ML) is now becoming a key driver empowering the next generation of drone technology and extending its reach to applications never envisioned before. Examples include precision agriculture, crowd detection, and even aerial supply transportation. Testing drone projects before actual deployment is usually performed via robotic simulators. However, extending testing to include the assessment of on-board ML algorithms is a daunting task. ML practitioners are now required to dedicate vast amounts of time for the development and configuration of the benchmarking infrastructure through a mixture of use-cases coded over the simulator to evaluate various key performance indicators. These indicators extend well beyond the accuracy of the ML algorithm and must capture drone-relevant data including flight performance, resource utilization, communication overhead and energy consumption. As most ML practitioners are not accustomed with all these demanding requirements, the evaluation of ML-driven drone applications can lead to sub-optimal, costly, and error-prone deployments. In this article we introduce FlockAI, an open and modular by design framework supporting ML practitioners with the rapid deployment and repeatable testing of ML-driven drone applications over the Webots simulator. To show the wide applicability of rapid testing with FlockAI, we introduce a proof-of-concept use-case encompassing different scenarios, ML algorithms and KPIs for pinpointing crowded areas in an urban environment.<\/jats:p>","DOI":"10.3390\/fi13120317","type":"journal-article","created":{"date-parts":[[2021,12,16]],"date-time":"2021-12-16T11:27:36Z","timestamp":1639654056000},"page":"317","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["FlockAI: A Testing Suite for ML-Driven Drone Applications"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9540-7342","authenticated-orcid":false,"given":"Demetris","family":"Trihinas","sequence":"first","affiliation":[{"name":"Department of Computer Science, School of Sciences and Engineering, University of Nicosia, Nicosia CY-2417, Cyprus"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michalis","family":"Agathocleous","sequence":"additional","affiliation":[{"name":"Department of Computer Science, School of Sciences and Engineering, University of Nicosia, Nicosia CY-2417, Cyprus"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Karlen","family":"Avogian","sequence":"additional","affiliation":[{"name":"Department of Computer Science, School of Sciences and Engineering, University of Nicosia, Nicosia CY-2417, Cyprus"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5283-9965","authenticated-orcid":false,"given":"Ioannis","family":"Katakis","sequence":"additional","affiliation":[{"name":"Department of Computer Science, School of Sciences and Engineering, University of Nicosia, Nicosia CY-2417, Cyprus"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,12,16]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Cui, J.Q., Phang, S.K., Ang, K.Z.Y., Wang, F., Dong, X., Ke, Y., Lai, S., Li, K., Li, X., and Lin, F. (2015, January 15\u201317). Drones for cooperative search and rescue in post-disaster situation. Proceedings of the 2015 IEEE 7th International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM), Siem Reap, Cambodia.","DOI":"10.1109\/ICCIS.2015.7274615"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1109\/MPOT.2018.2850386","article-title":"Drones: Augmenting Our Quality of Life","volume":"38","author":"Kyrkou","year":"2019","journal-title":"IEEE Potentials"},{"key":"ref_3","first-page":"183","article-title":"ADDSEN: Adaptive Data Processing and Dissemination for Drone Swarms in Urban Sensing","volume":"66","author":"Wu","year":"2017","journal-title":"IEEE Trans. Comput."},{"key":"ref_4","first-page":"507","article-title":"Agriculture drones: A modern breakthrough in precision agriculture","volume":"20","author":"Puri","year":"2017","journal-title":"J. Stat. Manag. Syst."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"769","DOI":"10.1016\/j.comcom.2020.07.011","article-title":"An architecture for adaptive task planning in support of IoT-based machine learning applications for disaster scenarios","volume":"160","author":"Sacco","year":"2020","journal-title":"Comput. Commun."},{"key":"ref_6","unstructured":"Tseng, C., Chau, C., Elbassioni, K.M., and Khonji, M. (2017). Flight Tour Planning with Recharging Optimization for Battery-operated Autonomous Drones. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"58383","DOI":"10.1109\/ACCESS.2018.2875040","article-title":"Comprehensive Energy Consumption Model for Unmanned Aerial Vehicles, Based on Empirical Studies of Battery Performance","volume":"6","author":"Abeywickrama","year":"2018","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"487","DOI":"10.1109\/TSC.2018.2808956","article-title":"Low-cost adaptive monitoring techniques for the internet of things","volume":"14","author":"Trihinas","year":"2018","journal-title":"IEEE Trans. Serv. Comput."},{"key":"ref_9","unstructured":"Dimitropoulos, S. (2021, November 09). If One Drone Isn\u2019t Enough, Try a Drone Swarm. Available online: https:\/\/www.bbc.com\/news\/business-49177704."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"V\u00e1s\u00e1rhelyi, G., Vir\u00e1gh, C., Somorjai, G., Nepusz, T., Eiben, A.E., and Vicsek, T. (2018). Optimized flocking of autonomous drones in confined environments. Sci. Robot., 3.","DOI":"10.1126\/scirobotics.aat3536"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"4523","DOI":"10.1109\/LRA.2019.2935377","article-title":"Learning Vision-Based Flight in Drone Swarms by Imitation","volume":"4","author":"Schilling","year":"2019","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"138669","DOI":"10.1109\/ACCESS.2019.2942944","article-title":"Machine Learning-Based Drone Detection and Classification: State-of-the-Art in Research","volume":"7","author":"Taha","year":"2019","journal-title":"IEEE Access"},{"key":"ref_13","first-page":"36","article-title":"When UAV Swarm Meets Edge-Cloud Computing: The QoS Perspective","volume":"33","author":"Chen","year":"2019","journal-title":"IEEE Netw."},{"key":"ref_14","unstructured":"(2021, November 09). Commercial Drones Are Here: The Future of Unmanned Aerial Systems. Available online: https:\/\/www.mckinsey.com\/industries\/travel-logistics-and-infrastructure\/our-insights\/commercial-drones-are-here-the-future-of-unmanned-aerial-systems."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"2015","DOI":"10.1007\/s00521-020-05097-x","article-title":"Deep reinforcement learning for drone navigation using sensor data","volume":"33","author":"Hodge","year":"2021","journal-title":"Neural Comput. Appl."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"San Juan, V., Santos, M., And\u00fajar, J.M., and Volchenkov, D. (2018). Intelligent UAV Map Generation and Discrete Path Planning for Search and Rescue Operations. Complex, 2018.","DOI":"10.1155\/2018\/6879419"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1109\/MCOM.2017.1700442","article-title":"Key Technologies and System Trade-offs for Detection and Localization of Amateur Drones","volume":"56","author":"Azari","year":"2018","journal-title":"IEEE Commun. Mag."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Shen, L., Wang, X., and Hu, H.M. (2020, January 6\u20138). Drone Video Object Detection using Convolutional Neural Networks with Time Domain Motion Features. Proceedings of the 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), Shenzhen, China.","DOI":"10.1109\/MIPR49039.2020.00039"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Besada, J.A., Bergesio, L., Campa\u00f1a, I., Vaquero-Melchor, D., L\u00f3pez-Araquistain, J., Bernardos, A.M., and Casar, J.R. (2018). Drone mission definition and implementation for automated infrastructure inspection using airborne sensors. Sensors, 18.","DOI":"10.3390\/s18041170"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.future.2020.08.046","article-title":"A drone-based networked system and methods for combating coronavirus disease (COVID-19) pandemic","volume":"115","author":"Kumar","year":"2021","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"2664","DOI":"10.1109\/LCOMM.2020.3013286","article-title":"Performance Evaluation of UAV-enabled Cellular Networks with Battery-limited Drones","volume":"24","author":"Qin","year":"2020","journal-title":"IEEE Commun. Lett."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Symeonides, M., Georgiou, Z., Trihinas, D., Pallis, G., and Dikaiakos, M.D. (2020, January 12\u201314). Fogify: A Fog Computing Emulation Framework. Proceedings of the 2020 IEEE\/ACM Symposium on Edge Computing (SEC), San Jose, CA, USA.","DOI":"10.1109\/SEC50012.2020.00011"},{"key":"ref_23","unstructured":"(2021, November 09). Webots Robot Simulator. Available online: https:\/\/cyberbotics.com\/."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Trihinas, D., Agathocleous, M., and Avogian, K. (2021, January 4\u20138). Composable Energy Modeling for ML-Driven Drone Applications. Proceedings of the 2021 IEEE International Conference on Cloud Engineering (IC2E), San Francisco, CA, USA.","DOI":"10.1109\/IC2E52221.2021.00039"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Tropea, M., Fazio, P., De Rango, F., and Cordeschi, N. (2020). A New FANET Simulator for Managing Drone Networks and Providing Dynamic Connectivity. Electronics, 9.","DOI":"10.3390\/electronics9040543"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"3747","DOI":"10.1109\/TWC.2017.2688328","article-title":"Energy-Efficient UAV Communication With Trajectory Optimization","volume":"16","author":"Zeng","year":"2017","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Marins, J.L., Cabreira, T.M., Kappel, K.S., and Ferreira, P.R. (2018, January 5\u20138). A Closed-Form Energy Model for Multi-rotors Based on the Dynamic of the Movement. Proceedings of the 2018 VIII Brazilian Symposium on Computing Systems Engineering (SBESC), Salvador, Brazil.","DOI":"10.1109\/SBESC.2018.00047"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"800","DOI":"10.1109\/TCC.2015.2511760","article-title":"Monitoring elastically adaptive multi-cloud services","volume":"6","author":"Trihinas","year":"2016","journal-title":"IEEE Trans. Cloud Comput."},{"key":"ref_29","unstructured":"(2021, November 09). VisDrone ECCV 2020 Crowd Counting Challenge. Available online: http:\/\/aiskyeye.com\/challenge\/crowd-counting\/."},{"key":"ref_30","unstructured":"Geitgey, A. (2021, November 09). Python Face Recognition Library. Available online: https:\/\/github.com\/ageitgey\/face_recognition."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hsu, H.J., and Chen, K.T. (2017, January 20\u201323). DroneFace: An Open Dataset for Drone Research. Proceedings of the 8th ACM on Multimedia Systems Conference, Taipei, Taiwan. MMSys\u201917.","DOI":"10.1145\/3083187.3083214"},{"key":"ref_32","unstructured":"(2021, November 09). TensorFlow. Face Detection using Frozen Inference Graphs, Available online: https:\/\/www.tensorflow.org\/."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MIM.2017.8121945","article-title":"DronesBench: An innovative bench to test drones","volume":"20","author":"Daponte","year":"2017","journal-title":"IEEE Instrum. Meas. Mag."},{"key":"ref_34","unstructured":"(2021, November 09). jMavSim. Available online: https:\/\/docs.px4.io\/master\/en\/."},{"key":"ref_35","unstructured":"Koubaa, A. (2016). RotorS\u2014A Modular Gazebo MAV Simulator Framework. Robot Operating System (ROS): The Complete Reference, Springer International Publishing."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1007\/s10846-021-01383-5","article-title":"The MRS UAV System: Pushing the Frontiers of Reproducible Research, Real-world Deployment, and Education with Autonomous Unmanned Aerial Vehicles","volume":"102","author":"Vrba","year":"2021","journal-title":"J. Intell. Robotic Syst."},{"key":"ref_37","first-page":"1729881419870937","article-title":"UTSim: A framework and simulator for UAV air traffic integration, control, and communication","volume":"16","author":"Sababha","year":"2019","journal-title":"Int. J. Adv. Robot. Syst."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Shah, S., Dey, D., Lovett, C., and Kapoor, A. (2018). Airsim: High-fidelity visual and physical simulation for autonomous vehicles. Field and Service Robotics, Springer.","DOI":"10.1007\/978-3-319-67361-5_40"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Maglogiannis, I., Iliadis, L., and Pimenidis, E. (2020). Deepbots: A Webots-Based Deep Reinforcement Learning Framework for Robotics. Artificial Intelligence Applications and Innovations, Springer International Publishing.","DOI":"10.1007\/978-3-030-49186-4"}],"container-title":["Future Internet"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-5903\/13\/12\/317\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:50:12Z","timestamp":1760169012000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-5903\/13\/12\/317"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,16]]},"references-count":39,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["fi13120317"],"URL":"https:\/\/doi.org\/10.3390\/fi13120317","relation":{},"ISSN":["1999-5903"],"issn-type":[{"value":"1999-5903","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,16]]}}}