{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T16:27:57Z","timestamp":1781713677690,"version":"3.54.5"},"reference-count":40,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2021,10,5]],"date-time":"2021-10-05T00:00:00Z","timestamp":1633392000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000844","name":"European Space Agency","doi-asserted-by":"publisher","award":["GENESIS"],"award-info":[{"award-number":["GENESIS"]}],"id":[{"id":"10.13039\/501100000844","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005632","name":"Narodowe Centrum Bada\u0144 i Rozwoju","doi-asserted-by":"publisher","award":["POIR.01.01.01-00-0356\/17"],"award-info":[{"award-number":["POIR.01.01.01-00-0356\/17"]}],"id":[{"id":"10.13039\/501100005632","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100007835","name":"Silesian University of Technology","doi-asserted-by":"publisher","award":["Grant for maintaining and developing research potential"],"award-info":[{"award-number":["Grant for maintaining and developing research potential"]}],"id":[{"id":"10.13039\/501100007835","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000016","name":"Canadian Space Agency","doi-asserted-by":"publisher","award":["CSA Grant"],"award-info":[{"award-number":["CSA Grant"]}],"id":[{"id":"10.13039\/501100000016","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Benchmarking deep learning algorithms before deploying them in hardware-constrained execution environments, such as imaging satellites, is pivotal in real-life applications. Although a thorough and consistent benchmarking procedure can allow us to estimate the expected operational abilities of the underlying deep model, this topic remains under-researched. This paper tackles this issue and presents an end-to-end benchmarking approach for quantifying the abilities of deep learning algorithms in virtually any kind of on-board space applications. The experimental validation, performed over several state-of-the-art deep models and benchmark datasets, showed that different deep learning techniques may be effectively benchmarked using the standardized approach, which delivers quantifiable performance measures and is highly configurable. We believe that such benchmarking is crucial in delivering ready-to-use on-board artificial intelligence in emerging space applications and should become a standard tool in the deployment chain.<\/jats:p>","DOI":"10.3390\/rs13193981","type":"journal-article","created":{"date-parts":[[2021,10,8]],"date-time":"2021-10-08T21:26:20Z","timestamp":1633728380000},"page":"3981","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":34,"title":["Benchmarking Deep Learning for On-Board Space Applications"],"prefix":"10.3390","volume":"13","author":[{"given":"Maciej","family":"Ziaja","sequence":"first","affiliation":[{"name":"KP Labs, Konarskiego 18C, 44-100 Gliwice, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Piotr","family":"Bosowski","sequence":"additional","affiliation":[{"name":"KP Labs, Konarskiego 18C, 44-100 Gliwice, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9265-3997","authenticated-orcid":false,"given":"Michal","family":"Myller","sequence":"additional","affiliation":[{"name":"KP Labs, Konarskiego 18C, 44-100 Gliwice, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Grzegorz","family":"Gajoch","sequence":"additional","affiliation":[{"name":"KP Labs, Konarskiego 18C, 44-100 Gliwice, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michal","family":"Gumiela","sequence":"additional","affiliation":[{"name":"KP Labs, Konarskiego 18C, 44-100 Gliwice, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Protich","sequence":"additional","affiliation":[{"name":"GSTS\u2014Global Spatial Technology Solutions, Dartmouth, NS B2Y 4M9, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Katherine","family":"Borda","sequence":"additional","affiliation":[{"name":"GSTS\u2014Global Spatial Technology Solutions, Dartmouth, NS B2Y 4M9, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dhivya","family":"Jayaraman","sequence":"additional","affiliation":[{"name":"GSTS\u2014Global Spatial Technology Solutions, Dartmouth, NS B2Y 4M9, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Renata","family":"Dividino","sequence":"additional","affiliation":[{"name":"GSTS\u2014Global Spatial Technology Solutions, Dartmouth, NS B2Y 4M9, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4026-1569","authenticated-orcid":false,"given":"Jakub","family":"Nalepa","sequence":"additional","affiliation":[{"name":"KP Labs, Konarskiego 18C, 44-100 Gliwice, Poland"},{"name":"Department of Algorithmics and Software, Faculty of Automatic Control, Electronics and Computer Science, Silesian University of Technology, Akademicka 16, 44-100 Gliwice, Poland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,10,5]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Arechiga, A.P., Michaels, A.J., and Black, J.T. (2018, January 23\u201326). Onboard Image Processing for Small Satellites. Proceedings of the IEEE National Aerospace Electronics Conference, NAECON, Dayton, OH, USA.","DOI":"10.1109\/NAECON.2018.8556744"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Bahl, G., Daniel, L., Moretti, M., and Lafarge, F. (2019, January 27\u201328). Low-Power Neural Networks for Semantic Segmentation of Satellite Images. Proceedings of the 2019 International Conference on Computer Vision Workshop, ICCVW 2019, Seoul, Korea.","DOI":"10.1109\/ICCVW.2019.00302"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1109\/LCA.2019.2907539","article-title":"Orbital Edge Computing: Machine Inference in Space","volume":"18","author":"Denby","year":"2019","journal-title":"IEEE Comput. Archit. Lett."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Wang, Y., Yang, J., Guo, X., and Qu, Z. (2019). Satellite Edge Computing for the Internet of Things in Aerospace. Sensor, 19.","DOI":"10.3390\/s19204375"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"991","DOI":"10.1016\/j.cja.2019.02.005","article-title":"Satellite Group Autonomous Operation Mechanism and Planning Algorithm for Marine Target Surveillance","volume":"32","author":"Zhang","year":"2019","journal-title":"Chin. J. Aeronaut."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Bohm Agostini, N., Dong, S., Karimi, E., Torrents Lapuerta, M., Cano, J., Abell\u00e1n, J.L., and Kaeli, D. (2020, January 9\u201311). Design Space Exploration of Accelerators and End-to-End DNN Evaluation with TFLITE-SOC. Proceedings of the 2020 IEEE 32nd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), Porto, Portugal.","DOI":"10.1109\/SBAC-PAD49847.2020.00013"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1109\/MM.2021.3066343","article-title":"The Vision Behind MLPerf: Understanding AI Inference Performance","volume":"41","author":"Reddi","year":"2021","journal-title":"IEEE Micro"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Boutros, A., Nurvitadhi, E., Ma, R., Gribok, S., Zhao, Z., Hoe, J.C., Betz, V., and Langhammer, M. (2020, January 9\u201311). Beyond Peak Performance: Comparing the Real Performance of AI-Optimized FPGAs and GPUs. Proceedings of the International Conference on Field-Programmable Technology, (IC)FPT 2020, Maui, HI, USA.","DOI":"10.1109\/ICFPT51103.2020.00011"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3444692","article-title":"A Survey on Edge Performance Benchmarking","volume":"54","author":"Varghese","year":"2021","journal-title":"ACM Comput. Surv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"7218758","DOI":"10.1155\/2019\/7218758","article-title":"An FPGA-Based Hardware Accelerator for CNNs Using On-Chip Memories Only: Design and Benchmarking with Intel Movidius Neural Compute Stick","volume":"2019","author":"Dinelli","year":"2019","journal-title":"Int. J. Reconfigurable Comput."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Hao, T., Huang, Y., Wen, X., Gao, W., Zhang, F., Zheng, C., Wang, L., Ye, H., Hwang, K., and Ren, Z. (2019). Edge AIBench: Towards Comprehensive End-to-End Edge Computing Benchmarking. International Symposium on Benchmarking, Measuring and Optimization, Springer. Lecture Notes in Computer Science.","DOI":"10.1007\/978-3-030-32813-9_3"},{"key":"ref_12","unstructured":"Caldas, S., Wu, P., Li, T., Kone\u010dn\u00fd, J., McMahan, H.B., Smith, V., and Talwalkar, A. (2018). LEAF: A Benchmark for Federated Settings. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"64270","DOI":"10.1109\/ACCESS.2018.2877890","article-title":"Benchmark Analysis of Representative Deep Neural Network Architectures","volume":"6","author":"Bianco","year":"2018","journal-title":"IEEE Access"},{"key":"ref_14","unstructured":"Mattson, P., Cheng, C., Coleman, C., Diamos, G., Micikevicius, P., Patterson, D., Tang, H., Wei, G.Y., Bailis, P., and Bittorf, V. (2019). MLPerf Training Benchmark. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Reuther, A., Michaleas, P., Jones, M., Gadepally, V., Samsi, S., and Kepner, J. (2019, January 24\u201326). Survey and Benchmarking of Machine Learning Accelerators. Proceedings of the 2019 IEEE High Performance Extreme Computing Conference, HPEC 2019, Waltham, MA, USA.","DOI":"10.1109\/HPEC.2019.8916327"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Wang, Y., Wang, Q., Shi, S., He, X., Tang, Z., Zhao, K., and Chu, X. (2020, January 11\u201314). Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training. Proceedings of the 20th IEEE\/ACM International Symposium on Cluster, Cloud and Internet Computing, CCGRID 2020, Melbourne, VIC, Australia.","DOI":"10.1109\/CCGrid49817.2020.00-15"},{"key":"ref_17","first-page":"3285","article-title":"Deep Learning for Autonomous Lunar Landing","volume":"167","author":"Furfaro","year":"2018","journal-title":"Adv. Astronaut. Sci."},{"key":"ref_18","unstructured":"Engineering, I., Way, E.J.E.R., Engineering, I., and Way, E.J.E.R. (2012, September 30). Lunar Landing, Available online: https:\/\/arc.aiaa.org\/doi\/10.2514\/6.2020-1910."},{"key":"ref_19","unstructured":"Zhang, J., Xia, Y., and Shen, G. (2018). A Novel Deep Neural Network Architecture for Mars Visual Navigation. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Nalepa, J., Myller, M., Cwiek, M., Zak, L., Lakota, T., Tulczyjew, L., and Kawulok, M. (2021). Towards On-Board Hyperspectral Satellite Image Segmentation: Understanding Robustness of Deep Learning through Simulating Acquisition Conditions. Remote Sens., 13.","DOI":"10.3390\/rs13081532"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1228","DOI":"10.1109\/LGRS.2019.2942832","article-title":"Transfer Learning for Segmenting Dimensionally Reduced Hyperspectral Images","volume":"17","author":"Nalepa","year":"2020","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1264","DOI":"10.1109\/LGRS.2019.2895697","article-title":"Validating Hyperspectral Image Segmentation","volume":"16","author":"Nalepa","year":"2019","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Wagstaff, K.L., Lu, Y., Stanboli, A., Grimes, K., Gowda, T., and Padams, J. (2018, January 2\u20137). Deep Mars: CNN Classification of Mars Imagery for the PDS Imaging Atlas. Proceedings of the 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11404"},{"key":"ref_24","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3\u20136). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012, Lake Tahoe, NV, USA."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1016\/j.icarus.2018.06.022","article-title":"Lunar Crater Identification via Deep Learning","volume":"317","author":"Silburt","year":"2019","journal-title":"Icarus"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"16591","DOI":"10.1109\/ACCESS.2021.3053408","article-title":"INet: Convolutional Networks for Biomedical Image Segmentation","volume":"9","author":"Weng","year":"2021","journal-title":"IEEE Access"},{"key":"ref_27","unstructured":"Xilinx (2021). Vitis AI User Guide, Xilinx. Technical Report UG1414 (v1.4)."},{"key":"ref_28","unstructured":"Xilinx (2021). PetaLinux Tools Documentation, Reference Guide, Xilinx. Technical Report UG1144 (v2021.1)."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"370","DOI":"10.1016\/j.neucom.2021.07.045","article-title":"Pruning and Quantization for Deep Neural Network Acceleration: A Survey","volume":"461","author":"Liang","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_30","unstructured":"Xilinx (2021). Vitis AI Optimizer User Guide, Xilinx. Technical Report UG1333 (v1.4)."},{"key":"ref_31","unstructured":"Xilinx (2021). Vitis AI Library User Guide, Xilinx. Technical Report UG1354 (v1.4)."},{"key":"ref_32","unstructured":"Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"276","DOI":"10.11613\/BM.2012.031","article-title":"Interrater Reliability: The Kappa Statistic","volume":"22","author":"McHugh","year":"2012","journal-title":"Biochem. Medica"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Wu, Y., Wan, G., Liu, L., Wei, Z., and Wang, S. (2021, January 12\u201314). Intelligent Crater Detection on Planetary Surface Using Convolutional Neural Network. Proceedings of the 2021 IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Chongqing, China.","DOI":"10.1109\/IAEAC50856.2021.9391002"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Redmon, J., Divvala, S., Girshick, R.B., and Farhadi, A. (July, January 26). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.91"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Girshick, R. (2015, January 11\u201318). Fast R-CNN. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.","DOI":"10.1109\/ICCV.2015.169"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"318","DOI":"10.1109\/TPAMI.2018.2858826","article-title":"Focal Loss for Dense Object Detection","volume":"42","author":"Lin","year":"2020","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1789","DOI":"10.1007\/s11263-021-01453-z","article-title":"Knowledge Distillation: A Survey","volume":"129","author":"Gou","year":"2021","journal-title":"Int. J. Comput. Vis."},{"key":"ref_39","unstructured":"Nalepa, J., Kuligowski, P., Gumiela, M., Drobik, M., and Nowak, M. (2020, January 12\u201314). Leopard: A New Chapter in On-Board Deep Learning-Powered Analysis of Hyperspectral Imagery. Proceedings of the 2020 IAC, IAF Earth Observation Symposium, Online."},{"key":"ref_40","unstructured":"Xilinx (2021). DPUCZDX8G for Zynq UltraScale+ MPSoCs, Product Guide, Xilinx. Technical Report PG338 (v3.3)."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/19\/3981\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:09:30Z","timestamp":1760166570000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/19\/3981"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,5]]},"references-count":40,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2021,10]]}},"alternative-id":["rs13193981"],"URL":"https:\/\/doi.org\/10.3390\/rs13193981","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,5]]}}}