{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T04:00:06Z","timestamp":1776139206382,"version":"3.50.1"},"reference-count":38,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2024,7,30]],"date-time":"2024-07-30T00:00:00Z","timestamp":1722297600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"State of Hesse as a part of \u201cLOEWE funding line 3\u201d","award":["1381\/22-86"],"award-info":[{"award-number":["1381\/22-86"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>For the accurate and automatic mapping of forest stands based on very-high-resolution satellite imagery and digital orthophotos, precise object detection at the individual tree level is necessary. Currently, supervised deep learning models are primarily applied for this task. To train a reliable model, it is crucial to have an accurate tree crown annotation dataset. The current method of generating these training datasets still relies on manual annotation and labeling. Because of the intricate contours of tree crowns, vegetation density in natural forests and the insufficient ground sampling distance of the imagery, manually generated annotations are error-prone. It is unlikely that the manually delineated tree crowns represent the true conditions on the ground. If these error-prone annotations are used as training data for deep learning models, this may lead to inaccurate mapping results for the models. This study critically validates manual tree crown annotations on two study sites: a forest-like plantation on a cemetery and a natural city forest. The validation is based on tree reference data in the form of an official tree register and tree segments extracted from UAV laser scanning (ULS) data for the quality assessment of a training dataset. The validation results reveal that the manual annotations detect only 37% of the tree crowns in the forest-like plantation area and 10% of the tree crowns in the natural forest correctly. Furthermore, it is frequent for multiple trees to be interpreted in the annotation as a single tree at both study sites.<\/jats:p>","DOI":"10.3390\/rs16152786","type":"journal-article","created":{"date-parts":[[2024,7,30]],"date-time":"2024-07-30T15:25:23Z","timestamp":1722353123000},"page":"2786","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Is Your Training Data Really Ground Truth? A Quality Assessment of Manual Annotation for Individual Tree Crown Delineation"],"prefix":"10.3390","volume":"16","author":[{"given":"Janik","family":"Steier","sequence":"first","affiliation":[{"name":"Department of Remote Sensing and Image Analysis, Technical University of Darmstadt, Franziska-Braun-Str. 7, 64287 Darmstadt, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8291-4791","authenticated-orcid":false,"given":"Mona","family":"Goebel","sequence":"additional","affiliation":[{"name":"Department of Remote Sensing and Image Analysis, Technical University of Darmstadt, Franziska-Braun-Str. 7, 64287 Darmstadt, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5969-8533","authenticated-orcid":false,"given":"Dorota","family":"Iwaszczuk","sequence":"additional","affiliation":[{"name":"Department of Remote Sensing and Image Analysis, Technical University of Darmstadt, Franziska-Braun-Str. 7, 64287 Darmstadt, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2024,7,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"3388","DOI":"10.1109\/TPAMI.2020.2981890","article-title":"Imbalance Problems in Object Detection: A Review","volume":"43","author":"Oksuz","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"791","DOI":"10.1007\/s00778-022-00775-9","article-title":"Data collection and quality challenges in deep learning: A data-centric AI perspective","volume":"32","author":"Whang","year":"2023","journal-title":"VLDB J."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1007\/s40725-023-00184-3","article-title":"A Systematic Review of Individual Tree Crown Detection and Delineation with Convolutional Neural Networks (CNN)","volume":"9","author":"Zhao","year":"2023","journal-title":"Curr. For. Rep."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Fujimoto, A., Haga, C., Matsui, T., Machimura, T., Hayashi, K., Sugita, S., and Takagi, H. (2019). An End to End Process Development for UAV-SfM Based Forest Monitoring: Individual Tree Detection, Species Classification and Carbon Dynamics Simulation. Forests, 10.","DOI":"10.3390\/f10080680"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Saarinen, N., Vastaranta, M., N\u00e4si, R., Rosnell, T., Hakala, T., Honkavaara, E., Wulder, M., Luoma, V., Tommaselli, A., and Imai, N. (2018). Assessing Biodiversity in Boreal Forests with UAV-Based Photogrammetric Point Clouds and Hyperspectral Imaging. Remote Sens., 10.","DOI":"10.3390\/rs10020338"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"202","DOI":"10.1016\/j.rse.2016.10.014","article-title":"Mapping individual tree health using full-waveform airborne laser scans and imaging spectroscopy: A case study for a floodplain eucalypt forest","volume":"187","author":"Shendryk","year":"2016","journal-title":"Remote Sens. Environ."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"22197","DOI":"10.1007\/s00521-022-07640-4","article-title":"Individual tree crown delineation in high-resolution remote sensing images based on U-Net","volume":"34","author":"Freudenberg","year":"2022","journal-title":"Neural Comput. Appl."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1016\/j.ecolind.2017.10.066","article-title":"Predicting stem diameters and aboveground biomass of individual trees using remote sensing data","volume":"85","author":"Dalponte","year":"2018","journal-title":"Ecol. Indic."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1139\/x04-142","article-title":"Tree growth prediction using size and exposed crown area","volume":"35","author":"Wyckoff","year":"2005","journal-title":"Can. J. For. Res."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Weinstein, B.G., Marconi, S., Bohlman, S., Zare, A., and White, E. (2019). Individual Tree-Crown Detection in RGB Imagery Using Semi-Supervised Deep Learning Neural Networks. Remote Sens., 11.","DOI":"10.1101\/532952"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"G. Braga, J.R., Peripato, V., Dalagnol, R., P. Ferreira, M., Tarabalka, Y., O. C. Arag\u00e3o, L.E., F. de Campos Velho, H., Shiguemori, E.H., and Wagner, F.H. (2020). Tree Crown Delineation Algorithm Based on a Convolutional Neural Network. Remote Sens., 12.","DOI":"10.3390\/rs12081288"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1016\/j.isprsjprs.2020.12.010","article-title":"Review on Convolutional Neural Networks (CNN) in vegetation remote sensing","volume":"173","author":"Kattenborn","year":"2021","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_13","unstructured":"Zimmermann, E., Szeto, J., and Ratle, F. (2024, February 14). An Empirical Study of Uncertainty in Polygon Annotation and the Impact of Quality Assurance. Available online: http:\/\/arxiv.org\/pdf\/2311.02707.pdf."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"641","DOI":"10.1002\/rse2.332","article-title":"Accurate delineation of individual tree crowns in tropical forests from aerial RGB imagery using Mask R-CNN","volume":"9","author":"Ball","year":"2023","journal-title":"Remote Sens. Ecol. Conserv."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"220","DOI":"10.1016\/j.isprsjprs.2022.05.002","article-title":"Deep learning-based individual tree crown delineation in mangrove forests using very-high-resolution satellite imagery","volume":"189","author":"Lassalle","year":"2022","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Elmes, A., Alemohammad, H., Avery, R., Caylor, K., Eastman, J., Fishgold, L., Friedl, M., Jain, M., Kohli, D., and Laso Bayas, J. (2020). Accounting for Training Data Error in Machine Learning Applied to Earth Observations. Remote Sens., 12.","DOI":"10.3390\/rs12061034"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"11229","DOI":"10.1109\/JSTARS.2021.3122345","article-title":"RandCrowns: A Quantitative Metric for Imprecisely Labeled Tree Crown Delineation","volume":"14","author":"Stewart","year":"2021","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"257","DOI":"10.5194\/isprs-archives-XLVIII-2-2024-257-2024","article-title":"Integrating Crowd-sourced Annotations of Tree Crowns using Markov Random Field and Multispectral Information","volume":"48","author":"Mei","year":"2024","journal-title":"Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci."},{"key":"ref_19","first-page":"102662","article-title":"Counting trees in a subtropical mega city using the instance segmentation method","volume":"106","author":"Sun","year":"2022","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_20","unstructured":"Perz, S.G. (2019). High-Resolution Remote Sensing Data as a Boundary Object to Facilitate Interdisciplinary Collaboration. Collaboration Across Boundaries for Social-Ecological Systems Science, Springer International Publishing."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"362","DOI":"10.1016\/j.isprsjprs.2018.09.013","article-title":"Individual tree crown delineation in a highly diverse tropical forest using very high resolution satellite images","volume":"145","author":"Wagner","year":"2018","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Foody, G., Pal, M., Rocchini, D., Garzon-Lopez, C., and Bastin, L. (2016). The Sensitivity of Mapping Methods to Reference Data Quality: Training Supervised Image Classifications with Imperfect Reference Data. ISPRS Int. J. Geo-Inf., 5.","DOI":"10.3390\/ijgi5110199"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Copass, C., Antonova, N., and Kennedy, R. (2019). Comparison of Office and Field Techniques for Validating Landscape Change Classification in Pacific Northwest National Parks. Remote Sens., 11.","DOI":"10.3390\/rs11010003"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"119","DOI":"10.2307\/3236006","article-title":"How reliable are our vegetation analyses?","volume":"3","year":"1992","journal-title":"J Veg. Sci."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1016\/j.compenvurbsys.2011.11.001","article-title":"An ontology of slums for image-based classification","volume":"36","author":"Kohli","year":"2012","journal-title":"Comput. Environ. Urban Syst."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1016\/j.compenvurbsys.2016.07.010","article-title":"Uncertainty analysis for image interpretations of urban slums","volume":"60","author":"Kohli","year":"2016","journal-title":"Comput. Environ. Urban Syst."},{"key":"ref_27","unstructured":"Meining, S. (2023, December 13). Waldtzustandsbericht 2020 f\u00fcr den Stadtwald Darmstadt. Available online: https:\/\/www.darmstadtnews.de\/wp-content\/uploads\/2021\/01\/Waldzustandsbericht_Darmstadt_2020.pdf."},{"key":"ref_28","unstructured":"The MathWorks Inc. (2024, July 19). 2022, Lidar Toolbox Version: 9.4 (R2022b). Available online: https:\/\/www.mathworks.com."},{"key":"ref_29","unstructured":"The MathWorks Inc. (2024, July 19). Extract Forest Metrics and Individual Tree Attributes from Aerial Lidar Data. Available online: https:\/\/www.mathworks.com\/help\/lidar\/ug\/extraction-of-forest-metrics-and-individual-tree-attributes.html."},{"key":"ref_30","unstructured":"(2024, July 19). CloudCompare (Version 2.13.2). Available online: http:\/\/www.cloudcompare.org\/."},{"key":"ref_31","unstructured":"QGIS.org (2024, July 19). 2024, QGIS Geographic Information System. QGIS Association, Version 3.28.2. Available online: http:\/\/www.qgis.org."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Thambawita, V., Str\u00fcmke, I., Hicks, S.A., Halvorsen, P., Parasa, S., and Riegler, M.A. (2021). Impact of Image Resolution on Deep Learning Performance in Endoscopy Image Classification: An Experimental Study Using a Large Dataset of Endoscopic Images. Diagnostics, 11.","DOI":"10.3390\/diagnostics11122183"},{"key":"ref_33","unstructured":"SuperAnnotate AI, Inc. (2024, July 19). Available online: https:\/\/www.superannotate.com\/."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"159","DOI":"10.5194\/isprs-annals-X-1-W1-2023-159-2023","article-title":"From Multiple Polygons to Single Geometry: Optimization of Polygon Integration for Crowdsourced Data","volume":"10","author":"Collmar","year":"2023","journal-title":"ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Hossain, M. (2012, January 21\u201322). Users\u2019 motivation to participate in online crowdsourcing platforms. Proceedings of the International Conference on Innovation Management and Technology Research (ICIMTR), Malacca, Malaysia.","DOI":"10.1109\/ICIMTR.2012.6236409"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Zhou, Z.-H. (2021). Machine Learning, Springer.","DOI":"10.1007\/978-981-15-1967-3"},{"key":"ref_37","unstructured":"FRANKFURT.DE-DAS OFFIZIELLE STADTPORTAL (2024, May 30). Baumkataster und Baumliste|Stadt Frankfurt am Main. Available online: https:\/\/frankfurt.de\/themen\/umwelt-und-gruen\/umwelt-und-gruen-a-z\/im-gruenen\/baeume\/baumkataster."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1109\/MGRS.2020.2975132","article-title":"Crowdsourcing in Remote Sensing: A Review of Applications and Future Directions","volume":"8","author":"Saralioglu","year":"2020","journal-title":"IEEE Geosci. Remote Sens. Mag."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/16\/15\/2786\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:26:41Z","timestamp":1760110001000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/16\/15\/2786"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,30]]},"references-count":38,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2024,8]]}},"alternative-id":["rs16152786"],"URL":"https:\/\/doi.org\/10.3390\/rs16152786","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,30]]}}}