{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T19:31:24Z","timestamp":1781206284910,"version":"3.54.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1009180","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2021,7,15]],"date-time":"2021-07-15T00:00:00Z","timestamp":1626307200000}}],"reference-count":43,"publisher":"Public Library of Science (PLoS)","issue":"7","license":[{"start":{"date-parts":[[2021,7,2]],"date-time":"2021-07-02T00:00:00Z","timestamp":1625184000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000936","name":"Gordon and Betty Moore Foundation","doi-asserted-by":"publisher","award":["GBMF4563"],"award-info":[{"award-number":["GBMF4563"]}],"id":[{"id":"10.13039\/100000936","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100008982","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1926542"],"award-info":[{"award-number":["1926542"]}],"id":[{"id":"10.13039\/501100008982","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100005825","name":"National Institute of Food and Agriculture","doi-asserted-by":"publisher","award":["1007080"],"award-info":[{"award-number":["1007080"]}],"id":[{"id":"10.13039\/100005825","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Broad scale remote sensing promises to build forest inventories at unprecedented scales. A crucial step in this process is to associate sensor data into individual crowns. While dozens of crown detection algorithms have been proposed, their performance is typically not compared based on standard data or evaluation metrics. There is a need for a benchmark dataset to minimize differences in reported results as well as support evaluation of algorithms across a broad range of forest types. Combining RGB, LiDAR and hyperspectral sensor data from the USA National Ecological Observatory Network\u2019s Airborne Observation Platform with multiple types of evaluation data, we created a benchmark dataset to assess crown detection and delineation methods for canopy trees covering dominant forest types in the United States. This benchmark dataset includes an R package to standardize evaluation metrics and simplify comparisons between methods. The benchmark dataset contains over 6,000 image-annotated crowns, 400 field-annotated crowns, and 3,000 canopy stem points from a wide range of forest types. In addition, we include over 10,000 training crowns for optional use. We discuss the different evaluation data sources and assess the accuracy of the image-annotated crowns by comparing annotations among multiple annotators as well as overlapping field-annotated crowns. We provide an example submission and score for an open-source algorithm that can serve as a baseline for future methods.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1009180","type":"journal-article","created":{"date-parts":[[2021,7,2]],"date-time":"2021-07-02T14:33:17Z","timestamp":1625236397000},"page":"e1009180","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":48,"title":["A benchmark dataset for canopy crown detection and delineation in co-registered airborne RGB, LiDAR and hyperspectral imagery from the National Ecological Observation Network"],"prefix":"10.1371","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2176-7935","authenticated-orcid":true,"given":"Ben G.","family":"Weinstein","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3805-4242","authenticated-orcid":true,"given":"Sarah J.","family":"Graves","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8096-754X","authenticated-orcid":true,"given":"Sergio","family":"Marconi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5559-9151","authenticated-orcid":true,"given":"Aditya","family":"Singh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4847-7604","authenticated-orcid":true,"given":"Alina","family":"Zare","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dylan","family":"Stewart","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4935-7321","authenticated-orcid":true,"given":"Stephanie A.","family":"Bohlman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ethan P.","family":"White","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2021,7,2]]},"reference":[{"key":"pcbi.1009180.ref001","unstructured":"How to map forest structure from aircraft, one tree at a time\u2014Dalponte\u20142018\u2014Ecology and Evolution\u2014Wiley Online Library. [cited 29 Mar 2021]. Available: https:\/\/onlinelibrary.wiley.com\/doi\/full\/10.1002\/ece3.4089"},{"key":"pcbi.1009180.ref002","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1016\/j.rse.2018.04.002","article-title":"Individual tree crown detection in sub-meter satellite imagery using Marked Point Processes and a geometrical-optical model.","volume":"211","author":"MF Gomes","year":"2018","journal-title":"Remote Sens Environ"},{"key":"pcbi.1009180.ref003","doi-asserted-by":"crossref","DOI":"10.1111\/geb.12371","article-title":"Assessing the general patterns of forest structure: Quantifying tree and forest allometric scaling relationships in the United States.","volume":"24","author":"L Duncanson","year":"2015","journal-title":"Glob Ecol Biogeogr"},{"key":"pcbi.1009180.ref004","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1016\/j.jag.2016.07.006","article-title":"A robust approach for tree segmentation in deciduous forests using small-footprint airborne LiDAR data.","volume":"52","author":"H Hamraz","year":"2016","journal-title":"Int J Appl Earth Obs Geoinformation"},{"key":"pcbi.1009180.ref005","doi-asserted-by":"crossref","first-page":"211","DOI":"10.3390\/rs11020211","article-title":"A Novel Approach for the Detection of Standing Tree Stems from Plot-Level Terrestrial Laser Scanning Data.","volume":"11","author":"W Zhang","year":"2019","journal-title":"Remote Sens."},{"key":"pcbi.1009180.ref006","doi-asserted-by":"crossref","first-page":"554","DOI":"10.1080\/07038992.2016.1196582","article-title":"Imputation of Individual Longleaf Pine (Pinus palustris Mill.) Tree Attributes from Field and LiDAR Data.","volume":"42","author":"CA Silva","year":"2016","journal-title":"Can J Remote Sens"},{"key":"pcbi.1009180.ref007","doi-asserted-by":"crossref","first-page":"312","DOI":"10.3390\/rs11030312","article-title":"Large Scale Palm Tree Detection in High Resolution Satellite Images Using U-Net.","volume":"11","author":"M Freudenberg","year":"2019","journal-title":"Remote Sens."},{"key":"pcbi.1009180.ref008","doi-asserted-by":"crossref","first-page":"309","DOI":"10.3390\/rs12020309","article-title":"Tree Species Traits Determine the Success of LiDAR-Based Crown Mapping in a Mixed Temperate Forest.","volume":"12","author":"JH Hastings","year":"2020","journal-title":"Remote Sens."},{"key":"pcbi.1009180.ref009","doi-asserted-by":"crossref","first-page":"34","DOI":"10.1016\/j.isprsjprs.2015.10.002","article-title":"A novel transferable individual tree crown delineation model based on Fishing Net Dragging and boundary classification.","volume":"110","author":"T Liu","year":"2015","journal-title":"ISPRS J Photogramm Remote Sens"},{"key":"pcbi.1009180.ref010","article-title":"Tree Crown Delineation Algorithm Based on a Convolutional Neural Network.","volume":"12","author":"JRG Braga","year":"2020","journal-title":"REMOTE Sens."},{"key":"pcbi.1009180.ref011","doi-asserted-by":"crossref","first-page":"1309","DOI":"10.3390\/rs11111309","article-title":"Individual Tree-Crown Detection in RGB Imagery Using Semi-Supervised Deep Learning Neural Networks.","volume":"11","author":"BG Weinstein","year":"2019","journal-title":"Remote Sens."},{"key":"pcbi.1009180.ref012","unstructured":"Are we ready for autonomous driving? The KITTI vision benchmark suite. [cited 29 Mar 2021]. Available: https:\/\/ieeexplore.ieee.org\/abstract\/document\/6248074\/"},{"key":"pcbi.1009180.ref013","doi-asserted-by":"crossref","first-page":"e1006494","DOI":"10.1371\/journal.pcbi.1006494","article-title":"Putting benchmarks in their rightful place: The heart of computational biology","volume":"14","author":"B Peters","year":"2018","journal-title":"PLOS Comput Biol"},{"key":"pcbi.1009180.ref014","article-title":"Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks.","author":"CG Northcutt","year":"2021","journal-title":"ArXiv210314749 Cs Stat."},{"key":"pcbi.1009180.ref015","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1186\/s13059-019-1738-8","article-title":"Essential guidelines for computational method benchmarking","volume":"20","author":"LM Weber","year":"2019","journal-title":"Genome Biol"},{"key":"pcbi.1009180.ref016","article-title":"Benchmarking Representation Learning for Natural World Image Collections.","author":"G Van Horn","year":"2021","journal-title":"ArXiv210316483 Cs."},{"key":"pcbi.1009180.ref017","doi-asserted-by":"crossref","first-page":"4725","DOI":"10.1080\/01431161.2010.494184","article-title":"A review of methods for automatic individual tree-crown detection and delineation from passive remote sensing","volume":"32","author":"Y Ke","year":"2011","journal-title":"Int J Remote Sens"},{"key":"pcbi.1009180.ref018","unstructured":"National Ecological Observatory Network (NEON). High-resolution orthorectified camera imagery mosaic (DP3.30010.001). 2021. Available: https:\/\/data.neonscience.org\/data-products\/DP3.30010.001"},{"key":"pcbi.1009180.ref019","unstructured":"National Ecological Observatory Network (NEON). Discrete return LiDAR point cloud (DP1.30003.001). National Ecological Observatory Network (NEON); 2021. doi: 10.48443\/6E8K-3343"},{"key":"pcbi.1009180.ref020","doi-asserted-by":"crossref","first-page":"112056","DOI":"10.1016\/j.rse.2020.112056","article-title":"A simulation method to infer tree allometry and forest structure from airborne laser scanning and forest inventories","volume":"251","author":"FJ Fischer","year":"2020","journal-title":"Remote Sens Environ."},{"key":"pcbi.1009180.ref021","doi-asserted-by":"crossref","DOI":"10.3390\/rs12081245","article-title":"Estimation of Forest Growing Stock Volume with UAV Laser Scanning Data: Can It Be Done without Field Data?","volume":"12","author":"S Puliti","year":"2020","journal-title":"Remote Sens."},{"key":"pcbi.1009180.ref022","doi-asserted-by":"crossref","first-page":"1218","DOI":"10.3390\/rs10081218","article-title":"Individual Tree Crown Segmentation and Classification of 13 Tree Species Using Airborne Hyperspectral Data.","volume":"10","author":"J Maschler","year":"2018","journal-title":"Remote Sens."},{"key":"pcbi.1009180.ref023","doi-asserted-by":"crossref","first-page":"3927","DOI":"10.1109\/JSTARS.2021.3069159","article-title":"Multisensor Data Fusion for Improved Segmentation of Individual Tree Crowns in Dense Tropical Forests.","volume":"14","author":"M Aubry-Kientz","year":"2021","journal-title":"IEEE J Sel Top Appl Earth Obs Remote Sens."},{"key":"pcbi.1009180.ref024","unstructured":"National Ecological Observatory Network (NEON). Spectrometer orthorectified surface directional reflectance\u2014flightline (DP1.30006.001). National Ecological Observatory Network (NEON); 2021. doi: 10.48443\/N3YS-2070"},{"key":"pcbi.1009180.ref025","unstructured":"National Ecological Observatory Network (NEON). Ecosystem structure (DP3.30015.001). National Ecological Observatory Network (NEON); 2021. doi: 10.48443\/05Q5-9352"},{"key":"pcbi.1009180.ref026","unstructured":"National Ecological Observatory Network (NEON). Woody plant vegetation structure (DP1.10098.001). National Ecological Observatory Network (NEON); 2021. Available: https:\/\/data.neonscience.org\/data-products\/DP1.10098.001"},{"key":"pcbi.1009180.ref027","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1016\/j.rse.2013.07.044","article-title":"An efficient, multi-layered crown delineation algorithm for mapping individual tree structure across multiple ecosystems","volume":"154","author":"LI Duncanson","year":"2014","journal-title":"Remote Sens Environ"},{"key":"pcbi.1009180.ref028","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/j.isprsjprs.2020.11.016","article-title":"Combining graph-cut clustering with object-based stem detection for tree segmentation in highly dense airborne lidar point clouds","volume":"172","author":"S Dersch","year":"2021","journal-title":"ISPRS J Photogramm Remote Sens"},{"key":"pcbi.1009180.ref029","doi-asserted-by":"crossref","first-page":"754","DOI":"10.1109\/TGRS.2019.2940146","article-title":"3D Segmentation of Trees Through a Flexible Multiclass Graph Cut Algorithm","volume":"58","author":"J Williams","year":"2020","journal-title":"IEEE Trans Geosci Remote Sens"},{"key":"pcbi.1009180.ref030","doi-asserted-by":"crossref","first-page":"1086","DOI":"10.3390\/rs11091086","article-title":"A Comparative Assessment of the Performance of Individual Tree Crowns Delineation Algorithms from ALS Data in Tropical Forests.","volume":"11","author":"M Aubry-Kientz","year":"2019","journal-title":"Remote Sens."},{"key":"pcbi.1009180.ref031","doi-asserted-by":"crossref","first-page":"111770","DOI":"10.1016\/j.rse.2020.111770","article-title":"Detection of sub-canopy forest structure using airborne LiDAR","volume":"244","author":"LR Jarron","year":"2020","journal-title":"Remote Sens Environ."},{"key":"pcbi.1009180.ref032","first-page":"336","article-title":"Applying LiDAR Individual Tree Detection to Management of Structurally Diverse Forest Landscapes.","volume":"116","author":"SMA Jeronimo","year":"2018","journal-title":"J For."},{"key":"pcbi.1009180.ref033","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1016\/j.rse.2018.09.009","article-title":"A tree-based approach to biomass estimation from remote sensing data in a tropical agricultural landscape.","volume":"218","author":"SJ Graves","year":"2018","journal-title":"Remote Sens Environ"},{"key":"pcbi.1009180.ref034","first-page":"2980","article-title":"Focal Loss for Dense Object Detection.","author":"T-Y Lin","year":"2017"},{"key":"pcbi.1009180.ref035","doi-asserted-by":"crossref","first-page":"1743","DOI":"10.1111\/2041-210X.13472","article-title":"DeepForest: A Python package for RGB deep learning tree crown delineation.","volume":"11","author":"BG Weinstein","year":"2020","journal-title":"Methods Ecol Evol."},{"key":"pcbi.1009180.ref036","article-title":"Training Data for the NeonTreeEvaluation Benchmark.","author":"Ben Weinstein","year":"2020","journal-title":"Zenodo"},{"key":"pcbi.1009180.ref037","article-title":"SpaceNet: A Remote Sensing Dataset and Challenge Series.","author":"A Van Etten","year":"2019","journal-title":"ArXiv180701232 Cs."},{"key":"pcbi.1009180.ref038","article-title":"Addressing Annotation Imprecision for Tree Crown Delineation Using the RandCrowns Index.","author":"D Stewart","year":"2021","journal-title":"ArXiv210502186 Cs."},{"key":"pcbi.1009180.ref039","doi-asserted-by":"crossref","first-page":"1025","DOI":"10.3390\/rs11091025","article-title":"A Real-Time Tree Crown Detection Approach for Large-Scale Remote Sensing Images on FPGAs.","volume":"11","author":"W Li","year":"2019","journal-title":"Remote Sens."},{"key":"pcbi.1009180.ref040","article-title":"Mask R-CNN.","author":"K He","year":"2018","journal-title":"ArXiv170306870 Cs"},{"key":"pcbi.1009180.ref041","first-page":"9759","article-title":"Bridging the Gap Between Anchor-Based and Anchor-Free Detection via Adaptive Training Sample Selection.","author":"S Zhang","year":"2020"},{"key":"pcbi.1009180.ref042","article-title":"MLCVNet: Multi-Level Context VoteNet for 3D Object Detection.","author":"Q Xie","year":"2020","journal-title":"ArXiv200405679 Cs."},{"key":"pcbi.1009180.ref043","doi-asserted-by":"crossref","first-page":"101061","DOI":"10.1016\/j.ecoinf.2020.101061","article-title":"Cross-site learning in deep learning RGB tree crown detection.","volume":"56","author":"BG Weinstein","year":"2020","journal-title":"Ecol Inform."}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1009180","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2021,7,15]],"date-time":"2021-07-15T00:00:00Z","timestamp":1626307200000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1009180","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,15]],"date-time":"2021-07-15T13:49:01Z","timestamp":1626356941000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1009180"}},"subtitle":[],"editor":[{"given":"Jacopo","family":"Grilli","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2021,7,2]]},"references-count":43,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2021,7,2]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1009180","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2020.11.16.385088","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,7,2]]}}}