{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T10:08:40Z","timestamp":1775470120613,"version":"3.50.1"},"reference-count":60,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2019,10,3]],"date-time":"2019-10-03T00:00:00Z","timestamp":1570060800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"publisher","award":["Linkage grant number (LP150100972)"],"award-info":[{"award-number":["Linkage grant number (LP150100972)"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>The utility of land cover maps for natural resources management relies on knowing the uncertainty associated with each map. The continuous advances typical of remote sensing, including the increasing availability of higher spatial and temporal resolution satellite data and data analysis capabilities, have created both opportunities and challenges for improving the application of accuracy assessment. There are well established accuracy assessment methods, but their underlying assumptions have not changed much in the last couple decades. Consequently, revisiting how map error and accuracy have been performed and reported over the last two decades is timely, to highlight areas where there is scope for better utilization of emerging opportunities. We conducted a quantitative literature review on accuracy assessment practices for mapping via remote sensing classification methods, in both terrestrial and marine environments. We performed a structured search for land and benthic cover mapping, limiting our search to journals within the remote sensing field, and papers published between 1998\u20132017. After an initial screening process, we assembled a database of 282 papers, and extracted and standardized information on various components of their reported accuracy assessments. We discovered that only 56% of the papers explicitly included an error matrix, and a very limited number (14%) reported overall accuracy with confidence intervals. The use of kappa continues to be standard practice, being reported in 50.4% of the literature published on or after 2012. Reference datasets used for validation were collected using a probability sampling design in 54% of the papers. For approximately 11% of the studies, the sampling design used could not be determined. No association was found between classification complexity (i.e. number of classes) and measured accuracy, independent from the size of the study area. Overall, only 32% of papers included an accuracy assessment that could be considered reproducible; that is, they included a probability-based sampling scheme to collect the reference dataset, a complete error matrix, and provided sufficient characterization of the reference datasets and sampling unit. Our findings indicate that considerable work remains to identify and adopt more statistically rigorous accuracy assessment practices to achieve transparent and comparable land and benthic cover maps.<\/jats:p>","DOI":"10.3390\/rs11192305","type":"journal-article","created":{"date-parts":[[2019,10,4]],"date-time":"2019-10-04T04:12:52Z","timestamp":1570162372000},"page":"2305","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":99,"title":["Trends in Remote Sensing Accuracy Assessment Approaches in the Context of Natural Resources"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5151-4739","authenticated-orcid":false,"given":"Lucia","family":"Morales-Barquero","sequence":"first","affiliation":[{"name":"Remote Sensing Research Centre, School of Earth and Environmental Sciences, University of Queensland, Brisbane QLD 4072, Australia"},{"name":"Department of Environment and Science, 41 Boggo Rd, Dutton Park QLD 4102, Australia"},{"name":"Joint Remote Sensing Research Program, School of Environment and Earth Sciences, University of Queensland, Brisbane QLD 4067, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3960-3522","authenticated-orcid":false,"given":"Mitchell","family":"Lyons","sequence":"additional","affiliation":[{"name":"Remote Sensing Research Centre, School of Earth and Environmental Sciences, University of Queensland, Brisbane QLD 4072, Australia"},{"name":"Joint Remote Sensing Research Program, School of Environment and Earth Sciences, University of Queensland, Brisbane QLD 4067, Australia"},{"name":"Centre for Ecosystem Science, School of Biological, Earth and Environmental Sciences, UNSW Australia, Sydney, NSW 2052, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2605-6104","authenticated-orcid":false,"given":"Stuart","family":"Phinn","sequence":"additional","affiliation":[{"name":"Remote Sensing Research Centre, School of Earth and Environmental Sciences, University of Queensland, Brisbane QLD 4072, Australia"},{"name":"Joint Remote Sensing Research Program, School of Environment and Earth Sciences, University of Queensland, Brisbane QLD 4067, Australia"}]},{"given":"Chris","family":"Roelfsema","sequence":"additional","affiliation":[{"name":"Remote Sensing Research Centre, School of Earth and Environmental Sciences, University of Queensland, Brisbane QLD 4072, Australia"}]}],"member":"1968","published-online":{"date-parts":[[2019,10,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1016\/S0034-4257(01)00295-4","article-title":"Status of land cover classification accuracy assessment","volume":"80","author":"Foody","year":"2002","journal-title":"Remote Sens. Environ."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1461","DOI":"10.1080\/014311699212560","article-title":"Quality assessment of image classification algorithms for land-cover mapping: A review and a proposal for a cost-based approach","volume":"20","author":"Smits","year":"1999","journal-title":"Int. J. Remote Sens."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1016\/j.rse.2012.09.017","article-title":"An accuracy assessment of forest disturbance mapping in the western Great Lakes","volume":"128","author":"Zimmerman","year":"2013","journal-title":"Remote Sens. Environ."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1016\/j.rse.2012.10.031","article-title":"Making better use of accuracy data in land change studies: Estimating accuracy and area and quantifying uncertainty using stratified estimation","volume":"129","author":"Olofsson","year":"2013","journal-title":"Remote Sens. Environ."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.ecolecon.2015.01.003","article-title":"Valuing map validation: The need for rigorous land cover map accuracy assessment in economic valuations of ecosystem services","volume":"111","author":"Foody","year":"2015","journal-title":"Ecol. Econ."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1016\/j.foreco.2015.07.017","article-title":"How sampling and scale limit accuracy assessment of vegetation maps: A comment on Loehle et al. (2015)","volume":"358","author":"Gregory","year":"2015","journal-title":"For. Ecol. Manag."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"823","DOI":"10.1080\/01431160600746456","article-title":"A survey of image classification methods and techniques for improving classification performance","volume":"28","author":"Lu","year":"2007","journal-title":"Int. J. Remote Sens."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.rse.2016.02.028","article-title":"A meta-analysis of remote sensing research on supervised pixel-based land-cover image classification processes: General guidelines for practitioners and future research","volume":"177","author":"Khatami","year":"2016","journal-title":"Remote Sens. Environ."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1109\/TGRS.2004.837325","article-title":"Results and implications of a study of fifteen years of satellite image classification experiments","volume":"43","author":"Wilkinson","year":"2005","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"648","DOI":"10.1016\/j.rse.2017.09.035","article-title":"Effect of classifier selection, reference sample size, reference class distribution and scene heterogeneity in per-pixel classification accuracy using 26 Landsat sites","volume":"204","author":"Heydari","year":"2018","journal-title":"Remote Sens. Environ."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"342","DOI":"10.1016\/S0034-4257(02)00126-8","article-title":"Effects of landscape characteristics on land-cover class accuracy","volume":"84","author":"Smith","year":"2003","journal-title":"Remote Sens. Environ."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"6914","DOI":"10.1080\/01431161.2013.810822","article-title":"Impact of training and validation sample selection on classification accuracy and accuracy assessment when using reference polygons in object-based classification","volume":"34","author":"Zhen","year":"2013","journal-title":"Int. J. Remote Sens."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"8489","DOI":"10.3390\/rs70708489","article-title":"On the Importance of Training Data Sample Selection in Random Forest Image Classification: A Case Study in Peatland Ecosystem Mapping","volume":"7","author":"Millard","year":"2015","journal-title":"Remote Sens."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1016\/j.isprsjprs.2012.04.001","article-title":"Comparison of support vector machine, neural network, and CART algorithms for the land-cover classification using limited training data points","volume":"70","author":"Shao","year":"2012","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Hsiao, L.-H., Cheng, K.-S., Hsiao, L.-H., and Cheng, K.-S. (2016). Assessing Uncertainty in LULC Classification Accuracy by Using Bootstrap Resampling. Remote Sens., 8.","DOI":"10.20944\/preprints201608.0214.v1"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"237","DOI":"10.2747\/1548-1603.44.3.237","article-title":"Improving classification accuracy assessments with statistical bootstrap resampling techniques","volume":"44","author":"Weber","year":"2007","journal-title":"GIScience Remote Sens."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1016\/j.rse.2018.02.026","article-title":"A comparison of resampling methods for remote sensing classification and accuracy assessment","volume":"208","author":"Lyons","year":"2018","journal-title":"Remote Sens. Environ."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1016\/S0034-4257(98)00010-8","article-title":"Design and Analysis for Thematic Map Accuracy Assessment - an application of satellite imagery","volume":"64","author":"Stehman","year":"1998","journal-title":"Remote Sens. Environ."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"2423","DOI":"10.1080\/014311699212100","article-title":"Basic probability sampling designs for thematic map accuracy assessment","volume":"20","author":"Stehman","year":"1999","journal-title":"Int. J. Remote Sens."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/j.rse.2014.02.015","article-title":"Good practices for estimating area and assessing accuracy of land change","volume":"148","author":"Olofsson","year":"2014","journal-title":"Remote Sens. Environ."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1073","DOI":"10.1080\/014311600210434","article-title":"A note on procedures used for accuracy assessment in land cover maps derived from AVHRR data","volume":"21","author":"Friedl","year":"2000","journal-title":"Int J Remote Sens."},{"key":"ref_22","unstructured":"Strahler, A.H., Boschetti, L., Foody, G.M., Friedl, M.A., Hansen, M.C., Herold, M., Mayaux, P., Morisette, J.T., Stehman, S.V., and Woodcock, C.E. (2006). Global Land Cover Validation: Recommendations for Evaluation and Accuracy Assessment of Global Land Cover Maps, European Communities."},{"key":"ref_23","unstructured":"Stein, A., Van der Meer, F., and Gorte, B. (1999). Accuracy assessment of spatial information. Spatial statistics for Remote Sensing, Springer."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Congalton, R.G., and Green, K. (2009). Assessing the Accuracy of Remotely Sensed Data: Principles and Practices, CRC Press. [2nd ed.].","DOI":"10.1201\/9781420055139"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"5243","DOI":"10.1080\/01431160903131000","article-title":"Sampling designs for accuracy assessment of land cover","volume":"30","author":"Stehman","year":"2009","journal-title":"Int. J. Remote Sens."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"5573","DOI":"10.1080\/01431160500285175","article-title":"On possible measures for evaluating the degree of uncertainty of fuzzy thematic maps","volume":"26","author":"Ricotta","year":"2005","journal-title":"Int. J. Remote Sens."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Foody, G. (2017). Impacts of Sample Design for Validation Data on the Accuracy of Feedforward Neural Network Classification. Appl. Sci., 7.","DOI":"10.3390\/app7090888"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"2827","DOI":"10.1109\/TGRS.2011.2174156","article-title":"Latent Class Modeling for Site- and Non-Site-Specific Classification Accuracy Assessment Without Ground Data","volume":"50","author":"Foody","year":"2012","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1023\/A:1025111108050","article-title":"Toward estimation of map accuracy without a probability test sample","volume":"10","author":"Steele","year":"2003","journal-title":"Environ. Ecol. Stat."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"3137","DOI":"10.1080\/01431160701442120","article-title":"Harshness in image classification accuracy assessment","volume":"29","author":"Foody","year":"2008","journal-title":"Int. J. Remote Sens."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Castilla, G. (2016). We Must all Pay More Attention to Rigor in Accuracy Assessment: Additional Comment to \u201cThe Improvement of Land Cover Classification by Thermal Remote Sensing\u201d. Remote Sens. 2015, 7, 8368\u20138390. Remote Sens., 8.","DOI":"10.3390\/rs8040288"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"3275","DOI":"10.1080\/01431160902755346","article-title":"The impact of imperfect ground reference data on the accuracy of land cover change estimation","volume":"30","author":"Foody","year":"2009","journal-title":"Int. J. Remote Sens."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"663","DOI":"10.1080\/01431160500185284","article-title":"An accuracy assessment framework for large-area land cover classification products derived from medium-resolution satellite data","volume":"27","author":"Wulder","year":"2006","journal-title":"Int. J. Remote Sens."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1016\/0034-4257(91)90048-B","article-title":"A review of assessing the accuracy of classifications of remotely sensed data","volume":"37","author":"Congalton","year":"1991","journal-title":"Remote Sens. Environ."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"4407","DOI":"10.1080\/01431161.2011.552923","article-title":"Death to Kappa: Birth of quantity disagreement and allocation disagreement for accuracy assessment","volume":"32","author":"Pontius","year":"2011","journal-title":"Int. J. Remote Sens."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1016\/j.rse.2006.10.010","article-title":"Comparative assessment of the measures of thematic classification accuracy","volume":"107","author":"Liu","year":"2007","journal-title":"Remote Sens. Environ."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1016\/j.envsoft.2018.07.014","article-title":"How good is my map? A tool for semi-automated thematic mapping and spatially explicit confidence assessment","volume":"108","author":"Mitchell","year":"2018","journal-title":"Environ. Model. Softw."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1239","DOI":"10.1007\/s00477-012-0660-3","article-title":"Quantifying uncertainty in remotely sensed land cover maps","volume":"27","author":"Cripps","year":"2013","journal-title":"Stoch. Environ. Res. RISK Assess."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"3044","DOI":"10.1016\/j.rse.2011.06.007","article-title":"Pixels, blocks of pixels, and polygons: Choosing a spatial unit for thematic accuracy assessment","volume":"115","author":"Stehman","year":"2011","journal-title":"Remote Sens. Environ."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1016\/S0034-4257(99)00090-5","article-title":"Practical Implications of Design-Based Sampling Inference for Thematic Map Accuracy Assessment","volume":"72","author":"Stehman","year":"2000","journal-title":"Remote Sens. Environ."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"199","DOI":"10.3390\/ijgi5110199","article-title":"The Sensitivity of Mapping Methods to Reference Data Quality: Training Supervised Image Classifications with Imperfect Reference Data","volume":"5","author":"Foody","year":"2016","journal-title":"ISPRS Int. J. Geo Inf."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1007\/s10661-018-7152-y","article-title":"A unified framework for land cover monitoring based on a discrete global sampling grid (GSG)","volume":"191","author":"Fehrmann","year":"2019","journal-title":"Environ. Monit. Assess."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"3966","DOI":"10.1080\/01431161.2011.636081","article-title":"Google Earth as a virtual globe tool for Earth science applications at the global scale: Progress and perspectives","volume":"33","author":"Yu","year":"2012","journal-title":"Int. J. Remote Sens."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1002\/rse2.15","article-title":"Framing the concept of satellite remote sensing essential biodiversity variables: Challenges and future directions","volume":"2","author":"Pettorelli","year":"2016","journal-title":"Remote Sens. Ecol. Conserv."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"585","DOI":"10.1016\/j.tree.2019.04.013","article-title":"Breaking the Habit(at)","volume":"34","author":"Coops","year":"2019","journal-title":"Trends Ecol. Evol."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1111\/cobi.12397","article-title":"Ten ways remote sensing can contribute to conservation","volume":"29","author":"Rose","year":"2015","journal-title":"Conserv. Biol."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"895","DOI":"10.1080\/13658816.2010.498378","article-title":"Thematic accuracy assessment of geographic object-based image classification","volume":"25","author":"Radoux","year":"2011","journal-title":"Int. J. Geogr. Inf. Sci."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1016\/j.isprsjprs.2016.03.008","article-title":"Optical remotely sensed time series data for land cover classification: A review","volume":"116","author":"White","year":"2016","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1016\/j.isprsjprs.2018.04.002","article-title":"A review of accuracy assessment for object-based image analysis: From per-pixel to per-polygon approaches","volume":"141","author":"Ye","year":"2018","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"111199","DOI":"10.1016\/j.rse.2019.05.018","article-title":"Key issues in rigorous accuracy assessment of land cover products","volume":"231","author":"Stehman","year":"2019","journal-title":"Remote Sens. Environ."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"1249","DOI":"10.1007\/s10980-012-9791-7","article-title":"Are landscape ecologists addressing uncertainty in their remote sensing data?","volume":"27","author":"Lechner","year":"2012","journal-title":"Landsc. Ecol."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"4573","DOI":"10.1080\/01431161.2014.930206","article-title":"Meta-discoveries from a synthesis of satellite-based land-cover mapping research","volume":"35","author":"Yu","year":"2014","journal-title":"Int. J. Remote Sens."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Goodman, J.A. (2013). Coral Reef Remote Sensing. Coral Reef Remote Sensing, Springer.","DOI":"10.1007\/978-90-481-9292-2"},{"key":"ref_54","unstructured":"(2019, April 04). The World Bank Land Area (sq. km)|Data. Available online: https:\/\/data.worldbank.org\/indicator\/ag.lnd.totl.k2."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Lesiv, M., See, L., Laso Bayas, J., Sturn, T., Schepaschenko, D., Karner, M., Moorthy, I., McCallum, I., Fritz, S., and Lesiv, M. (2018). Characterizing the Spatial and Temporal Availability of Very High Resolution Satellite Imagery in Google Earth and Microsoft Bing Maps as a Source of Reference Data. Land, 7.","DOI":"10.5194\/essd-2018-13"},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1016\/j.rse.2017.01.025","article-title":"Mapping per-pixel predicted accuracy of classified remote sensing images","volume":"191","author":"Khatami","year":"2017","journal-title":"Remote Sens. Environ."},{"key":"ref_57","first-page":"189","article-title":"Comparing the consistency of expert land cover knowledge","volume":"7","author":"Comber","year":"2005","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_58","first-page":"419","article-title":"Accuracy assessment of satellite derived land cover data: A review","volume":"60","author":"Janssen","year":"1994","journal-title":"Photogramm. Eng. Remote Sens."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1016\/j.isprsjprs.2017.06.001","article-title":"A review of supervised object-based land-cover image classification","volume":"130","author":"Ma","year":"2017","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"128","DOI":"10.1016\/j.rse.2003.04.005","article-title":"Multi-site evaluation of IKONOS data for classification of tropical coral reef environments","volume":"88","author":"Kramer","year":"2003","journal-title":"Remote Sens. Environ."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/11\/19\/2305\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:27:14Z","timestamp":1760189234000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/11\/19\/2305"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,3]]},"references-count":60,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2019,10]]}},"alternative-id":["rs11192305"],"URL":"https:\/\/doi.org\/10.3390\/rs11192305","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,3]]}}}