{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:08:37Z","timestamp":1760238517362,"version":"build-2065373602"},"reference-count":45,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2020,8,6]],"date-time":"2020-08-06T00:00:00Z","timestamp":1596672000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"USGS National Land Imaging (NLI) program","award":["140G0119C0001"],"award-info":[{"award-number":["140G0119C0001"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Providing rapid access to land surface change data and information is a goal of the U.S. Geological Survey. Through the Land Change Monitoring, Assessment, and Projection (LCMAP) initiative, we have initiated a monitoring capability that involves generating a suite of 10 annual land cover and land surface change datasets across the United States at a 30-m spatial resolution. During the LCMAP automated production, on a tile-by-tile basis, erroneous data can occasionally be generated due to hardware or software failure. While crucial to assure the quality of the data, rapid evaluation of results at the pixel level during production is a substantial challenge because of the massive data volumes. Traditionally, product quality relies on the validation after production, which is inefficient to reproduce the whole product when an error occurs. This paper presents a method for automatically evaluating LCMAP results during the production phase based on 14 indices to quickly find and flag erroneous tiles in the LCMAP products. The methods involved two types of comparisons: comparing LCMAP values across the temporal record to measure internal consistency and calculating the agreement with multiple intervals of the National Land Cover Database (NLCD) data to measure the consistency with existing products. We developed indices on a tile-by-tile basis in order to quickly find and flag potential erroneous tiles by comparing with surrounding tiles using local outlier factor analysis. The analysis integrates all indices into a local outlier score (LOS) to detect erroneous tiles that are distinct from neighboring tiles. Our analysis showed that the methods were sensitive to partially erroneous tiles in the simulated data with a LOS higher than 2. The rapid quality assessment methods also successfully identified erroneous tiles during the LCMAP production, in which land surface change results were not properly saved to the products. The LOS map and indices for rapid quality assessment also point to directions for further investigations. A map of all LOS values by tile for the published LCMAP shows all LOS values are below 2. We also investigated tiles with high LOS to ensure the distinction with neighboring tiles was reasonable. An index in this study shows the overall agreement between LCMAP and NLCD on a tile basis is above 71.5% and has an average at 89.1% across the 422 tiles in the conterminous United States. The workflow is suitable for other studies with a large volume of image products.<\/jats:p>","DOI":"10.3390\/rs12162524","type":"journal-article","created":{"date-parts":[[2020,8,6]],"date-time":"2020-08-06T09:41:21Z","timestamp":1596706881000},"page":"2524","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Methods of Rapid Quality Assessment for National-Scale Land Surface Change Monitoring"],"prefix":"10.3390","volume":"12","author":[{"given":"Qiang","family":"Zhou","sequence":"first","affiliation":[{"name":"ASRC Federal Data Solutions, Contractor to the U.S. Geological Survey, Earth Resources Observation and Science (EROS) Center, 47914 252nd Street, Sioux Falls, SD 57198, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christopher","family":"Barber","sequence":"additional","affiliation":[{"name":"U.S. Geological Survey (USGS), Earth Resources Observation and Science (EROS) Center, 47914 252nd Street, Sioux Falls, SD 57198, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"George","family":"Xian","sequence":"additional","affiliation":[{"name":"U.S. Geological Survey (USGS), Earth Resources Observation and Science (EROS) Center, 47914 252nd Street, Sioux Falls, SD 57198, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,8,6]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"13976","DOI":"10.1073\/pnas.0401545101","article-title":"Developing a science of land change: Challenges and methodological issues","volume":"101","author":"Rindfuss","year":"2004","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"4254","DOI":"10.1080\/01431161.2018.1452075","article-title":"Land cover 2.0","volume":"39","author":"Wulder","year":"2018","journal-title":"Int. J. Remote Sens."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"GB2008","DOI":"10.1029\/2003GB002142","article-title":"Improved estimates of net carbon emissions from land cover change in the tropics for the 1990s","volume":"18","author":"Achard","year":"2004","journal-title":"Glob. Biogeochem. Cycles"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"5125","DOI":"10.5194\/bg-9-5125-2012","article-title":"Carbon emissions from land use and land-cover change","volume":"9","author":"Houghton","year":"2012","journal-title":"Biogeosciences"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"12723","DOI":"10.1073\/pnas.1512542112","article-title":"Ecosystem carbon stocks and sequestration potential of federal lands across the conterminous United States","volume":"112","author":"Tan","year":"2015","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1002\/2014MS000361","article-title":"Quantitative attribution of major driving forces on soil organic carbon dynamics","volume":"7","author":"Wu","year":"2015","journal-title":"J. Adv. Model. Earth Syst."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1016\/j.rse.2015.12.043","article-title":"Evaluating Landsat 8 evapotranspiration for water use mapping in the Colorado River Basin","volume":"185","author":"Senay","year":"2016","journal-title":"Remote Sens. Environ."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1016\/j.rse.2003.10.018","article-title":"Remote sensing of vegetation and land-cover change in Arctic Tundra Ecosystems","volume":"89","author":"Stow","year":"2004","journal-title":"Remote Sens. Environ."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"850","DOI":"10.1126\/science.1244693","article-title":"High-resolution global maps of 21st-century forest cover change","volume":"342","author":"Hansen","year":"2013","journal-title":"Science"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1890\/130066","article-title":"Bringing an ecological view of change to Landsat-based remote sensing","volume":"12","author":"Kennedy","year":"2014","journal-title":"Front. Ecol. Environ."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"2914","DOI":"10.1016\/j.rse.2008.02.010","article-title":"North American forest disturbance mapped from a decadal Landsat record","volume":"112","author":"Masek","year":"2008","journal-title":"Remote Sens. Environ."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1016","DOI":"10.3390\/rs11091016","article-title":"Seasonal variation of the spatially non-stationary association between land surface temperature and urban landscape","volume":"11","author":"Liu","year":"2019","journal-title":"Remote Sens."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1016\/j.scs.2018.10.049","article-title":"An approach to examining performances of cool\/hot sources in mitigating\/enhancing land surface temperature under different temperature backgrounds based on landsat 8 image","volume":"44","author":"He","year":"2019","journal-title":"Sustain. Cities Soc."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1016\/j.rse.2015.12.040","article-title":"A time series analysis of urbanization induced land use and land cover change and its impact on land surface temperature with Landsat imagery","volume":"175","author":"Fu","year":"2016","journal-title":"Remote Sens. Environ."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"347","DOI":"10.5589\/m05-019","article-title":"Multitemporal land cover mapping for Canada: Methodology and products","volume":"31","author":"Latifovic","year":"2005","journal-title":"Can. J. Remote Sens."},{"key":"ref_16","unstructured":"Olthof, I., Latifovic, R., and Pouliot, D. (2020, July 10). Medium Resolution Land Cover Mapping of Canada from SPOT 4\/5 Data. Available online: http:\/\/ftp2.cits.rncan.gc.ca\/pub\/geott\/ess_pubs\/295\/295751\/of_0004_gc.pdf."},{"key":"ref_17","first-page":"840","article-title":"LUCC (Land Use and Cover Change) and the environmental-economic accounts system in Brazil","volume":"3","author":"Moreira","year":"2013","journal-title":"J. Earth Sci. Eng."},{"key":"ref_18","first-page":"2013","article-title":"Monitoramento da floresta Amaz\u00f4nica Brasileira por sat\u00e9lite","volume":"25","author":"Prodes","year":"2013","journal-title":"Inst. Nac. Pesqui. Espac. Proj. Prodes."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Lymburner, L., Tan, P., McIntyre, A., Lewis, A., and Thankappan, M. (2013, January 21\u201326). Dynamic Land Cover Dataset version 2: 2001-now\u2026a land cover odyssey. Proceedings of the 2013 IEEE International Geoscience and Remote Sensing Symposium-IGARSS, Melbourne, Australia.","DOI":"10.1109\/IGARSS.2013.6723532"},{"key":"ref_20","unstructured":"Deng, X., and Liu, J. (2012). Mapping land cover and land use changes in China. Remote Sensing of Land Use and Land Cover: Principles and Applications, CRC Press."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"2293","DOI":"10.1007\/s11430-014-4917-1","article-title":"A 30 meter land cover mapping of China with an efficient clustering algorithm CBEST","volume":"57","author":"Hu","year":"2014","journal-title":"Sci. China Earth Sci."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Feranec, J., Soukup, T., Hazeu, G., and Jaffrain, G. (2016). European Landscape Dynamics: CORINE Land Cover Data, CRC Press.","DOI":"10.1201\/9781315372860"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"95","DOI":"10.3390\/rs9010095","article-title":"Operational high resolution land cover map production at the country scale using satellite image time series","volume":"9","author":"Inglada","year":"2017","journal-title":"Remote Sens."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1080\/1747423X.2010.511681","article-title":"A new hybrid land cover dataset for Russia: A methodology for integrating statistics, remote sensing and in situ information","volume":"6","author":"Schepaschenko","year":"2011","journal-title":"J. Land Use Sci."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1038\/nature20584","article-title":"High-resolution mapping of global surface water and its long-term changes","volume":"540","author":"Pekel","year":"2016","journal-title":"Nature"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.3390\/rs9101065","article-title":"Nominal 30-m cropland extent map of continental Africa by integrating pixel-based and object-based algorithms using Sentinel-2 and Landsat-8 data on Google Earth Engine","volume":"9","author":"Xiong","year":"2017","journal-title":"Remote Sens."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1303","DOI":"10.1080\/014311600210191","article-title":"Development of a global land cover characteristics database and IGBP DISCover from 1 km AVHRR data","volume":"21","author":"Loveland","year":"2000","journal-title":"Int. J. Remote Sens."},{"key":"ref_28","first-page":"345","article-title":"Completion of the 2011 National Land Cover Database for the conterminous United States\u2013representing a decade of land cover change information","volume":"81","author":"Homer","year":"2015","journal-title":"Photogramm. Eng. Remote Sens."},{"key":"ref_29","first-page":"337","article-title":"Completion of the 2001 national land cover database for the counterminous United States","volume":"73","author":"Homer","year":"2007","journal-title":"Photogramm. Eng. Remote Sens."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1016\/j.isprsjprs.2018.09.006","article-title":"A new generation of the United States National Land Cover Database: Requirements, research priorities, design, and implementation strategies","volume":"146","author":"Yang","year":"2018","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"20666","DOI":"10.1073\/pnas.0704119104","article-title":"The emergence of land change science for global environmental change and sustainability","volume":"104","author":"Turner","year":"2007","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"111356","DOI":"10.1016\/j.rse.2019.111356","article-title":"Lessons learned implementing an operational continuous United States national land change monitoring capability: The Land Change Monitoring, Assessment, and Projection (LCMAP) approach","volume":"238","author":"Brown","year":"2019","journal-title":"Remote Sens. Environ."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1363","DOI":"10.3390\/rs10091363","article-title":"Analysis ready data: Enabling analysis of the Landsat archive","volume":"10","author":"Dwyer","year":"2018","journal-title":"Remote Sens."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.rse.2014.01.011","article-title":"Continuous change detection and classification of land cover using all available Landsat data","volume":"144","author":"Zhu","year":"2014","journal-title":"Remote Sens. Environ."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1002\/asmb.742","article-title":"Sequential design in quality control and validation of land cover databases","volume":"25","author":"Carfagna","year":"2009","journal-title":"Appl. Stoch. Models Bus. Ind."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Estima, J., and Painho, M. (2019). Photo based Volunteered Geographic Information initiatives: A comparative study of their suitability for helping quality control of Corine Land Cover. Geospatial Intelligence: Concepts, Methodologies, Tools, and Applications, IGI Global.","DOI":"10.4018\/978-1-5225-8054-6.ch048"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"3769","DOI":"10.1080\/01431160701881897","article-title":"Validation and comparison of 1 km global land cover products in China","volume":"29","author":"Wu","year":"2008","journal-title":"Int. J. Remote Sens."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"6975","DOI":"10.1080\/01431161.2012.695092","article-title":"A global land-cover validation data set, II: Augmenting a stratified sampling design to estimate accuracy by region and land-cover class","volume":"33","author":"Stehman","year":"2012","journal-title":"Int. J. Remote Sens."},{"key":"ref_39","unstructured":"Strahler, A.H., Boschetti, L., Foody, G.M., Friedl, M.A., Hansen, M.C., Herold, M., Mayaux, P., Morisette, J., Stehman, S.V., and Woodcock, C.E. (2006). Global Land Cover Validation: Recommendations for Evaluation and Accuracy Assessment of Global Land Cover Maps, European Communities."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"5768","DOI":"10.1080\/01431161.2012.674230","article-title":"A global land-cover validation data set, part I: Fundamental design principles","volume":"33","author":"Olofsson","year":"2012","journal-title":"Int. J. Remote Sens."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"111261","DOI":"10.1016\/j.rse.2019.111261","article-title":"Quality control and assessment of interpreter consistency of annual land cover reference data in an operational national monitoring program","volume":"238","author":"Pengra","year":"2019","journal-title":"Remote Sens. Environ."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"1133","DOI":"10.1016\/j.rse.2009.02.004","article-title":"Updating the 2001 National Land Cover Database land cover classification to 2006 by using Landsat imagery change detection methods","volume":"113","author":"Xian","year":"2009","journal-title":"Remote Sens. Environ"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"3","DOI":"10.4996\/fireecology.0301003","article-title":"A project for monitoring trends in burn severity","volume":"3","author":"Eidenshink","year":"2007","journal-title":"Fire Ecol."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2971","DOI":"10.3390\/rs11242971","article-title":"Overall Methodology Design for the United States National Land Cover Database 2016 Products","volume":"11","author":"Jin","year":"2019","journal-title":"Remote Sens."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Kriegel, H.P., Kr\u00f6ger, P., Schubert, E., and Zimek, A. (2009, January 2\u20136). LoOP: Local outlier probabilities. Proceedings of the 18th ACM Conference on Information and Knowledge Management, Hong Kong, China.","DOI":"10.1145\/1645953.1646195"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/12\/16\/2524\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:57:01Z","timestamp":1760176621000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/12\/16\/2524"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,6]]},"references-count":45,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2020,8]]}},"alternative-id":["rs12162524"],"URL":"https:\/\/doi.org\/10.3390\/rs12162524","relation":{},"ISSN":["2072-4292"],"issn-type":[{"type":"electronic","value":"2072-4292"}],"subject":[],"published":{"date-parts":[[2020,8,6]]}}}