{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,23]],"date-time":"2026-01-23T21:51:43Z","timestamp":1769205103170,"version":"3.49.0"},"reference-count":37,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2017,10,10]],"date-time":"2017-10-10T00:00:00Z","timestamp":1507593600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Volunteered geographic data are being used increasingly to support land cover mapping and validation, yet the reliability of the volunteered data still requires further research. This study proposes data-based guidelines to help design the data collection by assessing the reliability of volunteered data collected using the Geo-Wiki tool. We summarized the interpretation difficulties of the volunteers at a global scale, including those areas and land cover types that generate the most confusion. We also examined the factors affecting the reliability of majority opinion and individual classification. The results showed that the highest interpretation inconsistency of the volunteers occurred in the ecoregions of tropical and boreal forests (areas with relatively poor coverage of very high resolution images), the tundra (a unique region that the volunteers are unacquainted with), and savannas (transitional zones). The volunteers are good at identifying forests, snow\/ice and croplands, but not grasslands and wetlands. The most confusing pairs of land cover types are also captured in this study and they vary greatly with different biomes. The reliability can be improved by providing more high resolution ancillary data, more interpretation keys in tutorials, and tools that assist in coverage estimation for those areas and land cover types that are most prone to confusion. We found that the reliability of the majority opinion was positively correlated with the percentage of volunteers selecting this choice and negatively related to their self-evaluated uncertainty when very high resolution images were available. Factors influencing the reliability of individual classifications were also compared and the results indicated that the interpretation difficulty of the target sample played a more important role than the knowledge base of the volunteers. The professional background and local knowledge had an influence on the interpretation performance, especially in identifying vegetation land cover types other than croplands. These findings can help in building a better filtering system to improve the reliability of volunteered data used in land cover validation and other applications.<\/jats:p>","DOI":"10.3390\/rs9101034","type":"journal-article","created":{"date-parts":[[2017,10,10]],"date-time":"2017-10-10T10:34:29Z","timestamp":1507631669000},"page":"1034","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Assessing and Improving the Reliability of Volunteered Land Cover Reference Data"],"prefix":"10.3390","volume":"9","author":[{"given":"Yuanyuan","family":"Zhao","sequence":"first","affiliation":[{"name":"Ministry of Education Key Laboratory for Earth System Modeling, Department of Earth System Science, Tsinghua University, Beijing 100084, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Duole","family":"Feng","sequence":"additional","affiliation":[{"name":"Ministry of Education Key Laboratory for Earth System Modeling, Department of Earth System Science, Tsinghua University, Beijing 100084, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3115-2042","authenticated-orcid":false,"given":"Le","family":"Yu","sequence":"additional","affiliation":[{"name":"Ministry of Education Key Laboratory for Earth System Modeling, Department of Earth System Science, Tsinghua University, Beijing 100084, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2665-7065","authenticated-orcid":false,"given":"Linda","family":"See","sequence":"additional","affiliation":[{"name":"International Institute for Applied Systems Analysis, Laxenburg A-2361, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Steffen","family":"Fritz","sequence":"additional","affiliation":[{"name":"International Institute for Applied Systems Analysis, Laxenburg A-2361, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7206-4148","authenticated-orcid":false,"given":"Christoph","family":"Perger","sequence":"additional","affiliation":[{"name":"International Institute for Applied Systems Analysis, Laxenburg A-2361, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Peng","family":"Gong","sequence":"additional","affiliation":[{"name":"Ministry of Education Key Laboratory for Earth System Modeling, Department of Earth System Science, Tsinghua University, Beijing 100084, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2017,10,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1509","DOI":"10.1007\/s10980-015-0337-7","article-title":"The impact of land use\/land cover scale on modelling urban ecosystem services","volume":"31","author":"Grafius","year":"2016","journal-title":"Landsc. Ecol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"4283","DOI":"10.1016\/j.jclepro.2015.07.118","article-title":"Towards consensus on land use impacts on biodiversity in LCA: UNEP\/SETAC Life Cycle Initiative preliminary recommendations based on expert contributions","volume":"112","author":"Teixeira","year":"2016","journal-title":"J. Clean. Prod."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"380","DOI":"10.4081\/gh.2016.380","article-title":"Assessment of malaria transmission changes in Africa, due to the climate impact of land use change using Coupled Model Intercomparison Project Phase 5 earth system models","volume":"11","author":"Tompkins","year":"2016","journal-title":"Geospat. Health"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s10708-007-9111-y","article-title":"Citizens as sensors: The world of volunteered geography","volume":"69","author":"Goodchild","year":"2007","journal-title":"GeoJournal"},{"key":"ref_5","unstructured":"De Albuquerque, J.P., Almeida, J.P., Fonte, C.C., and Cardoso, A. (2016, January 22\u201325). How volunteered geographic information can be integrated into emergency management practice? First lessons learned from an urban fire simulation in the city of Coimbra. Proceedings of the 13th International Conference on Information Systems for Crisis Response and Management (ISCRAM 2016), Rio de Janeiro, Brazil."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1080\/17538941003759255","article-title":"Crowdsourcing geographic information for disaster response: A research frontier","volume":"3","author":"Goodchild","year":"2010","journal-title":"Int. J. Digit. Earth"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1016\/j.apgeog.2016.01.005","article-title":"Methods for assessing the credibility of volunteered geographic information in flood response: A case study in Brisbane, Australia","volume":"68","author":"Hung","year":"2016","journal-title":"Appl. Geogr."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Attard, M., Haklay, M., and Capineri, C. (2016). The potential of volunteered geographic information (VGI) in future transport systems. Urban Plan., 1.","DOI":"10.17645\/up.v1i4.612"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"5A","DOI":"10.2489\/jswc.68.1.5A","article-title":"The global Land-Potential Knowledge System (LandPKS): Supporting evidence-based, site-specific land use and management through cloud computing, mobile applications, and crowdsourcing","volume":"68","author":"Herrick","year":"2013","journal-title":"J. Soil Water Conserv."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.envsoft.2011.11.015","article-title":"Geo-Wiki: An online platform for improving global land cover","volume":"31","author":"Fritz","year":"2012","journal-title":"Environ. Model. Softw."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1269","DOI":"10.1080\/13658816.2015.1018266","article-title":"Usability of VGI for validation of land cover maps","volume":"29","author":"Fonte","year":"2015","journal-title":"Int. J. Geogr. Inf. Sci."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Laso Bayas, J.C., See, L., Fritz, S., Sturn, T., Perger, C., D\u00fcrauer, M., Karner, M., Moorthy, I., Schepaschenko, D., and Domian, D. (2016). Crowdsourcing In-Situ Data on Land Cover and Land Use Using Gamification and Mobile Technology. Remote Sens., 8.","DOI":"10.3390\/rs8110905"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1109\/MGRS.2015.2498840","article-title":"Supporting earth-observation calibration and validation: A new generation of tools for crowdsourcing and citizen science","volume":"4","author":"See","year":"2016","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_14","unstructured":"Arsanjani, J.J., Helbich, M., and Bakillah, M. (2013, January 21\u201324). Exploiting volunteered geographic information to ease land use mapping of an urban landscape. Proceedings of the International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, London, UK."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1177\/0265813515604767","article-title":"Automated identification and characterization of parcels with OpenStreetMap and points of interest","volume":"43","author":"Liu","year":"2016","journal-title":"Environ. Plan. B Plan. Des."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1016\/j.compenvurbsys.2017.02.002","article-title":"Employing crowdsourced geographic data and multi-temporal\/multi-sensor satellite imagery to monitor land cover change: A case study in an urbanizing region of the Philippines","volume":"64","author":"Johnson","year":"2017","journal-title":"Comput. Environ. Urban Syst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1016\/j.isprsjprs.2014.06.016","article-title":"Building a hybrid land cover map with crowdsourcing and geographically weighted regression","volume":"103","author":"See","year":"2015","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"208","DOI":"10.1016\/j.rse.2015.02.011","article-title":"Development of a global hybrid forest mask through the synergy of remote sensing, crowdsourcing and FAO statistics","volume":"162","author":"Schepaschenko","year":"2015","journal-title":"Remote Sens. Environ."},{"key":"ref_19","unstructured":"Foody, G.M., and Boyd, D.S. (2012, January 10\u201313). Exploring the potential role of volunteer citizen sensors in land cover map accuracy assessment. Proceedings of the 10th International Symposium on Spatial Accuracy Assessment in Natural Resources and Environmental Science (Accuracy 2012), Florianopolis, Brazil."},{"key":"ref_20","first-page":"37","article-title":"Using control data to determine the reliability of volunteered geographic information about land cover","volume":"23","author":"Comber","year":"2013","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"410","DOI":"10.1080\/17538947.2015.1039609","article-title":"Assessing quality of volunteer crowdsourcing contributions: lessons from the Cropland Capture game","volume":"9","author":"Salk","year":"2016","journal-title":"Int. J. Digit. Earth"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1109\/MIC.2013.20","article-title":"Quality control in crowdsourcing systems: Issues and directions","volume":"17","author":"Allahbakhsh","year":"2013","journal-title":"IEEE Internet Comput."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"See, L., Comber, A., Salk, C., Fritz, S., van der Velde, M., Perger, C., Schill, C., McCallum, I., Kraxner, F., and Obersteiner, M. (2013). Comparing the quality of crowdsourced data contributed by expert and non-experts. PLoS ONE, 8.","DOI":"10.1371\/journal.pone.0069958"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"774","DOI":"10.3390\/rs8090774","article-title":"Local knowledge and professional background have a minimal impact on volunteer citizen science performance in a land-cover classification task","volume":"8","author":"Salk","year":"2016","journal-title":"Remote Sens."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Comber, A., Mooney, P., Purves, R.S., Rocchini, D., and Walz, A. (2016). Crowdsourcing: It Matters Who the Crowd Are. The Impacts of between Group Variations in Recording Land Cover. PLoS ONE, 11.","DOI":"10.1371\/journal.pone.0158329"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1016\/j.techfore.2015.03.002","article-title":"Harnessing the power of volunteers, the internet and Google Earth to collect and validate global spatial information using Geo-Wiki","volume":"98","author":"See","year":"2015","journal-title":"Technol. Forecast. Soc. Chang."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1021\/es103338e","article-title":"Land availability for biofuel production","volume":"45","author":"Cai","year":"2010","journal-title":"Environ. Sci. Technol."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"170075","DOI":"10.1038\/sdata.2017.75","article-title":"A global dataset of crowdsourced land cover and land use reference data","volume":"4","author":"Fritz","year":"2017","journal-title":"Sci. Data"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"4795","DOI":"10.1080\/01431161.2014.930202","article-title":"Towards a common validation sample set for global land-cover mapping","volume":"35","author":"Zhao","year":"2014","journal-title":"Int. J. Remote Sens."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"135","DOI":"10.1127\/0941-2948\/2010\/0430","article-title":"Observed and projected climate shifts 1901\u20132100 depicted by world maps of the K\u00f6ppen-Geiger climate classification","volume":"19","author":"Rubel","year":"2010","journal-title":"Meteorol. Z."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Herrick, J.E., Beh, A., Barrios, E., Bouvier, I., Coetzee, M., Dent, D., Elias, E., Hengl, T., Karl, J.W., and Liniger, H. (2016). The Land-Potential Knowledge System (LandPKS): Mobile apps and collaboration for optimizing climate change investments. Ecosyst. Health Sustain., 2.","DOI":"10.1002\/ehs2.1209"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Kumar, N., Belhumeur, P., Biswas, A., Jacobs, D., Kress, W., Lopez, I., and Soares, J. (2012, January 7\u201313). Leafsnap: A computer vision system for automatic plant species identification. Proceedings of the Computer Vision\u2013ECCV 2012, Florence, Italy.","DOI":"10.1007\/978-3-642-33709-3_36"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1016\/j.ecolind.2012.09.014","article-title":"Remote sensing for conservation monitoring: Assessing protected areas, habitat extent, habitat condition, species diversity, and threats","volume":"33","author":"Nagendra","year":"2013","journal-title":"Ecol. Indicators"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"3431","DOI":"10.1007\/s10531-008-9479-0","article-title":"High resolution satellite imagery for tropical biodiversity studies: the devil is in the detail","volume":"17","author":"Nagendra","year":"2008","journal-title":"Biodivers. Conserv."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"839","DOI":"10.1111\/1365-2664.12261","article-title":"Satellite remote sensing for applied ecologists: Opportunities and challenges","volume":"51","author":"Pettorelli","year":"2014","journal-title":"J. Appl. Ecol."},{"key":"ref_36","unstructured":"Eurostat (2015). Overview of LUCAS, Eurostat."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"5309","DOI":"10.1080\/01431160903130994","article-title":"Incorporating reference classification uncertainty into the analysis of land cover accuracy","volume":"30","author":"Sarmento","year":"2009","journal-title":"Int. J. Remote Sens."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/9\/10\/1034\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:46:50Z","timestamp":1760208410000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/9\/10\/1034"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,10,10]]},"references-count":37,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2017,10]]}},"alternative-id":["rs9101034"],"URL":"https:\/\/doi.org\/10.3390\/rs9101034","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,10,10]]}}}