{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,25]],"date-time":"2026-04-25T09:33:09Z","timestamp":1777109589890,"version":"3.51.4"},"reference-count":33,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2018,2,25]],"date-time":"2018-02-25T00:00:00Z","timestamp":1519516800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020","doi-asserted-by":"publisher","award":["689812"],"award-info":[{"award-number":["689812"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000921","name":"European Cooperation in Science and Technology","doi-asserted-by":"publisher","award":["TD1202"],"award-info":[{"award-number":["TD1202"]}],"id":[{"id":"10.13039\/501100000921","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJGI"],"abstract":"<jats:p>Simple consensus methods are often used in crowdsourcing studies to label cases when data are provided by multiple contributors. A basic majority vote rule is often used. This approach weights the contributions from each contributor equally but the contributors may vary in the accuracy with which they can label cases. Here, the potential to increase the accuracy of crowdsourced data on land cover identified from satellite remote sensor images through the use of weighted voting strategies is explored. Critically, the information used to weight contributions based on the accuracy with which a contributor labels cases of a class and the relative abundance of class are inferred entirely from the contributed data only via a latent class analysis. The results show that consensus approaches do yield a classification that is more accurate than that achieved by any individual contributor. Here, the most accurate individual could classify the data with an accuracy of 73.91% while a basic consensus label derived from the data provided by all seven volunteers contributing data was 76.58%. More importantly, the results show that weighting contributions can lead to a statistically significant increase in the overall accuracy to 80.60% by ignoring the contributions from the volunteer adjudged to be the least accurate in labelling.<\/jats:p>","DOI":"10.3390\/ijgi7030080","type":"journal-article","created":{"date-parts":[[2018,2,27]],"date-time":"2018-02-27T03:36:12Z","timestamp":1519702572000},"page":"80","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":26,"title":["Increasing the Accuracy of Crowdsourced Information on Land Cover via a Voting Procedure Weighted by Information Inferred from the Contributed Data"],"prefix":"10.3390","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6464-3054","authenticated-orcid":false,"given":"Giles","family":"Foody","sequence":"first","affiliation":[{"name":"School of Geography, University of Nottingham, Nottingham NG7 2RD, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2665-7065","authenticated-orcid":false,"given":"Linda","family":"See","sequence":"additional","affiliation":[{"name":"International Institute for Applied Systems Analysis (IIASA), Schlossplatz 1, A-2361 Laxenburg, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Steffen","family":"Fritz","sequence":"additional","affiliation":[{"name":"International Institute for Applied Systems Analysis (IIASA), Schlossplatz 1, A-2361 Laxenburg, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Inian","family":"Moorthy","sequence":"additional","affiliation":[{"name":"International Institute for Applied Systems Analysis (IIASA), Schlossplatz 1, A-2361 Laxenburg, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7206-4148","authenticated-orcid":false,"given":"Christoph","family":"Perger","sequence":"additional","affiliation":[{"name":"International Institute for Applied Systems Analysis (IIASA), Schlossplatz 1, A-2361 Laxenburg, Austria"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7093-443X","authenticated-orcid":false,"given":"Christian","family":"Schill","sequence":"additional","affiliation":[{"name":"Faculty of Environment and Natural Resources, Albert-Ludwig University, 79085 Freiburg, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3040-552X","authenticated-orcid":false,"given":"Doreen","family":"Boyd","sequence":"additional","affiliation":[{"name":"School of Geography, University of Nottingham, Nottingham NG7 2RD, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2018,2,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1436","DOI":"10.1126\/science.1251554","article-title":"Next steps for citizen science","volume":"343","author":"Bonney","year":"2014","journal-title":"Science"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"See, L., Mooney, P., Foody, G., Bastin, L., Comber, A., Estima, J., Fritz, S., Kerle, N., Jiang, B., and Laakso, M. (2016). Crowdsourcing, citizen science or Volunteered Geographic Information? The current state of crowdsourced geographic information. ISPRS Int. J. Geo-Inf., 5, Available online: http:\/\/www.mdpi.com\/2220-9964\/5\/5\/55.","DOI":"10.3390\/ijgi5050055"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s10708-007-9111-y","article-title":"Citizens as sensors: The world of volunteered geography","volume":"69","author":"Goodchild","year":"2007","journal-title":"GeoJournal"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Pullar, D., and Hayes, S. (2017). Will the future maps for Australia be published by \u2018nobodies\u2019. J. Spat. Sci., 1\u20138. Available online: http:\/\/www.tandfonline.com\/doi\/abs\/10.1080\/14498596.2017.1361873.","DOI":"10.1080\/14498596.2017.1361873"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Capineri, C., Haklay, M., Huang, H., Antoniou, V., Kettunen, J., Ostermann, F., and Purves, R. (2016). European Handbook of Crowdsourced Geographic Information, Ubiquity Press. Available online: https:\/\/doi.org\/10.5334\/bax.","DOI":"10.5334\/bax"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Foody, G., See, L., Fritz, S., Mooney, P., Olteanu-Raimond, A.M., Fonte, C.C., and Antoniou, V. (2017). Mapping and the Citizen Sensor, Ubiquity Press. Available online: https:\/\/doi.org\/10.5334\/bbf.","DOI":"10.5334\/bbf"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1080\/17538941003759255","article-title":"Crowdsourcing geographic information for disaster response: A research frontier","volume":"3","author":"Goodchild","year":"2010","journal-title":"Int. J. Dig. Earth"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"97","DOI":"10.5194\/nhess-13-97-2013","article-title":"Collaborative damage mapping for emergency response: The role of Cognitive Systems Engineering","volume":"13","author":"Kerle","year":"2013","journal-title":"Nat. Hazards Earth Syst. Sci."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"11088","DOI":"10.1073\/pnas.1202383109","article-title":"Next-generation digital earth","volume":"109","author":"Goodchild","year":"2012","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Foody, G., See, L., Fritz, S., Mooney, P., Olteanu-Raimond, A.-M., Fonte, C.C., and Antoniou, V. (2017). A review of OpenStreetMap data. Mapping and the Citizen Sensor, Ubiquity Press. Available online: https:\/\/doi.org\/10.5334\/bbf.","DOI":"10.5334\/bbf"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Foody, G.M. (2015). Citizen science in support of remote sensing research. International Geoscience and Remote Sensing Symposium, IEEE. Available online: http:\/\/ieeexplore.ieee.org\/stamp\/stamp.jsp?tp=&arnumber=7326952.","DOI":"10.1109\/IGARSS.2015.7327053"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1007\/s10708-008-9188-y","article-title":"The credibility of volunteered geographic information","volume":"72","author":"Flanagin","year":"2008","journal-title":"GeoJournal"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Koswatte, S., McDougall, K., and Liu, X. (2017). VGI and crowdsourced data credibility analysis using spam email detection techniques. Int. J. Dig. Earth, 1\u201313. Available online: http:\/\/www.tandfonline.com\/doi\/abs\/10.1080\/17538947.2017.1341558.","DOI":"10.1080\/17538947.2017.1341558"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Foody, G., See, L., Fritz, S., Mooney, P., Olteanu-Raimond, A.-M., Fonte, C.C., and Antoniou, V. (2017). Assessing VGI data quality. Mapping and the Citizen Sensor, Ubiquity Press. Available online: https:\/\/doi.org\/10.5334\/bbf.","DOI":"10.5334\/bbf"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"315","DOI":"10.1179\/000870410X12911304958827","article-title":"How many volunteers does it take to map an area well? The validity of Linus\u2019 Law to volunteered geographic information","volume":"47","author":"Haklay","year":"2010","journal-title":"Cartogr. J."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"394","DOI":"10.1080\/1369118X.2011.641991","article-title":"The myth of amateur crowds: A critical discourse analysis of crowdsourcing coverage","volume":"15","author":"Brabham","year":"2012","journal-title":"Inf. Commun. Soc."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Comber, A., Mooney, P., Purves, R.S., Rocchini, D., and Walz, A. (2016). Crowdsourcing: It matters who the crowd are. The impacts of between group variations in recording land cover. PLoS ONE, 11, Available online: https:\/\/doi.org\/10.1371\/journal.pone.0158329.","DOI":"10.1371\/journal.pone.0158329"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"See, L., Comber, A., Salk, C., Fritz, S., van der Velde, M., Perger, C., Schill, C., McCallum, I., Kraxner, F., and Obersteiner, M. (2013). Comparing the quality of crowdsourced data contributed by expert and non-experts. PLoS ONE, 8, Available online: https:\/\/doi.org\/10.1371\/journal.pone.0069958.","DOI":"10.1371\/journal.pone.0069958"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1111\/tgis.12194","article-title":"Limitations of majority agreement in crowdsourced image interpretation","volume":"21","author":"Salk","year":"2017","journal-title":"Trans. GIS"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.spasta.2012.03.002","article-title":"Assuring the quality of volunteered geographic information","volume":"1","author":"Goodchild","year":"2012","journal-title":"Spat. Stat."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1121","DOI":"10.1037\/0022-3514.77.6.1121","article-title":"Unskilled and unaware of it: How difficulties in recognizing one\u2019s own incompetence lead to inflated self-assessments","volume":"77","author":"Kruger","year":"1999","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"532","DOI":"10.1038\/nature21054","article-title":"A solution to the single-question crowd wisdom problem","volume":"541","author":"Prelec","year":"2017","journal-title":"Nature"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Gengler, S., and Bogaert, P. (2016). Integrating crowdsourced data with a land cover product: A Bayesian data fusion approach. Remote Sens., 8, Available online: http:\/\/www.mdpi.com\/2072-4292\/8\/7\/545\/htm.","DOI":"10.3390\/rs8070545"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"847","DOI":"10.1111\/tgis.12033","article-title":"Assessing the accuracy of volunteered geographic information arising from multiple contributors to an internet based collaborative project","volume":"17","author":"Foody","year":"2013","journal-title":"Trans. GIS"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"336","DOI":"10.1080\/00087041.2015.1108658","article-title":"Accurate attribute mapping from volunteered geographic information: Issues of volunteer quantity and quality","volume":"52","author":"Foody","year":"2015","journal-title":"Cartogr. J."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.envsoft.2011.11.015","article-title":"Geo-Wiki: An online platform for improving global land cover","volume":"31","author":"Fritz","year":"2012","journal-title":"Environ. Modell. Softw."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"324","DOI":"10.1016\/j.techfore.2015.03.002","article-title":"Harnessing the power of volunteers, the internet and Google Earth to collect and validate global spatial information using Geo-Wiki","volume":"98","author":"See","year":"2015","journal-title":"Technol. Forecast. Soc. Chang."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"170075","DOI":"10.1038\/sdata.2017.75","article-title":"A global dataset of crowdsourced land cover and land use reference data","volume":"4","author":"Fritz","year":"2017","journal-title":"Sci. Data"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Boyd, D., Jackson, B., Wardlaw, J., Foody, G., Marsh, S., and Bales, K. (ISPRS J. Photogramm. Remote Sens., 2018). Slavery from space: Demonstrating the role for satellite remote sensing to inform evidence-based action related to UN SDG number 8, ISPRS J. Photogramm. Remote Sens., in press.","DOI":"10.1016\/j.isprsjprs.2018.02.012"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"2271","DOI":"10.1016\/j.rse.2010.05.003","article-title":"Assessing the accuracy of land cover change with imperfect ground reference data","volume":"114","author":"Foody","year":"2010","journal-title":"Remote Sens. Environ."},{"key":"ref_31","first-page":"1075","article-title":"The DISCover validation image interpretation process","volume":"65","author":"Scepan","year":"1999","journal-title":"Photogramm. Eng. Remote Sens."},{"key":"ref_32","first-page":"549","article-title":"Latent class analysis","volume":"Volume 2","author":"Bryman","year":"2003","journal-title":"The Sage Encyclopedia of Social Science Research Methods"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"531","DOI":"10.1016\/S0167-9473(02)00179-2","article-title":"Latent class models for classification","volume":"41","author":"Vermunt","year":"2003","journal-title":"Comput. Stat. Data Anal."}],"container-title":["ISPRS International Journal of Geo-Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2220-9964\/7\/3\/80\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T14:56:18Z","timestamp":1760194578000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2220-9964\/7\/3\/80"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,2,25]]},"references-count":33,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2018,3]]}},"alternative-id":["ijgi7030080"],"URL":"https:\/\/doi.org\/10.3390\/ijgi7030080","relation":{},"ISSN":["2220-9964"],"issn-type":[{"value":"2220-9964","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,2,25]]}}}