{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,3]],"date-time":"2026-02-03T17:23:45Z","timestamp":1770139425397,"version":"3.49.0"},"reference-count":43,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2019,7,31]],"date-time":"2019-07-31T00:00:00Z","timestamp":1564531200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100007316","name":"Klaus Tschira Stiftung","doi-asserted-by":"publisher","award":["HeiGIT"],"award-info":[{"award-number":["HeiGIT"]}],"id":[{"id":"10.13039\/501100007316","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["Open Access Publishing"],"award-info":[{"award-number":["Open Access Publishing"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>Reliable techniques to generate accurate data sets of human built-up areas at national, regional, and global scales are a key factor to monitor the implementation progress of the Sustainable Development Goals as defined by the United Nations. However, the scarce availability of accurate and up-to-date human settlement data remains a major challenge, e.g., for humanitarian organizations. In this paper, we investigated the complementary value of crowdsourcing and deep learning to fill the data gaps of existing earth observation-based (EO) products. To this end, we propose a novel workflow to combine deep learning (DeepVGI) and crowdsourcing (MapSwipe). Our strategy for allocating classification tasks to deep learning or crowdsourcing is based on confidence of the derived binary classification. We conducted case studies in three different sites located in Guatemala, Laos, and Malawi to evaluate the proposed workflow. Our study reveals that crowdsourcing and deep learning outperform existing EO-based approaches and products such as the Global Urban Footprint. Compared to a crowdsourcing-only approach, the combination increased the quality (measured by Matthew\u2019s correlation coefficient) of the generated human settlement maps by 3 to 5 percentage points. At the same time, it reduced the volunteer efforts needed by at least 80 percentage points for all study sites. The study suggests that for the efficient creation of human settlement maps, we should rely on human skills when needed and rely on automated approaches when possible.<\/jats:p>","DOI":"10.3390\/rs11151799","type":"journal-article","created":{"date-parts":[[2019,7,31]],"date-time":"2019-07-31T11:37:07Z","timestamp":1564573027000},"page":"1799","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":48,"title":["Mapping Human Settlements with Higher Accuracy and Less Volunteer Efforts by Combining Crowdsourcing and Deep Learning"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9738-4060","authenticated-orcid":false,"given":"Benjamin","family":"Herfort","sequence":"first","affiliation":[{"name":"GIScience Chair, Institute of Geography, Heidelberg University, 69120 Heidelberg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6336-8772","authenticated-orcid":false,"given":"Hao","family":"Li","sequence":"additional","affiliation":[{"name":"GIScience Chair, Institute of Geography, Heidelberg University, 69120 Heidelberg, Germany"}]},{"given":"Sascha","family":"Fendrich","sequence":"additional","affiliation":[{"name":"GIScience Chair, Institute of Geography, Heidelberg University, 69120 Heidelberg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1825-9996","authenticated-orcid":false,"given":"Sven","family":"Lautenbach","sequence":"additional","affiliation":[{"name":"GIScience Chair, Institute of Geography, Heidelberg University, 69120 Heidelberg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4916-9838","authenticated-orcid":false,"given":"Alexander","family":"Zipf","sequence":"additional","affiliation":[{"name":"GIScience Chair, Institute of Geography, Heidelberg University, 69120 Heidelberg, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2019,7,31]]},"reference":[{"key":"ref_1","unstructured":"United Nations Department of Economic and Social Affairs Population Division (2018). World Urbanization Prospects: The 2018 Revision, United Nations Department of Economic and Social Affairs Population Division. Technical Report."},{"key":"ref_2","unstructured":"United Nations (2019, July 31). Transforming Our World: The 2030 Agenda for sUstainable Development, Available online: http:\/\/xxx.lanl.gov\/abs\/arXiv:1011.1669v3."},{"key":"ref_3","unstructured":"United Nations Office for Disaster Risk Reduction (2015). Sendai Framework for Disaster Risk Reduction 2015\u20132030, United Nations Office for Disaster Risk Reduction. Technical Report."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2102","DOI":"10.1109\/JSTARS.2013.2271445","article-title":"A global human settlement layer from optical HR\/VHR RS data: Concept and first results","volume":"6","author":"Pesaresi","year":"2013","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1617","DOI":"10.1109\/LGRS.2013.2272953","article-title":"Urban footprint processor-Fully automated processing chain generating settlement masks from global data of the TanDEM-X mission","volume":"10","author":"Esch","year":"2013","journal-title":"IEEE Geosci. Remote Sens. Lett."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Tiecke, T.G., Liu, X., Zhang, A., Gros, A., Li, N., Yetman, G., Kilic, T., Murray, S., Blankespoor, B., and Prydz, E.B. (2017). Mapping the world Population One Building at a Time. arXiv.","DOI":"10.1596\/33700"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1016\/j.rse.2016.03.001","article-title":"How good is the map? A multi-scale cross-comparison framework for global settlement layers: Evidence from Central Europe","volume":"178","author":"Klotz","year":"2016","journal-title":"Remote Sens. Environ."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"524","DOI":"10.1109\/JSTARS.2016.2616120","article-title":"Mapping Human Settlements and Population at Country Scale from VHR Images","volume":"10","author":"Gueguen","year":"2017","journal-title":"IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Albuquerque, J., Herfort, B., and Eckle, M. (2016). The Tasks of the Crowd: A Typology of Tasks in Geographic Information Crowdsourcing and a Case Study in Humanitarian Mapping. Remote Sens., 8.","DOI":"10.3390\/rs8100859"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Hachmann, S., Jokar Arsanjani, J., and Vaz, E. (2017). Spatial data for slum upgrading: Volunteered Geographic Information and the role of citizen science. Habitat Int.","DOI":"10.1016\/j.habitatint.2017.04.011"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Scholz, S., Knight, P., Eckle, M., Marx, S., and Zipf, A. (2018). Volunteered Geographic Information for Disaster Risk Reduction\u2014The Missing Maps Approach and Its Potential within the Red Cross and Red Crescent Movement. Remote Sens., 10.","DOI":"10.3390\/rs10081239"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"See, L., Mooney, P., Foody, G., Bastin, L., Comber, A., Estima, J., Fritz, S., Kerle, N., Jiang, B., and Laakso, M. (2016). Crowdsourcing, Citizen Science or Volunteered Geographic Information? The Current State of Crowdsourced Geographic Information. ISPRS Int. J. Geo-Inf., 5.","DOI":"10.3390\/ijgi5050055"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1066","DOI":"10.3390\/ijgi2041066","article-title":"Measuring Completeness of Building Footprints in OpenStreetMap over Space and Time","volume":"2","author":"Hecht","year":"2013","journal-title":"ISPRS Int. J. Geo-Inf."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Chen, J., and Zipf, A. (2017, January 3\u20137). DeepVGI: Deep Learning with Volunteered Geographic Information. Proceedings of the WWW \u201917 Companion: Proceedings of the 26th International Conference Companion on World Wide Web, Perth, Australia.","DOI":"10.1145\/3041021.3054250"},{"key":"ref_15","unstructured":"Herfort, B., Reinmuth, M., de Albuquerque, J.P., and Zipf, A. (2017, January 9\u201312). Towards evaluating crowdsourced image classification on mobile devices to generate geographic information about human settlements. Proceedings of the 20th AGILE, Wageningen, The Netherlands."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1016\/j.isprsjprs.2017.10.012","article-title":"Breaking new ground in mapping human settlements from space \u2013 The Global Urban Footprint","volume":"134","author":"Esch","year":"2017","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"6","DOI":"10.2202\/1948-4682.1069","article-title":"Volunteered Geographic Information and Crowdsourcing Disaster Relief: A Case Study of the Haitian Earthquake","volume":"2","author":"Zook","year":"2010","journal-title":"World Med. Health Policy"},{"key":"ref_18","unstructured":"Degrossi, L.C. (2016, January 5\u20138). Potential of Collaborative Mapping for Disaster Relief: A Case Study of OpenStreetMap in the Nepal Earthquake 2015. Proceedings of the 2016 49th Hawaii International Conference on System Sciences (HICSS), Koloa, HI, USA."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"720","DOI":"10.1080\/13658816.2014.977905","article-title":"Crowdsourcing urban form and function","volume":"29","author":"Crooks","year":"2015","journal-title":"Int. J. Geogr. Inf. Sci."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1016\/j.spasta.2012.03.002","article-title":"Assuring the quality of volunteered geographic information","volume":"1","author":"Goodchild","year":"2012","journal-title":"Spat. Stat."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Fabrikant, S.I., Raubal, M., Bertolotto, M., Davies, C., Freundschuh, S., and Bell, S. (2015, January 12\u201316). A Conceptual Quality Framework for Volunteered Geographic Information. Proceedings of the Spatial Information Theory: 12th International Conference, COSIT 2015, Santa Fe, NM, USA.","DOI":"10.1007\/978-3-319-23374-1"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"700","DOI":"10.1080\/13658816.2013.867495","article-title":"Quality assessment for building footprints data on OpenStreetMap","volume":"28","author":"Fan","year":"2014","journal-title":"Int. J. Geogr. Inf. Sci."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Comber, A., Mooney, P., Purves, R.S., Rocchini, D., and Walz, A. (2016). Crowdsourcing: It matters who the crowd are. The impacts of between group variations in recording land cover. PLoS ONE, 11.","DOI":"10.1371\/journal.pone.0158329"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MGRS.2017.2762307","article-title":"Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources","volume":"5","author":"Zhu","year":"2017","journal-title":"IEEE Geosci. Remote Sens. Mag."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Jean, N., Burke, M., Xie, M., Davis, W.M., Lobell, D.B., and Ermon, S. (2016). Combining satellite imagery and machine learning to predict poverty. Science.","DOI":"10.1126\/science.aaf7894"},{"key":"ref_26","unstructured":"Li, H., Herfort, B., and Zipf, A. (2019, January 17\u201320). Estimating OpenStreetMap Missing Built-up Areas using Pre-trained Deep Neural Networks. Proceedings of the 22nd AGILE, At Limassol, Cyprus."},{"key":"ref_27","unstructured":"Kampffmeyer, M., Salberg, A.B., and Jenssen, R. (July, January 26). Semantic Segmentation of Small Objects and Modeling of Uncertainty in Urban Remote Sensing Images Using Deep Convolutional Neural Networks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Las Vegas, NV, USA."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"252","DOI":"10.1080\/10095020.2017.1373955","article-title":"Repurposing a deep learning network to filter and classify volunteered photographs for land cover and land use characterization","volume":"20","author":"Tracewski","year":"2017","journal-title":"Geo-Spat. Inf. Sci."},{"key":"ref_29","first-page":"1873","article-title":"Building detection in very high resolution multispectral data with deep learning features","volume":"2015","author":"Vakalopoulou","year":"2015","journal-title":"Int. Geosci. Remote Sens. Symp. (IGARSS)"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"180217","DOI":"10.1038\/sdata.2018.217","article-title":"Exploiting deep learning and volunteered geographic information for mapping buildings in Kano, Nigeria","volume":"5","author":"Yuan","year":"2018","journal-title":"Sci. Data"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Chen, J., Zhou, Y., Zipf, A., and Fan, H. (2018). Deep Learning From Multiple Crowds: A Case Study of Humanitarian Mapping. IEEE Trans. Geosci. Remote Sens.","DOI":"10.1109\/TGRS.2018.2868748"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1016\/j.isprsjprs.2018.11.010","article-title":"Correcting rural building annotations in OpenStreetMap using convolutional neural networks","volume":"147","author":"Lobry","year":"2019","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_33","unstructured":"Maso, J., Pomakis, K., and Julia, N. (2010). OpenGIS Web Map Tile Service Implementation Standard, Open Geospatial Consortium Inc."},{"key":"ref_34","first-page":"21","article-title":"SSD: Single shot multibox detector","volume":"Volume 9905 LNCS","author":"Liu","year":"2016","journal-title":"Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref_35","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2019, July 31). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Available online: tensorflow.org."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll\u00e1r, P., and Zitnick, C.L. (2014, January 24\u201327). Microsoft COCO: Common Objects in Context. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., and Guadarrama, S. (2017, January 21\u201326). Speed\/accuracy trade-offs for modern convolutional object detectors. Proceedings of the IEEE CVPR, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.351"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1016\/j.patcog.2019.02.023","article-title":"The impact of class imbalance in classification performance metrics based on the binary confusion matrix","volume":"91","author":"Luque","year":"2019","journal-title":"Pattern Recognit."},{"key":"ref_39","unstructured":"Jones, E., Oliphant, T., and Peterson, P. (2019, July 31). Available online: https:\/\/scholar.google.com\/scholar?cluster=2086009121748039507&hl=en&oi=scholarr."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Scott, D.W. (2015). Multivariate Density Estimation: Theory, Practice, and Visualization, John Wiley & Sons.","DOI":"10.1002\/9781118575574"},{"key":"ref_41","unstructured":"McFadden, D. (2019, July 31). Available online: https:\/\/eml.berkeley.edu\/reprints\/mcfadden\/zarembka.pdf."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Seabold, S., and Perktold, J. (2010, January 28\u201330). Statsmodels: Econometric and statistical modeling with python. Proceedings of the 9th Python in Science Conference, Austin, TX, USA.","DOI":"10.25080\/Majora-92bf1922-011"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Salk, C.F., Sturn, T., See, L., and Fritz, S. (2016). Limitations of Majority Agreement in Crowdsourced Image Interpretation. Trans. GIS.","DOI":"10.1111\/tgis.12194"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/11\/15\/1799\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:11:52Z","timestamp":1760188312000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/11\/15\/1799"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7,31]]},"references-count":43,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2019,8]]}},"alternative-id":["rs11151799"],"URL":"https:\/\/doi.org\/10.3390\/rs11151799","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,7,31]]}}}