{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,17]],"date-time":"2026-01-17T23:32:33Z","timestamp":1768692753912,"version":"3.49.0"},"reference-count":29,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2015,2,26]],"date-time":"2015-02-26T00:00:00Z","timestamp":1424908800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000203","name":"U.S. Geological Survey","doi-asserted-by":"publisher","award":["G12PC00073"],"award-info":[{"award-number":["G12PC00073"]}],"id":[{"id":"10.13039\/100000203","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>We recruit an online labor force through Amazon.com\u2019s Mechanical Turk platform to identify clouds and cloud shadows in Landsat satellite images. We find that a large group of workers can be mobilized quickly and relatively inexpensively. Our results indicate that workers\u2019 accuracy is insensitive to wage, but deteriorates with the complexity of images and with time-on-task. In most instances, human interpretation of cloud impacted area using a majority rule was more accurate than an automated algorithm (Fmask) commonly used to identify clouds and cloud shadows. However, cirrus-impacted pixels were better identified by Fmask than by human interpreters. Crowd-sourced interpretation of cloud impacted pixels appears to be a promising means by which to augment or potentially validate fully automated algorithms.<\/jats:p>","DOI":"10.3390\/rs70302334","type":"journal-article","created":{"date-parts":[[2015,2,26]],"date-time":"2015-02-26T10:07:43Z","timestamp":1424945263000},"page":"2334-2351","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Cloud-Sourcing: Using an Online Labor Force to Detect Clouds and Cloud Shadows in Landsat Images"],"prefix":"10.3390","volume":"7","author":[{"given":"Ling","family":"Yu","sequence":"first","affiliation":[{"name":"Department of Agricultural and Applied Economics, Virginia Tech, 208 Hutcheson Hall (0401), 250 Drillfield Drive, Blacksburg, VA 24060, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sheryl","family":"Ball","sequence":"additional","affiliation":[{"name":"Department of Economics, Virginia Tech, 3016 Pamplin Hall (0316), Blacksburg, VA 24061, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Christine","family":"Blinn","sequence":"additional","affiliation":[{"name":"Department of Forest Resources and Environmental Conservation, Virginia Tech, 310 West Campus Dr., Blacksburg, VA 24061, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Klaus","family":"Moeltner","sequence":"additional","affiliation":[{"name":"Department of Agricultural and Applied Economics, Virginia Tech, 208 Hutcheson Hall (0401), 250 Drillfield Drive, Blacksburg, VA 24060, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Seth","family":"Peery","sequence":"additional","affiliation":[{"name":"Virginia Tech Information Technology, 1700 Pratt Drive (0214), Blacksburg, VA 24061, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2189-6013","authenticated-orcid":false,"given":"Valerie","family":"Thomas","sequence":"additional","affiliation":[{"name":"Department of Forest Resources and Environmental Conservation, Virginia Tech, 310 West Campus Dr., Blacksburg, VA 24061, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3649-835X","authenticated-orcid":false,"given":"Randolph","family":"Wynne","sequence":"additional","affiliation":[{"name":"Department of Forest Resources and Environmental Conservation, Virginia Tech, 310 West Campus Dr., Blacksburg, VA 24061, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2015,2,26]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1016\/j.rse.2011.10.028","article-title":"Object-based cloud and cloud shadow detection in Landsat imagery","volume":"118","author":"Zhu","year":"2012","journal-title":"Remote Sens. Environ."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"4907","DOI":"10.3390\/rs6064907","article-title":"Automated detection of cloud and cloud shadow in single-date Landsat imagery using neural networks and spatial post-processing","volume":"6","author":"Hughes","year":"2014","journal-title":"Remote Sens."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"776","DOI":"10.3390\/rs6010776","article-title":"Cloud and cloud-shadow detection in SPOT5 HRG imagery with automated morphological feature extraction","volume":"6","author":"Fisher","year":"2014","journal-title":"Remote Sens."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"345","DOI":"10.3390\/rs1030345","article-title":"Geo-Wiki.Org: The use of crowdsourcing to improve global land cover","volume":"1","author":"Fritz","year":"2009","journal-title":"Remote Sens."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"See, L., Schepaschenko, D., Lesiv, M., McCallum, I., Fritz, S., Comber, A., Perger, C., Schill, C., Zhao, Y., and Maus, V. (2015). Building a hybrid land cover map with crowdsourcing and geographically weighted regression. ISPRS J. Photogramm. Remote Sens., in press.","DOI":"10.1016\/j.isprsjprs.2014.06.016"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Fritz, S., See, L., McCallum, I., You, L., Bun, A., Moltchanova, E., Duerauer, M., Albrecht, F., Schill, C., and Perger, C. (2015). Mapping global cropland and field size. Glob. Chang. Biol., in press.","DOI":"10.1111\/gcb.12838"},{"key":"ref_7","first-page":"89","article-title":"Volunteered geographic information for disaster management with application to rapid flood estimation","volume":"64","author":"Poser","year":"2010","journal-title":"Geomatica"},{"key":"ref_8","unstructured":"Horita, F., Degrossi, L., Assis, L., Zipf, A., and de Albuquerque, J. (2013, January 15\u201317). The use of volunteered geographic information and crowdsourcing in disaster management: A systematic literature review, Chicago, IL, USA."},{"key":"ref_9","unstructured":"Hansen, L. (2015, January 15). Global forest watch-fires: Improving remote sensing through community engagement, San Jose, CA, USA."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s10708-007-9111-y","article-title":"Citizen as sensors: The world of volunteered geography","volume":"69","author":"Goodchild","year":"2007","journal-title":"GeoJournal"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1007\/s10683-011-9273-9","article-title":"The online laboratory: Conducting experiments in real labor markets","volume":"14","author":"Horton","year":"2011","journal-title":"Exp. Econ."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3758\/s13428-011-0124-6","article-title":"Conducting behavioral research on Amazon\u2019s Mechanical Turk","volume":"44","author":"Mason","year":"2012","journal-title":"Behav. Res. Methods."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1093\/pan\/mpr057","article-title":"Evaluating online labor markets for experimental research: Amazon.com\u2019s Mechanical Turk","volume":"20","author":"Berinsky","year":"2012","journal-title":"Polit. Anal."},{"key":"ref_14","unstructured":"Available online: http:\/\/docs.aws.amazon.com\/AWSMechTurk\/latest\/AWSMturkAPI\/ApiReference_ExternalQuestionArticle.html."},{"key":"ref_15","unstructured":"Available online: http:\/\/turkernation.com\/."},{"key":"ref_16","unstructured":"Available online: http:\/\/mturkforum.com\/forum.php."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1016\/j.jebo.2013.03.003","article-title":"Breaking monotony with meaning: Motivation in crowdsourcing markets","volume":"90","author":"Chandler","year":"2013","journal-title":"J. Econ. Behav. Organ."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1177\/1745691610393980","article-title":"Amazon\u2019s Mechanical Turk: A new source of inexpensive, yet high-quality data?","volume":"6","author":"Buhrmester","year":"2011","journal-title":"Perspect. Psychol. Sci."},{"key":"ref_19","first-page":"1100","article-title":"Financial incentives and the \u201cperformance of the crowds\u201d","volume":"11","author":"Mason","year":"2009","journal-title":"SIGKDD Explor."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1257\/jep.14.3.159","article-title":"Fairness and retaliation: The economics of reciprocity","volume":"14","author":"Fehr","year":"2000","journal-title":"J. Econ. Perspect."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1016\/j.jebo.2003.11.011","article-title":"Fairness as a source of hysteresis in employment and relative wages","volume":"57","author":"Scott","year":"2005","journal-title":"J. Econ. Behav. Organ."},{"key":"ref_22","first-page":"1347","article-title":"Fairness perception and reservation wages \u2014 the behavioral effects of minimum wage laws","volume":"121","author":"Falk","year":"2006","journal-title":"Q. J. Econ."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1111\/j.1468-0262.2007.00734.x","article-title":"Fairness and contract design","volume":"75","author":"Fehr","year":"2007","journal-title":"Econometrica"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"650","DOI":"10.2307\/3087479","article-title":"Motivation and optimal incentive contracts","volume":"33","author":"Murdock","year":"2002","journal-title":"RAND J. Econ."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"136","DOI":"10.1037\/npe0000006","article-title":"Reconciling economics and psychology on intrinsic motivation","volume":"6","author":"Bruno","year":"2013","journal-title":"J. Neurosci. Psychol. Econ."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"2875","DOI":"10.1257\/aer.103.7.2875","article-title":"Information and quality when motivation is intrinsic: Evidence from surgeon report cards","volume":"103","author":"Kolstad","year":"2013","journal-title":"Am. Econ. Rev."},{"key":"ref_27","unstructured":"Willett, W., Heer, J., and Agrawala, M. (2014, January 5\u201310). Strategies for crowdsourcing social data analysis, Austin, TX, USA."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1002\/bdm.1753","article-title":"Data collection in a flat world: The strength and weaknesses of Mechanical Turk samples","volume":"26","author":"Goodman","year":"2013","journal-title":"J. Behav. Decis. Mak."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Shaw, A., Horton, J., and Chen, D. (2011, January 19\u201323). Designing incentives for inexpert human raters, Hangzhou, China.","DOI":"10.1145\/1958824.1958865"}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/7\/3\/2334\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T20:42:57Z","timestamp":1760215377000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/7\/3\/2334"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,2,26]]},"references-count":29,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2015,3]]}},"alternative-id":["rs70302334"],"URL":"https:\/\/doi.org\/10.3390\/rs70302334","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2015,2,26]]}}}