{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T22:30:04Z","timestamp":1767911404173,"version":"3.49.0"},"reference-count":25,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2021,9,30]],"date-time":"2021-09-30T00:00:00Z","timestamp":1632960000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100006517","name":"Edna Bailey Sussman Foundation","doi-asserted-by":"publisher","award":["24608"],"award-info":[{"award-number":["24608"]}],"id":[{"id":"10.13039\/100006517","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["OCE-1637630"],"award-info":[{"award-number":["OCE-1637630"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Remote Sensing"],"abstract":"<jats:p>The Total Operating Characteristic (TOC) measures how the ranks of an index variable distinguish between presence and absence in a binary reference variable. Previous methods to generate the TOC required the reference data to derive from a census or a simple random sample. However, many researchers apply stratified random sampling to collect reference data because stratified random sampling is more efficient than simple random sampling for many applications. Our manuscript derives a new methodology that uses stratified random sampling to generate the TOC. An application to flood mapping illustrates how the TOC compares the abilities of three indices to diagnose water. The TOC shows visually and quantitatively each index\u2019s diagnostic ability relative to baselines. Results show that the Modified Normalized Difference Water Index has the greatest diagnostic ability, while the Normalized Difference Vegetation Index has diagnostic ability greater than the Normalized Difference Water Index at the threshold where the Diagnosed Presence equals the Abundance of water. Some researchers consider only one accuracy metric at only one threshold, whereas the TOC allows visualization of several metrics at all thresholds. The TOC gives more information and clearer interpretation compared to the popular Relative Operating Characteristic. Our software generates the TOC from a census, simple random sample, or stratified random sample. The TOC Curve Generator is free as an executable file at a website that our manuscript gives.<\/jats:p>","DOI":"10.3390\/rs13193922","type":"journal-article","created":{"date-parts":[[2021,10,8]],"date-time":"2021-10-08T21:26:20Z","timestamp":1633728380000},"page":"3922","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["The Total Operating Characteristic from Stratified Random Sampling with an Application to Flood Mapping"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1785-550X","authenticated-orcid":false,"given":"Zhen","family":"Liu","sequence":"first","affiliation":[{"name":"Clark Labs, Clark University, Worcester, MA 01602, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7287-5875","authenticated-orcid":false,"given":"Robert Gilmore","family":"Pontius Jr","sequence":"additional","affiliation":[{"name":"School of Geography, Clark University, Worcester, MA 01602, USA"}]}],"member":"1968","published-online":{"date-parts":[[2021,9,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1038\/s41586-021-03695-w","article-title":"Satellite imaging reveals increased proportion of population exposed to floods","volume":"596","author":"Tellman","year":"2021","journal-title":"Nature"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1435","DOI":"10.1080\/01431168608948946","article-title":"Analysis of the Dynamics of African Vegetation Using the Normalized Difference Vegetation Index","volume":"7","author":"Townsend","year":"1986","journal-title":"Int. J. Remote Sens."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1425","DOI":"10.1080\/01431169608948714","article-title":"The Use of the Normalized Difference Water Index (NDWI) in the Delineation of Open Water Features","volume":"17","author":"McFeeters","year":"1996","journal-title":"Int. J. Remote Sens."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3025","DOI":"10.1080\/01431160600589179","article-title":"Modification of Normalised Difference Water Index (NDWI) to Enhance Open Water Features in Remotely Sensed Imagery","volume":"27","author":"Xu","year":"2006","journal-title":"Int. J. Remote Sens."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Leiden, J., Sorn, S., Hem, S., Huy, R., Buchy, P., Tarantola, A., and Capelle, J. (2017). Assessing the performance of remotely-sensed flooding indicators and their potential contribution to early warning for leptospirosis in Cambodia. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0181044"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e0181044","DOI":"10.1007\/s11707-020-0818-0","article-title":"Rapid flood inundation mapping by differencing water indices from pre- and post-flood Landsat images","volume":"15","author":"Sivanpillai","year":"2021","journal-title":"Front. Earth Sci."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1016\/j.rse.2013.08.029","article-title":"Automated Water Extraction Index: A New Technique for Surface Water Mapping Using Landsat Imagery","volume":"140","author":"Feyisa","year":"2014","journal-title":"Remote Sens. Environ."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"834","DOI":"10.1111\/1752-1688.12626","article-title":"Intercomparison of Satellite Remote Sensing-Based Flood Inundation Mapping Techniques","volume":"54","author":"Munasinghe","year":"2018","journal-title":"J. Am. Water Resour. Assoc."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1307","DOI":"10.14358\/PERS.75.11.1307","article-title":"Analysis of Dynamic Thresholds for the Normalized Difference Water Index","volume":"75","author":"Ji","year":"2009","journal-title":"Photogramm. Eng. Remote Sens."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"570","DOI":"10.1080\/13658816.2013.862623","article-title":"The Total Operating Characteristic to Measure Diagnostic Ability for Multiple Thresholds","volume":"28","author":"Si","year":"2014","journal-title":"Int. J. Geogr. Inf. Sci."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1581","DOI":"10.1007\/s10980-017-0520-0","article-title":"Multiple-Scale Prediction of Forest Loss Risk across Borneo","volume":"32","author":"Cushman","year":"2017","journal-title":"Landsc. Ecol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1080\/10402381.2015.1053010","article-title":"Predicting the Extent of Lakeshore Development Using GIS Datasets","volume":"31","author":"Dustin","year":"2015","journal-title":"Lake Reserv. Manag."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2919","DOI":"10.1007\/s10980-021-01272-0","article-title":"Validating Models of One-Way Land Change: An Example Case of Forest Insect Disturbance","volume":"36","author":"Harati","year":"2021","journal-title":"Landsc. Ecol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"447","DOI":"10.3390\/ijgi4020447","article-title":"Simulating Urban Growth Using a Random Forest-Cellular Automata (RF-CA) Model","volume":"4","author":"Kamusoko","year":"2015","journal-title":"IJGI"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"101595","DOI":"10.1016\/j.compenvurbsys.2021.101595","article-title":"Integrating a Forward Feature Selection Algorithm, Random Forest, and Cellular Automata to Extrapolate Urban Growth in the Tehran-Karaj Region of Iran","volume":"87","author":"Minaei","year":"2021","journal-title":"Comput. Environ. Urban Syst."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1285","DOI":"10.1126\/science.3287615","article-title":"Measuring the Accuracy of Diagnostic Systems","volume":"240","author":"Swets","year":"1988","journal-title":"Science"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1038\/scientificamerican1000-82","article-title":"Better DECISIONS through SCIENCE","volume":"283","author":"Swets","year":"2000","journal-title":"Sci. Am."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/j.rse.2014.02.015","article-title":"Good Practices for Estimating Area and Assessing Accuracy of Land Change","volume":"148","author":"Olofsson","year":"2014","journal-title":"Remote Sens. Environ."},{"key":"ref_19","unstructured":"(2021, September 24). Vox Media. What the Historic Midwest Floods Look Like from Space\u2014And from the Ground. Available online: https:\/\/www.vox.com\/energy-and-environment\/2019\/3\/18\/18271101\/nebraska-flooding-photos."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"4407","DOI":"10.1080\/01431161.2011.552923","article-title":"Death to Kappa: Birth of Quantity Disagreement and Allocation Disagreement for Accuracy Assessment","volume":"32","author":"Millones","year":"2011","journal-title":"Int. J. Remote Sens."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"111630","DOI":"10.1016\/j.rse.2019.111630","article-title":"Explaining the Unsuitability of the Kappa Coefficient in the Assessment and Comparison of the Accuracy of Thematic Maps Obtained by Image Classification","volume":"239","author":"Foody","year":"2020","journal-title":"Remote Sens. Environ."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1017\/S0376892997000088","article-title":"A Review of Methods for the Assessment of Prediction Errors in Conservation Presence\/Absence Models","volume":"24","author":"Fielding","year":"1997","journal-title":"Environ. Conserv."},{"key":"ref_23","unstructured":"Liu, Z. (2021, September 24). TOC Curve Generator. Available online: https:\/\/lazygis.github.io\/projects\/TOCCurveGenerator."},{"key":"ref_24","unstructured":"Pontius Jr, R.G., Santacruz, A., Tayyebi, A., and Parmentier, B. (2021, September 24). TOC: Total Operating Characteristic Curve and ROC Curve. Available online: https:\/\/cran.r-project.org\/web\/packages\/TOC."},{"key":"ref_25","first-page":"367","article-title":"Recommendations for using the relative operating characteristic (ROC)","volume":"29","author":"Parmentier","year":"1997","journal-title":"Landsc. Ecol."}],"container-title":["Remote Sensing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/19\/3922\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T07:07:55Z","timestamp":1760166475000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2072-4292\/13\/19\/3922"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,30]]},"references-count":25,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2021,10]]}},"alternative-id":["rs13193922"],"URL":"https:\/\/doi.org\/10.3390\/rs13193922","relation":{},"ISSN":["2072-4292"],"issn-type":[{"value":"2072-4292","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,30]]}}}