{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,13]],"date-time":"2026-01-13T08:50:02Z","timestamp":1768294202927,"version":"3.49.0"},"reference-count":39,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2025,3,23]],"date-time":"2025-03-23T00:00:00Z","timestamp":1742688000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"United States National Aeronautics and Space Administration","award":["80NSSC23K0508"],"award-info":[{"award-number":["80NSSC23K0508"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJGI"],"abstract":"<jats:p>The Total Operating Characteristic (TOC) is an improvement on the quantitative method called the Relative Operating Characteristic (ROC), both of which plot the association between a binary variable and a rank variable. TOC curves reveal the sizes of the four entries in the confusion matrix at each threshold, which make TOC curves more easily interpretable than ROC curves. The TOC has become popular, especially to assess the fit of simulation models to predict land change. However, the literature has shown variation in how authors apply and interpret the TOC, creating some misleading conclusions. Our manuscript lists best practices when applying and interpreting the TOC to help scientists learn from TOC curves. An example illustrates these practices by applying the TOC to measure the ability to predict the gain of crop in Western Bahia, Brazil. The application compares four ways to design the rank variable based on the distance to either pixels or patches of either the presence or change of crop. The results show that the gain of crop during the validation time interval is more strongly associated with the distance to patches rather than pixels of crop. The Discussion Section reveals that if authors show the TOC curves, then readers can interpret the results in ways that the authors might have missed. The Conclusion encourages scientists to follow best practices to learn the wealth of information that the TOC reveals.<\/jats:p>","DOI":"10.3390\/ijgi14040134","type":"journal-article","created":{"date-parts":[[2025,3,24]],"date-time":"2025-03-24T04:48:18Z","timestamp":1742791698000},"page":"134","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Best Practices for Applying and Interpreting the Total Operating Characteristic"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-7338-7012","authenticated-orcid":false,"given":"Tanner","family":"Honnef","sequence":"first","affiliation":[{"name":"Graduate School of Geography, Clark University, Worcester, MA 01610, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7287-5875","authenticated-orcid":false,"suffix":"Jr.","given":"Robert Gilmore","family":"Pontius","sequence":"additional","affiliation":[{"name":"Graduate School of Geography, Clark University, Worcester, MA 01610, USA"}]}],"member":"1968","published-online":{"date-parts":[[2025,3,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"570","DOI":"10.1080\/13658816.2013.862623","article-title":"The Total Operating Characteristic to Measure Diagnostic Ability for Multiple Thresholds","volume":"28","author":"Pontius","year":"2014","journal-title":"Int. J. Geogr. Inf. Sci."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1285","DOI":"10.1126\/science.3287615","article-title":"Measuring the Accuracy of Diagnostic Systems","volume":"240","author":"Swets","year":"1988","journal-title":"Science"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1038\/scientificamerican1000-82","article-title":"Better DECISIONS through SCIENCE","volume":"283","author":"Swets","year":"2000","journal-title":"Sci. Am."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Saito, T., and Rehmsmeier, M. (2015). The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0118432"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Pontius, R.G. (2022). Metrics That Make a Difference: How to Analyze Change and Error, Springer International Publishing. Advances in Geographic Information Science.","DOI":"10.1007\/978-3-030-70765-1"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1111\/j.1466-8238.2007.00358.x","article-title":"AUC: A Misleading Measure of the Performance of Predictive Distribution Models","volume":"17","author":"Lobo","year":"2008","journal-title":"Glob. Ecol. Biogeogr."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1016\/j.ecolmodel.2007.11.008","article-title":"Rethinking Receiver Operating Characteristic Analysis Applications in Ecological Niche Modeling","volume":"213","author":"Peterson","year":"2008","journal-title":"Ecol. Model."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"492","DOI":"10.1016\/j.patrec.2012.12.004","article-title":"When Is the Area under the Receiver Operating Characteristic Curve an Appropriate Measure of Classifier Performance?","volume":"34","author":"Hand","year":"2013","journal-title":"Pattern Recognit. Lett."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1581","DOI":"10.1007\/s10980-017-0520-0","article-title":"Multiple-Scale Prediction of Forest Loss Risk across Borneo","volume":"32","author":"Cushman","year":"2017","journal-title":"Landsc. Ecol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1301","DOI":"10.1080\/15481603.2022.2110197","article-title":"Simulation of Urban Land Expansion in China at 30 m Resolution through 2050 under Shared Socioeconomic Pathways","volume":"59","author":"Zhuang","year":"2022","journal-title":"GIScience Remote Sens."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"94","DOI":"10.1016\/j.landurbplan.2017.09.019","article-title":"A Future Land Use Simulation Model (FLUS) for Simulating Multiple Land Use Scenarios by Coupling Human and Natural Effects","volume":"168","author":"Liu","year":"2017","journal-title":"Landsc. Urban Plan."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Deng, Z., and Quan, B. (2022). Intensity Characteristics and Multi-Scenario Projection of Land Use and Land Cover Change in Hengyang, China. Int. J. Environ. Res. Public Health, 19.","DOI":"10.3390\/ijerph19148491"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"2919","DOI":"10.1007\/s10980-021-01272-0","article-title":"Validating Models of One-Way Land Change: An Example Case of Forest Insect Disturbance","volume":"36","author":"Harati","year":"2021","journal-title":"Landsc. Ecol"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2353252","DOI":"10.1080\/10106049.2024.2353252","article-title":"Three Novel Cost-Sensitive Machine Learning Models for Urban Growth Modelling","volume":"39","author":"Ahmadlou","year":"2024","journal-title":"Geocarto Int."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"112731","DOI":"10.1016\/j.jenvman.2021.112731","article-title":"Integration of Hard and Soft Supervised Machine Learning for Flood Susceptibility Mapping","volume":"291","author":"Andaryani","year":"2021","journal-title":"J. Environ. Manag."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1016\/j.envsoft.2018.03.016","article-title":"Fuzzy Definition of Rural Urban Interface: An Application Based on Land Use Change Scenarios in Portugal","volume":"104","author":"Amato","year":"2018","journal-title":"Environ. Model. Softw."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1080\/17538947.2021.2017035","article-title":"An Efficient Built-up Land Expansion Model Using a Modified U-Net","volume":"15","author":"Shojaei","year":"2022","journal-title":"Int. J. Digit. Earth"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"101595","DOI":"10.1016\/j.compenvurbsys.2021.101595","article-title":"Integrating a Forward Feature Selection Algorithm, Random Forest, and Cellular Automata to Extrapolate Urban Growth in the Tehran-Karaj Region of Iran","volume":"87","author":"Minaei","year":"2021","journal-title":"Comput. Environ. Urban Syst."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1481","DOI":"10.1007\/s10980-016-0341-6","article-title":"Quantifying Landscape Pattern and Ecosystem Service Value Changes in Four Rapidly Urbanizing Hill Stations of Southeast Asia","volume":"31","author":"Estoque","year":"2016","journal-title":"Landsc. Ecol"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"106395","DOI":"10.1016\/j.landusepol.2022.106395","article-title":"Harnessing the Indirect Effect of Urban Expansion for Mitigating Agriculture-Environment Trade-Offs in the Loess Plateau","volume":"122","author":"Wang","year":"2022","journal-title":"Land Use Policy"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1007\/s11069-014-1583-8","article-title":"A Dual Effect of Urban Expansion on Flood Risk in the Pearl River Delta (China) Revealed by Land-Use Scenarios and Direct Runoff Simulation","volume":"77","author":"Du","year":"2015","journal-title":"Nat Hazards"},{"key":"ref_22","first-page":"2497","article-title":"Four Fundamental Questions to Evaluate Land Change Models with an Illustration of a Cellular Automata\u2013Markov Model","volume":"113","author":"Viana","year":"2023","journal-title":"Ann. Am. Assoc. Geogr."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1007\/978-3-319-52522-8_8","article-title":"Lessons and Challenges in Land Change Modeling Derived from Synthesis of Cross-Case Comparisons","volume":"Volume 19","author":"Behnisch","year":"2018","journal-title":"Trends in Spatial Analysis and Modelling"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Pontius, R.G., Francis, T., and Millones, M. (2025). A Call to Interpret Disagreement Components during Classification Assessment. Int. J. Geogr. Inf. Sci., 1\u201318.","DOI":"10.1080\/13658816.2025.2469830"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"400","DOI":"10.1111\/j.1751-5823.2012.00183.x","article-title":"Assessing the Performance of Classification Methods","volume":"80","author":"Hand","year":"2012","journal-title":"Int Stat. Rev."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Pousa, R., Costa, M.H., Pimenta, F.M., Fontes, V.C., Brito, V.F.A.d., and Castro, M. (2019). Climate Change and Intense Irrigation Growth in Western Bahia, Brazil: The Urgent Need for Hydroclimatic Monitoring. Water, 11.","DOI":"10.3390\/w11050933"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Souza, C.M., Z. Shimbo, J., Rosa, M.R., Parente, L.L., A. Alencar, A., Rudorff, B.F.T., Hasenack, H., Matsumoto, M., G. Ferreira, L., and Souza-Filho, P.W.M. (2020). Reconstructing Three Decades of Land Use and Land Cover Changes in Brazilian Biomes with Landsat Archive and Earth Engine. Remote Sens., 12.","DOI":"10.3390\/rs12172735"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Liu, Z., and Pontius, R.G. (2021). The Total Operating Characteristic from Stratified Random Sampling with an Application to Flood Mapping. Remote Sens., 13.","DOI":"10.3390\/rs13193922"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Pontius, R.G. (2018). Criteria to Confirm Models that Simulate Deforestation and Carbon Disturbance. Land, 7.","DOI":"10.3390\/land7030105"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Bilintoh, T.M., Pontius, R.G., and Liu, Z. (2024). Analyzing the Losses and Gains of a Land Category: Insights from the Total Operating Characteristic. Land, 13.","DOI":"10.2139\/ssrn.4727386"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"2409484","DOI":"10.1080\/15481603.2024.2409484","article-title":"Methods to Compare Sites Concerning a Category\u2019s Change during Various Time Intervals","volume":"61","author":"Bilintoh","year":"2024","journal-title":"GISci. Remote Sens."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"924","DOI":"10.1080\/15481603.2020.1829376","article-title":"A Cellular Automata Approach of Urban Sprawl Simulation with Bayesian Spatially-Varying Transformation Rules","volume":"57","author":"Chen","year":"2020","journal-title":"GISci. Remote Sens."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"447","DOI":"10.3390\/ijgi4020447","article-title":"Simulating Urban Growth Using a Random Forest-Cellular Automata (RF-CA) Model","volume":"4","author":"Kamusoko","year":"2015","journal-title":"ISPRS Int. J. Geo-Inf."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1186\/s13717-021-00350-0","article-title":"Multi-Temporal Analysis of Past and Future Land Cover Change in the Highly Urbanized State of Selangor, Malaysia","volume":"11","author":"Azari","year":"2022","journal-title":"Ecol. Process"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"952","DOI":"10.1016\/j.ecolind.2018.05.036","article-title":"A Neural Network and Landscape Metrics to Propose a Flexible Urban Growth Boundary: A Case Study","volume":"93","author":"Chakraborti","year":"2018","journal-title":"Ecol. Indic."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Naghibi, F., and Delavar, M. (2016). Discovery of Transition Rules for Cellular Automata Using Artificial Bee Colony and Particle Swarm Optimization Algorithms in Urban Growth Modeling. ISPRS Int. J. Geo-Inf., 5.","DOI":"10.3390\/ijgi5120241"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Naghibi, F., Delavar, M., and Pijanowski, B. (2016). Urban Growth Modeling Using Cellular Automata with Multi-Temporal Remote Sensing Images Calibrated by the Artificial Bee Colony Optimization Algorithm. Sensors, 16.","DOI":"10.3390\/s16122122"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1007\/s10661-020-8144-2","article-title":"Modelling Agriculture, Forestry and Other Land Use (AFOLU) in Response to Climate Change Scenarios for the SAARC Nations","volume":"192","author":"Singh","year":"2020","journal-title":"Environ. Monit. Assess."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1016\/j.scs.2018.01.039","article-title":"Spatiotemporal Patterns of Urban Land Use Change in the Rapidly Growing City of Lusaka, Zambia: Implications for Sustainable Urban Development","volume":"39","author":"Simwanda","year":"2018","journal-title":"Sustain. Cities Soc."}],"container-title":["ISPRS International Journal of Geo-Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2220-9964\/14\/4\/134\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T16:58:43Z","timestamp":1760029123000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2220-9964\/14\/4\/134"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,23]]},"references-count":39,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2025,4]]}},"alternative-id":["ijgi14040134"],"URL":"https:\/\/doi.org\/10.3390\/ijgi14040134","relation":{},"ISSN":["2220-9964"],"issn-type":[{"value":"2220-9964","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,23]]}}}