{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T04:44:12Z","timestamp":1784090652872,"version":"3.55.0"},"reference-count":20,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2013,9,10]],"date-time":"2013-09-10T00:00:00Z","timestamp":1378771200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/3.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJGI"],"abstract":"<jats:p>The Receiver Operating Characteristic (ROC) is widely used for assessing the performance of classification algorithms. In GIScience, ROC has been applied to assess models aimed at predicting events, such as land use\/cover change (LUCC), species distribution and disease risk. However, GIS software packages offer few statistical tests and guidance tools for ROC analysis and interpretation. This paper presents a suite of GIS tools designed to facilitate ROC curve analysis for GIS users by applying proper statistical tests and analysis procedures. The tools are freely available as models and submodels of Dinamica EGO freeware. The tools give the ROC curve, the area under the curve (AUC), partial AUC, lower and upper AUCs, the confidence interval of AUC, the density of event in probability bins and tests to evaluate the difference between the AUCs of two models. We present first the procedures and statistical tests implemented in Dinamica EGO, then the application of the tools to assess LUCC and species distribution models. Finally, we interpret and discuss the ROC-related statistics resulting from various case studies.<\/jats:p>","DOI":"10.3390\/ijgi2030869","type":"journal-article","created":{"date-parts":[[2013,9,11]],"date-time":"2013-09-11T03:51:10Z","timestamp":1378871470000},"page":"869-887","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":132,"title":["A Suite of Tools for ROC Analysis of Spatial Models"],"prefix":"10.3390","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6138-9879","authenticated-orcid":false,"given":"Jean-Fran\u00e7ois","family":"Mas","sequence":"first","affiliation":[{"name":"Centro de Investigaciones en Geograf\u00eda Ambiental, Universidad Nacional Aut\u00f3noma de M\u00e9xico, Antigua Carretera a P\u00e1tzcuaro 8701, Col. Ex-Hacienda de San Jos\u00e9 de La Huerta, Morelia C.P. 58190, MIC, Mexico"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Britaldo","family":"Soares Filho","sequence":"additional","affiliation":[{"name":"Centro de Sensoriamento Remoto, Universidade Federal de Minas Gerais, Av. Ant\u00f4nio Carlos 6627, Belo Horizonte-MG, 31270-901, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7287-5875","authenticated-orcid":false,"given":"Robert","family":"Pontius","sequence":"additional","affiliation":[{"name":"Graduate School of Geography, Clark University, 950 Main Street, Worcester, MA 01610-1477, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michelle","family":"Farf\u00e1n Guti\u00e9rrez","sequence":"additional","affiliation":[{"name":"Centro de Investigaciones en Geograf\u00eda Ambiental, Universidad Nacional Aut\u00f3noma de M\u00e9xico, Antigua Carretera a P\u00e1tzcuaro 8701, Col. Ex-Hacienda de San Jos\u00e9 de La Huerta, Morelia C.P. 58190, MIC, Mexico"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hermann","family":"Rodrigues","sequence":"additional","affiliation":[{"name":"Centro de Sensoriamento Remoto, Universidade Federal de Minas Gerais, Av. Ant\u00f4nio Carlos 6627, Belo Horizonte-MG, 31270-901, Brazil"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2013,9,10]]},"reference":[{"key":"ref_1","unstructured":"Swets, J.A. (1996). Signal Detection Theory and ROC Analysis in Psychology and Diagnostics, Lawrence Erlbaum Associates. [1st ed.]."},{"key":"ref_2","unstructured":"Christodoulakis, G., and Satchell, S. (2008). The Analytics of Risk Model Validation, Elsevier. [1st ed.]."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"198","DOI":"10.1093\/bib\/bbm064","article-title":"ROC analysis: Applications to the classification of biological sequences and 3D structures","volume":"9","author":"Sonego","year":"2008","journal-title":"Brief. Bioinform."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.agee.2012.07.014","article-title":"A geospatial modeling framework for assessing biofuels-related land-use and land-cover change","volume":"161","author":"Li","year":"2012","journal-title":"Agr. Ecosyst. Environ."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1111\/1467-9671.00159","article-title":"Using the relative operating characteristic to quantify certainty in prediction of location of land cover change in India","volume":"7","author":"Pontius","year":"2003","journal-title":"Trans. GIS"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Pontius, R.G., and Parmentier, B. (2013). Recommendations for using the relative operating characteristic (ROC). Landsc. Ecol., submitted for publication.","DOI":"10.1007\/s10980-013-9984-8"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"861","DOI":"10.1016\/j.patrec.2005.10.010","article-title":"An introduction to ROC analysis","volume":"27","author":"Fawcett","year":"2006","journal-title":"Pattern. Recogni. Lett."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.C., and M\u00fcller, M. (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinforma., 12.","DOI":"10.1186\/1471-2105-12-77"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"80","DOI":"10.1016\/j.envsoft.2013.01.010","article-title":"A hybrid analytical-heuristic method for calibrating land-use change models","volume":"43","author":"Rodrigues","year":"2013","journal-title":"Environ. Model. Soft."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1016\/j.ecolmodel.2007.11.008","article-title":"Rethinking receiver operating characteristic analysis applications in ecological Niche modelling","volume":"213","author":"Peterson","year":"2008","journal-title":"Ecol. Model."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1177\/0272989X8900900307","article-title":"Analyzing a portion of the ROC curve","volume":"9","author":"McClish","year":"1989","journal-title":"Med. Decis. Making"},{"key":"ref_12","unstructured":"Santini, S. Available online:http:\/\/arantxa.ii.uam.es\/~ssantini\/writing\/notes\/s667_binomial.pdf."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1016\/S0167-8809(01)00187-6","article-title":"Land-cover change model validation by an ROC method for the Ipswich Watershed, Massachusetts, USA","volume":"85","author":"Pontius","year":"2001","journal-title":"Agr. Ecosyst. Environ."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1111\/j.1466-8238.2007.00358.x","article-title":"AUC: A Misleading measure of the performance of predictive distribution models","volume":"17","author":"Lobo","year":"2008","journal-title":"Glob. Ecol. Biogeogr."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1016\/j.ecolmodel.2005.03.026","article-title":"Maximum entropy modeling of species geographic distributions","volume":"190","author":"Phillips","year":"2006","journal-title":"Ecol. Model."},{"key":"ref_16","unstructured":"Mas, J.F., Farf\u00e1n, M., Ghilen, C., Lima, T., and Soares Filho, B. (2013, January 21\u201325). Una Comparaci\u00f3n de dos Enfoques de Modelaci\u00f3n de Nicho Ecol\u00f3gico. Proceedings of Memorias de la XX Reuni\u00f3n SELPER, San Luis Potos\u00ed, M\u00e9xico."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"745","DOI":"10.1111\/j.1529-8817.2003.00769.x","article-title":"Simulating the response of land-cover changes to road paving and governance along a major Amazon highway: The Santar\u00e9m-Cuiab\u00e1 Corridor","volume":"10","author":"Alencar","year":"2004","journal-title":"Glob. Change Biol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1016\/j.ecocom.2012.01.004","article-title":"Assessing simulated land use\/cover maps using similarity and fragmentation indices","volume":"11","author":"Mas","year":"2012","journal-title":"Ecol. Complex"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"325","DOI":"10.1007\/s10708-004-5049-5","article-title":"Calibration and validation of a model of forest disturbance in the western Ghats, India 1920\u20131990","volume":"61","author":"Pontius","year":"2004","journal-title":"GeoJournal"},{"key":"ref_20","unstructured":"(2013). R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing."}],"container-title":["ISPRS International Journal of Geo-Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2220-9964\/2\/3\/869\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T21:49:12Z","timestamp":1760219352000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2220-9964\/2\/3\/869"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,9,10]]},"references-count":20,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2013,9]]}},"alternative-id":["ijgi2030869"],"URL":"https:\/\/doi.org\/10.3390\/ijgi2030869","relation":{},"ISSN":["2220-9964"],"issn-type":[{"value":"2220-9964","type":"electronic"}],"subject":[],"published":{"date-parts":[[2013,9,10]]}}}