{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,8]],"date-time":"2026-07-08T19:21:21Z","timestamp":1783538481023,"version":"3.55.0"},"reference-count":41,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2025,6,30]],"date-time":"2025-06-30T00:00:00Z","timestamp":1751241600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["41971349"],"award-info":[{"award-number":["41971349"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJGI"],"abstract":"<jats:p>Geospatial code generation is emerging as a key direction in the integration of artificial intelligence and geoscientific analysis. However, there remains a lack of standardized tools for automatic evaluation in this domain. To address this gap, we propose AutoGEEval, the first multimodal, unit-level automated evaluation framework for geospatial code generation tasks on the Google Earth Engine (GEE) platform powered by large language models (LLMs). Built upon the GEE Python API, AutoGEEval establishes a benchmark suite (AutoGEEval-Bench) comprising 1325 test cases that span 26 GEE data types. The framework integrates both question generation and answer verification components to enable an end-to-end automated evaluation pipeline\u2014from function invocation to execution validation. AutoGEEval supports multidimensional quantitative analysis of model outputs in terms of accuracy, resource consumption, execution efficiency, and error types. We evaluate 18 state-of-the-art LLMs\u2014including general-purpose, reasoning-augmented, code-centric, and geoscience-specialized models\u2014revealing their performance characteristics and potential optimization pathways in GEE code generation. This work provides a unified protocol and foundational resource for the development and assessment of geospatial code generation models, advancing the frontier of automated natural language to domain-specific code translation.<\/jats:p>","DOI":"10.3390\/ijgi14070256","type":"journal-article","created":{"date-parts":[[2025,6,30]],"date-time":"2025-06-30T10:03:48Z","timestamp":1751277828000},"page":"256","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["AutoGEEval: A Multimodal and Automated Evaluation Framework for Geospatial Code Generation on GEE with Large Language Models"],"prefix":"10.3390","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3971-0512","authenticated-orcid":false,"given":"Huayi","family":"Wu","sequence":"first","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"},{"name":"Collaborative Innovation Center of Geospatial Technology, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhangxiao","family":"Shen","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shuyang","family":"Hou","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jianyuan","family":"Liang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haoyue","family":"Jiao","sequence":"additional","affiliation":[{"name":"School of Resource and Environmental Sciences, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-4461-3421","authenticated-orcid":false,"given":"Yaxian","family":"Qing","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7386-7337","authenticated-orcid":false,"given":"Xiaopu","family":"Zhang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xu","family":"Li","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9467-9680","authenticated-orcid":false,"given":"Zhipeng","family":"Gui","sequence":"additional","affiliation":[{"name":"School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0865-3850","authenticated-orcid":false,"given":"Xuefeng","family":"Guan","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9022-6991","authenticated-orcid":false,"given":"Longgang","family":"Xiang","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,6,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1092","DOI":"10.1126\/science.abq1158","article-title":"Competition-level code generation with alphacode","volume":"378","author":"Li","year":"2022","journal-title":"Science"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1016\/j.compedu.2018.10.005","article-title":"Learning to code or coding to learn? A systematic review","volume":"128","author":"Popat","year":"2019","journal-title":"Comput. Educ."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1016\/0304-3975(94)90190-2","article-title":"An overview of transaction logic","volume":"133","author":"Bonner","year":"1994","journal-title":"Theor. Comput. Sci."},{"key":"ref_4","unstructured":"Jiang, J., Wang, F., Shen, J., Kim, S., and Kim, S. (2024). A survey on large language models for code generation. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Wang, J., and Chen, Y. (2023, January 18\u201319). A review on code generation with llms: Application and evaluation. Proceedings of the 2023 IEEE International Conference on Medical Artificial Intelligence (MedAI), Beijing, China.","DOI":"10.1109\/MedAI59581.2023.00044"},{"key":"ref_6","unstructured":"Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., and Li, Y.K. (2024). DeepSeek-Coder: When the Large Language Model Meets Programming\u2014The Rise of Code Intelligence. arXiv."},{"key":"ref_7","unstructured":"Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., and Lu, K. (2024). Qwen2. 5-coder technical report. arXiv."},{"key":"ref_8","unstructured":"Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., Adi, Y., Liu, J., Sauvestre, R., and Remez, T. (2023). Code llama: Open foundation models for code. arXiv."},{"key":"ref_9","unstructured":"Rahman, M.M., and Kundu, A. (2024). Code Hallucination. arXiv."},{"key":"ref_10","unstructured":"Li, D., and Murr, L. (2024). HumanEval on Latest GPT Models\u20142024. arXiv."},{"key":"ref_11","unstructured":"Yu, Z., Zhao, Y., Cohan, A., and Zhang, X.-P. (2024). HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation. arXiv."},{"key":"ref_12","unstructured":"Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. (2024). Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"111741","DOI":"10.1016\/j.jss.2023.111741","article-title":"Out of the bleu: How should we assess quality of the code generation models?","volume":"203","author":"Evtikhiev","year":"2023","journal-title":"J. Syst. Softw."},{"key":"ref_14","unstructured":"Liu, J., Xie, S., Wang, J., Wei, Y., Ding, Y., and Zhang, L. (2024). Evaluating language models for efficient code generation. arXiv."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Zhou, S., Alon, U., Agarwal, S., and Neubig, G. (2023). Codebertscore: Evaluating code generation with pretrained models of code. arXiv.","DOI":"10.18653\/v1\/2023.emnlp-main.859"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Capolupo, A., Monterisi, C., Caporusso, G., and Tarantino, E. (2020, January 1\u20134). Extracting land cover data using GEE: A review of the classification indices. Proceedings of the Computational Science and Its Applications\u2014ICCSA 2020, Cagliari, Italy.","DOI":"10.1007\/978-3-030-58811-3_56"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.isprsjprs.2020.04.001","article-title":"Google Earth Engine for geo-big data applications: A meta-analysis and systematic review","volume":"164","author":"Tamiminia","year":"2020","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1111\/j.1467-9671.2004.00193.x","article-title":"Tangible User Interfaces (TUIs): A novel paradigm for GIS","volume":"8","author":"Ratti","year":"2004","journal-title":"Trans. GIS"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhao, Q., Yu, L., Li, X., Peng, D., Zhang, Y., and Gong, P. (2021). Progress and trends in the application of Google Earth and Google Earth Engine. Remote Sens., 13.","DOI":"10.3390\/rs13183778"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Mutanga, O., and Kumar, L. (2019). Google earth engine applications. Remote Sens., 11.","DOI":"10.3390\/rs11050591"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Hou, S., Shen, Z., Zhao, A., Liang, J., Gui, Z., Guan, X., Li, R., and Wu, H. (2025). GeoCode-GPT: A large language model for geospatial code generation. Int. J. Appl. Earth Obs. Geoinf., 104456.","DOI":"10.1016\/j.jag.2025.104456"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Hou, S., Liang, J., Zhao, A., and Wu, H. (2025). GEE-OPs: An operator knowledge base for geospatial code generation on the Google Earth Engine platform powered by large language models. Geo-Spat. Inf. Sci., 1\u201322.","DOI":"10.1080\/10095020.2025.2505556"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Yang, L., Driscol, J., Sarigai, S., Wu, Q., Chen, H., and Lippitt, C.D. (2022). Google Earth Engine and artificial intelligence (AI): A comprehensive review. Remote Sens., 14.","DOI":"10.3390\/rs14143253"},{"key":"ref_24","unstructured":"Hou, S., Shen, Z., Liang, J., Zhao, A., Gui, Z., Li, R., and Wu, H. (2024). Can large language models generate geospatial code?. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Gramacki, P., Martins, B., and Szyma\u0144ski, P. (2024). Evaluation of Code LLMs on Geospatial Code Generation. arXiv.","DOI":"10.1145\/3687123.3698286"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Hou, S., Jiao, H., Shen, Z., Liang, J., Zhao, A., Zhang, X., Wang, J., and Wu, H. (2024). Chain-of-Programming (CoP): Empowering Large Language Models for Geospatial Code Generation. arXiv.","DOI":"10.1080\/17538947.2025.2509812"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"113624","DOI":"10.1016\/j.knosys.2025.113624","article-title":"Geo-FuB: A method for constructing an Operator-Function knowledge base for geospatial code generation with large language models","volume":"319","author":"Shuyang","year":"2025","journal-title":"Knowl.-Based Syst."},{"key":"ref_28","unstructured":"Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A.J., Welihinda, A., Hayes, A., and Radford, A. (2024). Gpt-4o system card. arXiv."},{"key":"ref_29","unstructured":"Menick, J., Lu, K., Zhao, S., Wallace, E., Ren, H., Hu, H., Stathas, N., and Such, F.P. (2024). GPT-4o Mini: Advancing Cost-efficient Intelligence, Open AI."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Anderson, I. (2025). Comparative Analysis Between Industrial Design Methodologies Versus the Scientific Method: AI: Claude 3.7 Sonnet. Preprints.","DOI":"10.20944\/preprints202505.1975.v1"},{"key":"ref_31","unstructured":"Team, G.R., Abeyruwan, S., Ainslie, J., Alayrac, J.-B., Arenas, M.G., Armstrong, T., Balakrishna, A., Baruch, R., Bauza, M., and Blokzijl, M. (2025). Gemini robotics: Bringing ai into the physical world. arXiv."},{"key":"ref_32","unstructured":"Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., and Ruan, C. (2024). Deepseek-v3 technical report. arXiv."},{"key":"ref_33","unstructured":"Yang, A., Yu, B., Li, C., Liu, D., Huang, F., Huang, H., Jiang, J., Tu, J., Zhang, J., and Zhou, J. (2025). Qwen2. 5-1M Technical Report. arXiv."},{"key":"ref_34","unstructured":"Arrieta, A., Ugarte, M., Valle, P., Parejo, J.A., and Segura, S. (2025). o3-mini vs DeepSeek-R1: Which One is Safer?. arXiv."},{"key":"ref_35","unstructured":"Zheng, C., Zhang, Z., Zhang, B., Lin, R., Lu, K., Yu, B., Liu, D., Zhou, J., and Lin, J. (2024). Processbench: Identifying process errors in mathematical reasoning. arXiv."},{"key":"ref_36","unstructured":"Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., and Bi, X. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv."},{"key":"ref_37","unstructured":"Zhu, Q., Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., and Ma, S. (2024). Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv."},{"key":"ref_38","unstructured":"Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., and Brockman, G. (2021). Evaluating large language models trained on code. arXiv."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1109\/TCI.2016.2644865","article-title":"Loss functions for image restoration with neural networks","volume":"3","author":"Zhao","year":"2016","journal-title":"IEEE Trans. Comput. Imaging"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: From error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1680","DOI":"10.1109\/TIP.2005.864165","article-title":"Quality-aware images","volume":"15","author":"Wang","year":"2006","journal-title":"IEEE Trans. Image Process."}],"container-title":["ISPRS International Journal of Geo-Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2220-9964\/14\/7\/256\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T18:02:10Z","timestamp":1760032930000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2220-9964\/14\/7\/256"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,30]]},"references-count":41,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2025,7]]}},"alternative-id":["ijgi14070256"],"URL":"https:\/\/doi.org\/10.3390\/ijgi14070256","relation":{},"ISSN":["2220-9964"],"issn-type":[{"value":"2220-9964","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,30]]}}}