{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,27]],"date-time":"2026-03-27T14:18:53Z","timestamp":1774621133118,"version":"3.50.1"},"reference-count":40,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2025,9,28]],"date-time":"2025-09-28T00:00:00Z","timestamp":1759017600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["41971349"],"award-info":[{"award-number":["41971349"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.mdpi.com"],"crossmark-restriction":true},"short-container-title":["IJGI"],"abstract":"<jats:p>With the widespread adoption of large language models (LLMs) in code generation tasks, geospatial code generation has emerged as a critical frontier in the integration of artificial intelligence and geoscientific analysis. This growing trend underscores the urgent need for systematic evaluation methodologies to assess the generation capabilities of LLMs in geospatial contexts. In particular, geospatial computation and visualization tasks in the JavaScript environment rely heavily on the orchestration of diverse frontend libraries and ecosystems, posing elevated demands on a model\u2019s semantic comprehension and code synthesis capabilities. To address this challenge, we propose GeoJSEval\u2014the first multimodal, function-level automatic evaluation framework for LLMs in JavaScript-based geospatial code generation tasks. The framework comprises three core components: a standardized test suite (GeoJSEval-Bench), a code submission engine, and an evaluation module. It includes 432 function-level tasks and 2071 structured test cases, spanning five widely used JavaScript geospatial libraries that support spatial analysis and visualization functions, as well as 25 mainstream geospatial data types. GeoJSEval enables multidimensional quantitative evaluation across metrics such as accuracy, output stability, resource consumption, execution efficiency, and error type distribution. Moreover, it integrates boundary testing mechanisms to enhance robustness and evaluation coverage. We conduct a comprehensive assessment of 20 state-of-the-art LLMs using GeoJSEval, uncovering significant performance disparities and bottlenecks in spatial semantic understanding, code reliability, and function invocation accuracy. GeoJSEval offers a foundational methodology, evaluation resource, and practical toolkit for the standardized assessment and optimization of geospatial code generation models, with strong extensibility and promising applicability in real-world scenarios. This manuscript represents the peer-reviewed version of our earlier preprint previously made available on arXiv.<\/jats:p>","DOI":"10.3390\/ijgi14100382","type":"journal-article","created":{"date-parts":[[2025,9,29]],"date-time":"2025-09-29T12:59:46Z","timestamp":1759150786000},"page":"382","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["GeoJSEval: An Automated Evaluation Framework for Large Language Models on JavaScript-Based Geospatial Computation and Visualization Code Generation"],"prefix":"10.3390","volume":"14","author":[{"given":"Guanyu","family":"Chen","sequence":"first","affiliation":[{"name":"School of Resource and Environmental Sciences, Wuhan University, Wuhan 430079, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5699-3919","authenticated-orcid":false,"given":"Haoyue","family":"Jiao","sequence":"additional","affiliation":[{"name":"School of Resource and Environmental Sciences, Wuhan University, Wuhan 430079, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6984-9959","authenticated-orcid":false,"given":"Shuyang","family":"Hou","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0483-5502","authenticated-orcid":false,"given":"Ziqi","family":"Liu","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}]},{"given":"Lutong","family":"Xie","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}]},{"given":"Shaowen","family":"Wu","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3971-0512","authenticated-orcid":false,"given":"Huayi","family":"Wu","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0865-3850","authenticated-orcid":false,"given":"Xuefeng","family":"Guan","sequence":"additional","affiliation":[{"name":"State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430079, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9467-9680","authenticated-orcid":false,"given":"Zhipeng","family":"Gui","sequence":"additional","affiliation":[{"name":"School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430079, China"}]}],"member":"1968","published-online":{"date-parts":[[2025,9,28]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"552","DOI":"10.1017\/S095679681300018X","article-title":"Idris, a general-purpose dependently typed programming language: Design and implementation","volume":"23","author":"Brady","year":"2013","journal-title":"J. Funct. Program."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1080\/20964471.2019.1661662","article-title":"Geospatial data ontology: The semantic foundation of geospatial data integration and sharing","volume":"3","author":"Sun","year":"2019","journal-title":"Big Earth Data"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/j.isprsjprs.2015.10.012","article-title":"Geospatial big data handling theory and methods: A review and research challenges","volume":"115","author":"Li","year":"2016","journal-title":"ISPRS J. Photogramm. Remote Sens."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/0198-9715(92)90050-2","article-title":"The integration of spatial analysis and GIS","volume":"16","author":"Ding","year":"1992","journal-title":"Comput. Environ. Urban Syst."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Breunig, M., Bradley, P.E., Jahn, M., Kuper, P., Mazroob, N., R\u00f6sch, N., Al-Doori, M., Stefanakis, E., and Jadidi, M. (2020). Geospatial data management research: Progress and future directions. ISPRS Int. J. Geo-Inf., 9.","DOI":"10.3390\/ijgi9020095"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1080\/13658816.2011.574292","article-title":"On-demand base maps on the web generalized according to user profiles","volume":"26","author":"Foerster","year":"2012","journal-title":"Int. J. Geogr. Inf. Sci."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1016\/j.cageo.2017.11.007","article-title":"3D geospatial visualizations: Animation and motion effects on spatial objects","volume":"111","author":"Evangelidis","year":"2018","journal-title":"Comput. Geosci."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"637","DOI":"10.1016\/j.cageo.2004.03.010","article-title":"Implementation of the analytical hierarchy process with VBA in ArcGIS","volume":"30","author":"Marinoni","year":"2004","journal-title":"Comput. Geosci."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2219","DOI":"10.3390\/ijgi4042219","article-title":"Processing: A python framework for the seamless integration of geoprocessing tools in QGIS","volume":"4","author":"Graser","year":"2015","journal-title":"ISPRS Int. J. Geo-Inf."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Zhao, Q., Yu, L., Li, X., Peng, D., Zhang, Y., and Gong, P. (2021). Progress and trends in the application of Google Earth and Google Earth Engine. Remote Sens., 13.","DOI":"10.3390\/rs13183778"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1016\/S0166-3615(01)00072-0","article-title":"A review of web-based product data management systems","volume":"44","author":"Liu","year":"2001","journal-title":"Comput. Ind."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"915","DOI":"10.1016\/j.jss.2008.12.050","article-title":"Lightweight query-based analysis of workflow process dependencies","volume":"82","author":"Dai","year":"2009","journal-title":"J. Syst. Softw."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1517","DOI":"10.1016\/j.cageo.2008.10.004","article-title":"A WebGIS-based system for rainfall-runoff prediction and real-time water resources assessment for Beijing","volume":"35","author":"Jia","year":"2009","journal-title":"Comput. Geosci."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1371","DOI":"10.1007\/s10664-015-9389-1","article-title":"What Java developers know about compatibility, and why this matters","volume":"21","author":"Dietrich","year":"2016","journal-title":"Empir. Softw. Eng."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Hou, D., Chen, J., and Wu, H. (2016). Discovering land cover web map services from the deep web with javascript invocation rules. ISPRS Int. J. Geo-Inf., 5.","DOI":"10.3390\/ijgi5070105"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"872","DOI":"10.1109\/JAS.2025.125498","article-title":"Exploring DeepSeek: A Survey on Advances, Applications, Challenges and Future Directions","volume":"12","author":"Deng","year":"2025","journal-title":"IEEE\/CAA J. Autom. Sin."},{"key":"ref_17","first-page":"104456","article-title":"GeoCode-GPT: A large language model for geospatial code generation","volume":"138","author":"Hou","year":"2025","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"101764","DOI":"10.1016\/j.cose.2020.101764","article-title":"Detecting malicious JavaScript code based on semantic analysis","volume":"93","author":"Fang","year":"2020","journal-title":"Comput. Secur."},{"key":"ref_19","first-page":"1","article-title":"Deep learning for code intelligence: Survey, benchmark and toolkit","volume":"56","author":"Wan","year":"2024","journal-title":"ACM Comput. Surv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Quarati, A., De Martino, M., and Rosim, S. (2021). Geospatial open data usage and metadata quality. ISPRS Int. J. Geo-Inf., 10.","DOI":"10.3390\/ijgi10010030"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wu, H., Shen, Z., Hou, S., Liang, J., Jiao, H., Qing, Y., Zhang, X., Li, X., Gui, Z., and Guan, X. (2025). AutoGEEval: A Multimodal and Automated Evaluation Framework for Geospatial Code Generation on GEE with Large Language Models. ISPRS Int. J. Geo-Inf., 14.","DOI":"10.3390\/ijgi14070256"},{"key":"ref_22","unstructured":"Hou, S., Shen, Z., Wu, H., Jiao, H., Liu, Z., Xie, L., Liu, C., Liang, J., Qing, Y., and Zhang, X. (2025). AutoGEEval++: A Multi-Level and Multi-Geospatial-Modality Automated Evaluation Framework for Large Language Models in Geospatial Code Generation on Google Earth Engine. arXiv."},{"key":"ref_23","unstructured":"Chen, G., Jiao, H., Hou, S., Liu, Z., Xie, L., Wu, S., Wu, H., Guan, X., and Gui, Z. (2025). GeoJSEval: An automated evaluation framework for large language models on JavaScript-based geospatial computation and visualization code generation. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"4668","DOI":"10.1080\/17538947.2023.2278895","article-title":"Autonomous GIS: The next-generation AI-powered GIS","volume":"16","author":"Li","year":"2023","journal-title":"Int. J. Digit. Earth"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Pakdil, M.E., and \u00c7elik, R.N. (2022). Serverless geospatial data processing workflow system design. ISPRS Int. J. Geo-Inf., 11.","DOI":"10.3390\/ijgi11010020"},{"key":"ref_26","first-page":"104477","article-title":"FAIR principles in workflows: A GIScience workflow management system for reproducible and replicable studies","volume":"138","author":"Hu","year":"2025","journal-title":"Int. J. Appl. Earth Obs. Geoinf."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1080\/19475683.2013.782467","article-title":"A visualization-enhanced graphical user interface for geospatial resource discovery","volume":"19","author":"Gui","year":"2013","journal-title":"Ann. GIS"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1109\/2.366157","article-title":"Scaling up visual programming languages","volume":"28","author":"Burnett","year":"1995","journal-title":"Computer"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1007\/s10515-024-00464-7","article-title":"A survey on robustness attacks for deep code models","volume":"31","author":"Qu","year":"2024","journal-title":"Autom. Softw. Eng."},{"key":"ref_30","first-page":"1","article-title":"In-ide code generation from natural language: Promise and challenges","volume":"31","author":"Xu","year":"2022","journal-title":"ACM Trans. Softw. Eng. Methodol. (TOSEM)"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"187","DOI":"10.1007\/s11219-006-9216-4","article-title":"Theories, tools and research methods in program comprehension: Past, present and future","volume":"14","author":"Storey","year":"2006","journal-title":"Softw. Qual. J."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2509812","DOI":"10.1080\/17538947.2025.2509812","article-title":"Chain-of-programming (CoP): Empowering large language models for geospatial code generation task","volume":"18","author":"Hou","year":"2025","journal-title":"Int. J. Digit. Earth"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"113624","DOI":"10.1016\/j.knosys.2025.113624","article-title":"Geo-FuB: A method for constructing an Operator-Function knowledge base for geospatial code generation with large language models","volume":"319","author":"Hou","year":"2025","journal-title":"Knowl. Based Syst."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Hou, S., Liang, J., Zhao, A., and Wu, H. (2025). GEE-OPs: An operator knowledge base for geospatial code generation on the Google Earth Engine platform powered by large language models. Geo-Spat. Inf. Sci., 1\u201322.","DOI":"10.1080\/10095020.2025.2505556"},{"key":"ref_35","first-page":"5299","article-title":"A review of intelligent self-driving vehicle software research","volume":"13","author":"Gwak","year":"2019","journal-title":"KSII Trans. Internet Inf. Syst. (TIIS)"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Hochmair, H.H., Juh\u00e1sz, L., and Li, H. (2025). Advancing AI-Driven Geospatial Analysis and Data Generation: Methods, Applications and Future Directions. ISPRS Int. J. Geo-Inf., 14.","DOI":"10.3390\/ijgi14020056"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1016\/j.compenvurbsys.2017.05.003","article-title":"A review of the emergent ecosystem of collaborative geospatial tools for addressing environmental challenges","volume":"65","author":"Palomino","year":"2017","journal-title":"Comput. Environ. Urban Syst."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1006\/inco.2000.9999","article-title":"Compositionality in dataflow synchronous languages: Specification and distributed code generation","volume":"163","author":"Benveniste","year":"2000","journal-title":"Inf. Comput."},{"key":"ref_39","first-page":"3411","article-title":"Invalidator: Automated patch correctness assessment via semantic and syntactic reasoning","volume":"49","author":"Luong","year":"2023","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Wei, Q., Yao, Z., Cui, Y., Wei, B., Jin, Z., and Xu, X. (2024). Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysis. J. Biomed. Inform., 151.","DOI":"10.1016\/j.jbi.2024.104620"}],"container-title":["ISPRS International Journal of Geo-Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2220-9964\/14\/10\/382\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,29]],"date-time":"2025-09-29T13:27:25Z","timestamp":1759152445000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2220-9964\/14\/10\/382"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,28]]},"references-count":40,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2025,10]]}},"alternative-id":["ijgi14100382"],"URL":"https:\/\/doi.org\/10.3390\/ijgi14100382","relation":{},"ISSN":["2220-9964"],"issn-type":[{"value":"2220-9964","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,28]]}}}