{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:20:14Z","timestamp":1760242814311,"version":"build-2065373602"},"reference-count":45,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2016,8,5]],"date-time":"2016-08-05T00:00:00Z","timestamp":1470355200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJGI"],"abstract":"<jats:p>With the advance of the World-Wide Web (WWW) technology, people can easily share content on the Web, including geospatial data and web services. Thus, the \u201cbig geospatial data management\u201d issues start attracting attention. Among the big geospatial data issues, this research focuses on discovering distributed geospatial resources. As resources are scattered on the WWW, users cannot find resources of their interests efficiently. While the WWW has Web search engines addressing web resource discovery issues, we envision that the geospatial Web (i.e., GeoWeb) also requires GeoWeb search engines. To realize a GeoWeb search engine, one of the first steps is to proactively discover GeoWeb resources on the WWW. Hence, in this study, we propose the GeoWeb Crawler, an extensible Web crawling framework that can find various types of GeoWeb resources, such as Open Geospatial Consortium (OGC) web services, Keyhole Markup Language (KML) and Environmental Systems Research Institute, Inc (ESRI) Shapefiles. In addition, we apply the distributed computing concept to promote the performance of the GeoWeb Crawler. The result shows that for 10 targeted resources types, the GeoWeb Crawler discovered 7351 geospatial services and 194,003 datasets. As a result, the proposed GeoWeb Crawler framework is proven to be extensible and scalable to provide a comprehensive index of GeoWeb.<\/jats:p>","DOI":"10.3390\/ijgi5080136","type":"journal-article","created":{"date-parts":[[2016,8,5]],"date-time":"2016-08-05T10:10:24Z","timestamp":1470391824000},"page":"136","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["GeoWeb Crawler: An Extensible and Scalable Web Crawling Framework for Discovering Geospatial Web Resources"],"prefix":"10.3390","volume":"5","author":[{"given":"Chih-Yuan","family":"Huang","sequence":"first","affiliation":[{"name":"Center for Space and Remote Sensing Research, National Central University, Taoyuan 320, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hao","family":"Chang","sequence":"additional","affiliation":[{"name":"Department of Civil Engineering, National Central University, Taoyuan 320, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2016,8,5]]},"reference":[{"key":"ref_1","first-page":"403","article-title":"Spatial decision support systems","volume":"1","author":"Densham","year":"1991","journal-title":"Geogr. Inf. Syst. Princ. Appl."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1016\/0167-9236(94)00018-N","article-title":"Spatial decision support systems: An overview of technology and a test of efficacy","volume":"14","author":"Crossland","year":"1995","journal-title":"Decis. Support Syst."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Burrough, P.A. (1986). Principles of geographical information systems for land resources assessment. Geocar Int., 1.","DOI":"10.1080\/10106048609354060"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1270","DOI":"10.1016\/j.envsoft.2006.08.003","article-title":"A web-based gis decision support system for managing and planning usda\u2019s conservation reserve program (crp)","volume":"22","author":"Rao","year":"2007","journal-title":"Environ. Model. Softw."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"2011","DOI":"10.1111\/j.1749-8198.2008.00167.x","article-title":"Web mapping 2.0: The neogeography of the geoweb","volume":"2","author":"Haklay","year":"2008","journal-title":"Geogr. Compass"},{"key":"ref_6","unstructured":"Lake, R., and Farley, J. (2009). The Geospatial Web, Springer."},{"key":"ref_7","unstructured":"Taylor, B. The World is Your Javascript-Enabled Oyster. Official Google Blog. Available online: https:\/\/googleblog.blogspot.com\/2005\/06\/world-is-your-javascript-enabled_29.html."},{"key":"ref_8","unstructured":"Kuhn, W. Introduction to Spatial Data Infrastructures. Presentation held on March 2005. Available online: https:\/\/collaboration.worldbank.org\/docs\/DOC-3031."},{"key":"ref_9","unstructured":"Laney, D. (2001). 3D Data Management: Controlling Data Volume, Velocity and Variety, META Group."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"13402","DOI":"10.3390\/s131013402","article-title":"Geocens: A geospatial cyberinfrastructure for the world-wide sensor web","volume":"13","author":"Liang","year":"2013","journal-title":"Sensors"},{"key":"ref_11","unstructured":"Botts, M., Percivall, G., Reed, C., and Davidson, J. (2006). Geosensor Networks, Springer."},{"key":"ref_12","unstructured":"Liang, S., Chen, S., Huang, C., Li, R., Chang, Y., Badger, J., and Rezel, R. (2010, January 14). Capturing the long tail of sensor web. Proceedings of International Workshop on Role of Volunteered Geographic Information in Advancing Science, Zurich, Switzerland."},{"key":"ref_13","unstructured":"Chris, A. (2006). The Long Tail: Why the Future of Business is Selling Less of More, Hyperion."},{"key":"ref_14","unstructured":"Morais, C.D. Where is the Phrase \u201c80% of Data is Geographic\u201d from?. Available online: https:\/\/www.gislounge.com\/80-percent-data-is-geographic\/."},{"key":"ref_15","unstructured":"Sullivan, D. Major Search Engines and Directories. Available online: http:\/\/www.leepublicschools.net\/Technology\/Search-Engines_Directories.pdf."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Hicks, C., Scheffer, M., Ngu, A.H., and Sheng, Q.Z. (2012, January 8\u201310). Discovery and cataloging of deep web sources. Proceedings of the 2012 IEEE 13th International Conference on Information Reuse and Integration (IRI), Las Vegas, NV, USA.","DOI":"10.1109\/IRI.2012.6303014"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1145\/1327452.1327492","article-title":"Mapreduce: Simplified data processing on large clusters","volume":"51","author":"Dean","year":"2008","journal-title":"Commun. ACM"},{"key":"ref_18","unstructured":"Mirtaheri, S.M., Din\u00e7kt\u00fcrk, M.E., Hooshmand, S., Bochmann, G.V., Jourdan, G.-V., and Onut, I.V. (2013, January 18). A brief history of web crawlers. Proceedings of the 2013 Conference of the Center for Advanced Studies on Collaborative Research, Riverton, NJ, USA."},{"key":"ref_19","unstructured":"Lopez-Pellicer, F.J., Renter\u00eda-Agualimpia, W., Nogueras-Iso, J., Zarazaga-Soria, F.J., and Muro-Medrano, P.R. (2012). Bridging the Geographic Information Sciences, Springer."},{"key":"ref_20","unstructured":"Bai, Y., Yang, C., Guo, L., and Cheng, Q. (2003, January 21\u201325). Opengis wms-based prototype system of spatial information search engine. Proceedings of the 2003 IEEE International Geoscience and Remote Sensing Symposium, 2003. IGARSS'03, Toulouse, France."},{"key":"ref_21","unstructured":"Li, W. (2014, January 3\u20137). Polarhub: A global hub for polar data discovery. Proceedings of the AGU Fall Meeting Abstracts, San Francisco, CA, USA."},{"key":"ref_22","first-page":"1","article-title":"A geospatial search engine for discovering multi-format geospatial data across the web","volume":"9","author":"Bone","year":"2014","journal-title":"Int. J. Digital Earth"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1109\/MIC.2006.96","article-title":"Enhancing the us navy\u2019s gidb portal with web services","volume":"10","author":"Sample","year":"2006","journal-title":"IEEE Internet Computing"},{"key":"ref_24","unstructured":"Chen, N., Liping Di B, G.Y.B., and Chen, Z. Geospatial Sensor Web Data Discovery and Retrieval Service Based on Middleware. Available online: http:\/\/citeseerx.ist.psu.edu\/viewdoc\/summary?doi=10.1.1.184.4226."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1127","DOI":"10.1080\/13658810903514172","article-title":"An active crawler for discovering geospatial web services and their distribution pattern\u2014A case study of OGC web map service","volume":"24","author":"Li","year":"2010","journal-title":"Int. J. Geogr. Inf. Sci."},{"key":"ref_26","unstructured":"Lopez-Pellicer, F.J., B\u00e9jar, R., Florczyk, A.J., Muro-Medrano, P.R., and Zarazaga-Soria, F.J. (2010, January 23\u201325). State of Play of OGC Web Services Across the Web. Proceedings of the INSPIRE Conference, Krakow, Poland."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Patil, S., Bhattacharjee, S., and Ghosh, S.K. (2014, January 6\u20139). A spatial web crawler for discovering geo-servers and semantic referencing with spatial features. Proceedings of the International Conference on Distributed Computing and Internet Technology, Bhubaneswar, India.","DOI":"10.1007\/978-3-319-04483-5_7"},{"key":"ref_28","unstructured":"Gibotti, F.R., C\u00e2mara, G., and Nogueira, R.A. (2005). Geodiscover\u2014A Specialized Search Engine to Discover Geospatial Data in the Web, GeoInfo."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1145\/362686.362692","article-title":"Space\/time trade-offs in hash coding with allowable errors","volume":"13","author":"Bloom","year":"1970","journal-title":"Commun. ACM"},{"key":"ref_30","unstructured":"Page, L., Brin, S., Motwani, R., and Winograd, T. The Pagerank Citation Ranking: Bringing Order to the Web. Available online: http:\/\/ilpubs.stanford.edu:8090\/422\/."},{"key":"ref_31","unstructured":"Kleene, S.C. Representation of Events in Nerve Nets and Finite Automata. Available online: www.dtic.mil\/cgi-bin\/GetTRDoc?Location=U2&doc=GetTRDoc.pdf&AD=ADA596138."},{"key":"ref_32","unstructured":"De La Beaujardiere, J. Open GIS\u00ae Web Map Server Interface Implementation Specification. Revision 1.0.0. Available online: http:\/\/portal.opengeospatial.org\/files\/?artifact_id=7196."},{"key":"ref_33","unstructured":"De La Beaujardiere, J. Open GIS\u00ae Web Map Server Implementation Specification. Version: 1.3.0. Available online: http:\/\/portal.opengeospatial.org\/files\/?artifact_id=14416."},{"key":"ref_34","unstructured":"Na, A., and Priest, M. Sensor Observation Service. Available online: http:\/\/portal.opengeospatial.org\/files\/?artifact_id=26667."},{"key":"ref_35","unstructured":"Br\u00f6ring, A., Stasch, C., and Echterhoff, J. OGC Sensor Observation Service Interface Standard. Available online: https:\/\/portal.opengeospatial.org\/files\/?artifact_id=47599."},{"key":"ref_36","unstructured":"Vretanos, P.A. Web Feature Service Implementation Specification. Available online: http:\/\/portal.opengeospatial.org\/files\/?artifact_id=7176."},{"key":"ref_37","unstructured":"Vretanos, P.A. Opengis Web Feature Service 2.0 Interface Standard. Available online: http:\/\/portal.opengeospatial.org\/files\/?artifact_id=39967."},{"key":"ref_38","unstructured":"Evans, J.D. Web Coverage Service (WCS), Version 1.0. 0 (Corrigendum). Available online: https:\/\/portal.opengeospatial.org\/files\/05-076."},{"key":"ref_39","unstructured":"Whiteside, A., and Evans, J.D. Web Coverage Service (WCS) Implementation Standard. Available online: https:\/\/portal.opengeospatial.org\/files\/07-067r5."},{"key":"ref_40","unstructured":"Baumann, P. OGC WCS 2.0 Interface Standard\u2014Core. Open Geospatial Consortium. Available online: https:\/\/portal.opengeospatial.org\/files\/09-110r4."},{"key":"ref_41","unstructured":"Maso, J., Pomakis, K., and Julia, N. Open GIS Web Map Tile Service Implementation Standard. Available online: http:\/\/portal.opengeospatial.org\/files\/?artifact_id=35326."},{"key":"ref_42","unstructured":"Schut, P., and Whiteside, A. Open GIS Web Processing Service, OGC Project Document. Available online: http:\/\/portal.opengeospatial.org\/files\/?artifact_id=24151."},{"key":"ref_43","first-page":"370","article-title":"ESRI shapefile technical description","volume":"16","author":"ESRI","year":"1998","journal-title":"Comput. Stat"},{"key":"ref_44","unstructured":"Nebert, D., Whiteside, A., and Vretanos, P. Open GIS Catalogue Services Specification. Available online: http:\/\/portal.opengeospatial.org\/files\/?artifact_id=20555."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1080\/15427951.2004.10129096","article-title":"Network applications of bloom filters: A survey","volume":"1","author":"Broder","year":"2004","journal-title":"Internet Math."}],"container-title":["ISPRS International Journal of Geo-Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2220-9964\/5\/8\/136\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T19:27:54Z","timestamp":1760210874000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2220-9964\/5\/8\/136"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,8,5]]},"references-count":45,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2016,8]]}},"alternative-id":["ijgi5080136"],"URL":"https:\/\/doi.org\/10.3390\/ijgi5080136","relation":{},"ISSN":["2220-9964"],"issn-type":[{"type":"electronic","value":"2220-9964"}],"subject":[],"published":{"date-parts":[[2016,8,5]]}}}