{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T12:25:21Z","timestamp":1788870321126,"version":"build-2803163510"},"reference-count":38,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2010,11,1]],"date-time":"2010-11-01T00:00:00Z","timestamp":1288569600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["5.36E+12"],"award-info":[{"award-number":["5.36E+12"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2010,11]]},"abstract":"<jats:p>When crawling resources, for example, number of machines, crawl-time, and so on, are limited, so a crawler has to decide an optimal order in which to crawl and recrawl Web pages. Ideally, crawlers should request only those Web pages that have changed since the last crawl; in practice, a crawler may not know whether a Web page has changed before downloading it. In this article, we identify features of Web pages that are correlated to their change frequency. We design a crawling algorithm that clusters Web pages based on features that correlate to their change frequencies obtained by examining past history. The crawler downloads a sample of Web pages from each cluster, and depending upon whether a significant number of these Web pages have changed in the last crawl cycle, it decides whether to recrawl the entire cluster. To evaluate the performance of our incremental crawler, we develop an evaluation framework that measures which crawling policy results in the best search results for the end-user. We run experiments on a real Web data set of about 300,000 distinct URLs distributed among 210 Web sites. The results demonstrate that the clustering-based sampling algorithm effectively clusters the pages with similar change patterns, and our clustering-based crawling algorithm outperforms existing algorithms in that it can improve the quality of the user experience for those who query the search engine.<\/jats:p>","DOI":"10.1145\/1852102.1852103","type":"journal-article","created":{"date-parts":[[2010,11,23]],"date-time":"2010-11-23T10:00:38Z","timestamp":1290506438000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":32,"title":["Clustering-based incremental web crawling"],"prefix":"10.1145","volume":"28","author":[{"given":"Qingzhao","family":"Tan","sequence":"first","affiliation":[{"name":"The Pennsylvania State University, University Park, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Prasenjit","family":"Mitra","sequence":"additional","affiliation":[{"name":"The Pennsylvania State University, University Park, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2010,11,23]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the 10th International Symposium on String Processing and Information Retrieval (SPIRE). 28--42","author":"Ali H.","unstructured":"Ali , H. and Williams , H. E . 2003. What's changed&quest; measuring document change in Web crawling for search engines . In Proceedings of the 10th International Symposium on String Processing and Information Retrieval (SPIRE). 28--42 . Ali, H. and Williams, H. E. 2003. What's changed&quest; measuring document change in Web crawling for search engines. In Proceedings of the 10th International Symposium on String Processing and Information Retrieval (SPIRE). 28--42."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2005.10.020"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1097047.1097062"},{"key":"e_1_2_1_4_1","volume-title":"Procedings of the IADIS International Conference (WWW\/INTERNET). 307--314","author":"Bouras C.","unstructured":"Bouras , C. , Poulopoulos , V. , and Thanou , A . 2005. Creating a polite adaptive and selective incremental crawler . In Procedings of the IADIS International Conference (WWW\/INTERNET). 307--314 . Bouras, C., Poulopoulos, V., and Thanou, A. 2005. Creating a polite adaptive and selective incremental crawler. In Procedings of the IADIS International Conference (WWW\/INTERNET). 307--314."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1389-1286(00)00045-1"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.841784"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(98)00110-X"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1389-1286(99)00052-3"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 26th International Conference on Very Large Data Bases (VLDB). Morgan Kaufmann Publishers Inc.","author":"Cho J.","unstructured":"Cho , J. and Garcia-Molina , H . 2000a. The evolution of the Web and implications for an incremental crawler . In Proceedings of the 26th International Conference on Very Large Data Bases (VLDB). Morgan Kaufmann Publishers Inc. , San Francisco, CA, 200--209. Cho, J. and Garcia-Molina, H. 2000a. The evolution of the Web and implications for an incremental crawler. In Proceedings of the 26th International Conference on Very Large Data Bases (VLDB). Morgan Kaufmann Publishers Inc., San Francisco, CA, 200--209."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/335191.335391"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/958942.958945"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 28th International Conference on Very Large Data Bases (VLDB). VLDB Endowment, 514--525","author":"Cho J.","unstructured":"Cho , J. and Ntoulas , A . 2002. Effective change detection using sampling . In Proceedings of the 28th International Conference on Very Large Data Bases (VLDB). VLDB Endowment, 514--525 . Cho, J. and Ntoulas, A. 2002. Effective change detection using sampling. In Proceedings of the 28th International Conference on Very Large Data Bases (VLDB). VLDB Endowment, 514--525."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/988672.988676"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the 1st Recherche d'Informations Assistee par Ordinateur Conference (RIAO). 481--491","author":"de Bra P.","unstructured":"de Bra , P. , jan Houben , G. , Kornatzky , Y. , and Post , R . 1994. Information retrieval in distributed hypertexts . In Proceedings of the 1st Recherche d'Informations Assistee par Ordinateur Conference (RIAO). 481--491 . de Bra, P., jan Houben, G., Kornatzky, Y., and Post, R. 1994. Information retrieval in distributed hypertexts. In Proceedings of the 1st Recherche d'Informations Assistee par Ordinateur Conference (RIAO). 481--491."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the USENIX Symposium on Internet Technologies and Systems on USENIX Symposium on Internet Technologies and Systems (USITS). USENIX Association","author":"Douglis F.","unstructured":"Douglis , F. , Feldmann , A. , Krishnamurthy , B. , and Mogul , J . 1997. Rate of change and other metrics: a live study of the World Wide Web . In Proceedings of the USENIX Symposium on Internet Technologies and Systems on USENIX Symposium on Internet Technologies and Systems (USITS). USENIX Association , Berkeley, CA, 147--158. Douglis, F., Feldmann, A., Krishnamurthy, B., and Mogul, J. 1997. Rate of change and other metrics: a live study of the World Wide Web. In Proceedings of the USENIX Symposium on Internet Technologies and Systems on USENIX Symposium on Internet Technologies and Systems (USITS). USENIX Association, Berkeley, CA, 147--158."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/371920.371960"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/775152.775246"},{"key":"e_1_2_1_18_1","unstructured":"Grimmett G. and Stirzaker D. 1992. Probability and Random Processes 2nd Ed. Oxford University Press Oxford UK.  Grimmett G. and Stirzaker D. 1992. Probability and Random Processes 2nd Ed. Oxford University Press Oxford UK."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(98)00038-5"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1019213109274"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/354756.354772"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/775152.775156"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1150402.1150479"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of The 3rd World Conference of the WWW, Internet, and Intranet (WebNet). 92--97","author":"Liu B.","unstructured":"Liu , B. , Abdulla , G. , Johnson , T. , and Fox , E. A . 1998. Web response time and proxy caching . In Proceedings of The 3rd World Conference of the WWW, Internet, and Intranet (WebNet). 92--97 . Liu, B., Abdulla, G., Johnson, T., and Fox, E. A. 1998. Web response time and proxy caching. In Proceedings of The 3rd World Conference of the WWW, Internet, and Intranet (WebNet). 92--97."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383995"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1389-1286(00)00035-9"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/988672.988674"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1367497.1367557"},{"key":"e_1_2_1_29_1","volume-title":"Technical Rep. 1999--66, Stanford InfoLab. November.","author":"Page L.","year":"1999","unstructured":"Page , L. , Brin , S. , Motwani , R. , and Winograd , T . 1999 . The pagerank citation ranking: Bringing order to the Web . Technical Rep. 1999--66, Stanford InfoLab. November. Page, L., Brin, S., Motwani, R., and Winograd, T. 1999. The pagerank citation ranking: Bringing order to the Web. Technical Rep. 1999--66, Stanford InfoLab. November."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1060745.1060805"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1287\/isre.1080.0231"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.253.5023.974"},{"key":"e_1_2_1_33_1","volume-title":"Statistics: Concepts and Applications. Benjamin\/Cummings Publishing Company.","author":"Schelfler W.","year":"1988","unstructured":"Schelfler , W. 1988 . Statistics: Concepts and Applications. Benjamin\/Cummings Publishing Company. Schelfler, W. 1988. Statistics: Concepts and Applications. Benjamin\/Cummings Publishing Company."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/331403.331405"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321440.1321516"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of International Conference of Web Engineering (ICWE). 285--300","author":"Tan Q.","unstructured":"Tan , Q. , Zhuang , Z. , Mitra , P. , and Giles , C. L . 2007b. Efficiently detecting Web page updates using samples . In Proceedings of International Conference of Web Engineering (ICWE). 285--300 . Tan, Q., Zhuang, Z., Mitra, P., and Giles, C. L. 2007b. Efficiently detecting Web page updates using samples. In Proceedings of International Conference of Web Engineering (ICWE). 285--300."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/511446.511465"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526870"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1852102.1852103","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1852102.1852103","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T08:08:17Z","timestamp":1750234097000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1852102.1852103"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,11]]},"references-count":38,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2010,11]]}},"alternative-id":["10.1145\/1852102.1852103"],"URL":"https:\/\/doi.org\/10.1145\/1852102.1852103","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2010,11]]},"assertion":[{"value":"2008-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2010-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2010-11-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}