{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,8]],"date-time":"2026-06-08T11:01:20Z","timestamp":1780916480994,"version":"3.54.1"},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2004,11,1]],"date-time":"2004-11-01T00:00:00Z","timestamp":1099267200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Internet Technol."],"published-print":{"date-parts":[[2004,11]]},"abstract":"<jats:p>Topical crawlers are increasingly seen as a way to address the scalability limitations of universal search engines, by distributing the crawling process across users, queries, or even client computers. The context available to such crawlers can guide the navigation of links with the goal of efficiently locating highly relevant target pages. We developed a framework to fairly evaluate topical crawling algorithms under a number of performance metrics. Such a framework is employed here to evaluate different algorithms that have proven highly competitive among those proposed in the literature and in our own previous research. In particular we focus on the tradeoff between exploration and exploitation of the cues available to a crawler, and on adaptive crawlers that use machine learning techniques to guide their search. We find that the best performance is achieved by a novel combination of explorative and exploitative bias, and introduce an evolutionary crawler that surpasses the performance of the best nonadaptive crawler after sufficiently long crawls. We also analyze the computational complexity of the various crawlers and discuss how performance and complexity scale with available resources. Evolutionary crawlers achieve high efficiency and scalability by distributing the work across concurrent agents, resulting in the best performance\/cost ratio.<\/jats:p>","DOI":"10.1145\/1031114.1031117","type":"journal-article","created":{"date-parts":[[2005,1,26]],"date-time":"2005-01-26T16:35:53Z","timestamp":1106757353000},"page":"378-419","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":163,"title":["Topical web crawlers"],"prefix":"10.1145","volume":"4","author":[{"given":"Filippo","family":"Menczer","sequence":"first","affiliation":[{"name":"Indiana University, Bloomington, IN"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gautam","family":"Pant","sequence":"additional","affiliation":[{"name":"University of Utah, Salt Lake City, UT"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Padmini","family":"Srinivasan","sequence":"additional","affiliation":[{"name":"University of Iowa, Iowa City, IA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2004,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/371920.371955"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1389-1286(99)00045-6"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 9th International World-Wide Web Conference.","author":"Brewington B. E.","unstructured":"Brewington , B. E. and Cybenko , G . 2000. How dynamic is the Web? In Proceedings of the 9th International World-Wide Web Conference. Brewington, B. E. and Cybenko, G. 2000. How dynamic is the Web? In Proceedings of the 9th International World-Wide Web Conference."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(98)00110-X"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1389-1286(99)00052-3"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 26th International Conference on Very Large Databases (VLDB).","author":"Cho J.","unstructured":"Cho , J. and Garcia-Molina , H . 2000. The evolution of the Web and implications for an incremental crawler . In Proceedings of the 26th International Conference on Very Large Databases (VLDB). Cho, J. and Garcia-Molina, H. 2000. The evolution of the Web and implications for an incremental crawler. In Proceedings of the 26th International Conference on Very Large Databases (VLDB)."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(98)00108-1"},{"key":"e_1_2_1_8_1","unstructured":"Cyveillance. 2000. Sizing the internet. White paper. http:\/\/www.cyveillance.com\/.  Cyveillance. 2000. Sizing the internet. White paper. http:\/\/www.cyveillance.com\/."},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 1st International World Wide Web Conference","author":"De Bra P.","unstructured":"De Bra , P. and Post , R . 1994. Information retrieval in the World Wide Web: Making client-based searching feasible . In Proceedings of the 1st International World Wide Web Conference ( Geneva). De Bra, P. and Post, R. 1994. Information retrieval in the World Wide Web: Making client-based searching feasible. In Proceedings of the 1st International World Wide Web Conference (Geneva)."},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 26th International Conference on Very Large Databases (VLDB","author":"Diligenti M.","year":"2000","unstructured":"Diligenti , M. , Coetzee , F. , Lawrence , S. , Giles , C. L. , and Gori , M . 2000. Focused crawling using context graphs . In Proceedings of the 26th International Conference on Very Large Databases (VLDB 2000 ). Cairo, Egypt, 527--534. Diligenti, M., Coetzee, F., Lawrence, S., Giles, C. L., and Gori, M. 2000. Focused crawling using context graphs. In Proceedings of the 26th International Conference on Very Large Databases (VLDB 2000). Cairo, Egypt, 527--534."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.989932"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/276627.276652"},{"key":"e_1_2_1_13_1","volume-title":"Efficient computation of pagerank. Tech. rep","author":"Haveliwala T.","unstructured":"Haveliwala , T. 1999. Efficient computation of pagerank. Tech. rep ., Stanford Database Group . Haveliwala, T. 1999. Efficient computation of pagerank. Tech. rep., Stanford Database Group."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the 8th International World Wide Web Conference","author":"Henzinger M.","unstructured":"Henzinger , M. , Heydon , A. , Mitzenmacher , M. , and Najork , M . 1999. Measuring search engine quality using random walks on the Web . In Proceedings of the 8th International World Wide Web Conference ( Toronto). 213--225. Henzinger, M., Heydon, A., Mitzenmacher, M., and Najork, M. 1999. Measuring search engine quality using random walks on the Web. In Proceedings of the 8th International World Wide Web Conference (Toronto). 213--225."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the 7th International World-Wide Web Conference.","author":"Hersovici M.","unstructured":"Hersovici , M. , Jacovi , M. , Maarek , Y. S. , Pelleg , D. , Shtalhaim , M. , and Ur , S . 1998. The shark-search algorithm---An application: Tailored Web site mapping . In Proceedings of the 7th International World-Wide Web Conference. Hersovici, M., Jacovi, M., Maarek, Y. S., Pelleg, D., Shtalhaim, M., and Ur, S. 1998. The shark-search algorithm---An application: Tailored Web site mapping. In Proceedings of the 7th International World-Wide Web Conference."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/324133.324140"},{"key":"e_1_2_1_17_1","doi-asserted-by":"crossref","unstructured":"Kleinberg J. and Lawrence S. 2001. The structure of the Web. Science 294 5548 1849--1850.  Kleinberg J. and Lawrence S. 2001. The structure of the Web. Science 294 5548 1849--1850.","DOI":"10.1126\/science.1067014"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 41st Annual IEEE Symposium on Foundations of Computer Science. IEEE Computer Society Press","author":"Kumar S.","unstructured":"Kumar , S. , Raghavan , P. , Rajagopalan , S. , Sivakumar , D. , Tomkins , A. , and Upfal , E . 2000. Stochastic models for the Web graph . In Proceedings of the 41st Annual IEEE Symposium on Foundations of Computer Science. IEEE Computer Society Press , Silver Spring, MD, 57--65. Kumar, S., Raghavan, P., Rajagopalan, S., Sivakumar, D., Tomkins, A., and Upfal, E. 2000. Stochastic models for the Web graph. In Proceedings of the 41st Annual IEEE Symposium on Foundations of Computer Science. IEEE Computer Society Press, Silver Spring, MD, 57--65."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1389-1286(99)00040-7"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.280.5360.98"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1038\/21987"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 16th International Joint Conference on Artificial Intelligence. Morgan Kaufmann","author":"McCallum A.","unstructured":"McCallum , A. , Nigam , K. , Rennie , J. , and Seymore , K . 1999. A machine learning approach to building domain-specific search engines . In Proceedings of the 16th International Joint Conference on Artificial Intelligence. Morgan Kaufmann , San Francisco, CA, 662--667. McCallum, A., Nigam, K., Rennie, J., and Seymore, K. 1999. A machine learning approach to building domain-specific search engines. In Proceedings of the 16th International Joint Conference on Artificial Intelligence. Morgan Kaufmann, San Francisco, CA, 662--667."},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the 14th International Conference on Machine Learning. Morgan Kaufmann","author":"Menczer F.","year":"1997","unstructured":"Menczer , F. 1997 . ARACHNID: Adaptive Retrieval Agents Choosing Heuristic Neighborhoods for Information Discovery . In Proceedings of the 14th International Conference on Machine Learning. Morgan Kaufmann , San Francisco, CA, 227--235. Menczer, F. 1997. ARACHNID: Adaptive Retrieval Agents Choosing Heuristic Neighborhoods for Information Discovery. In Proceedings of the 14th International Conference on Machine Learning. Morgan Kaufmann, San Francisco, CA, 227--235."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-9236(02)00106-9"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.20081"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/280765.280794"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007653114902"},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Menczer F. and Monge A. 1999. Scalable Web search by adaptive online agents: An InfoSpiders case study. In Intelligent Information Agents: Agent-Based Information Discovery and Management on the Internet M. Klusch Ed. Springer Berlin 323--347.  Menczer F. and Monge A. 1999. Scalable Web search by adaptive online agents: An InfoSpiders case study. In Intelligent Information Agents: Agent-Based Information Discovery and Management on the Internet M. Klusch Ed. Springer Berlin 323--347.","DOI":"10.1007\/978-3-642-60018-0_17"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383995"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/267658.267747"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/371920.371965"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/4236.914644"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/4236.914645"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 7th European Conference on Research and Advanced Technology for Digital Libraries (ECDL), T. Koch and I. Solvberg, Eds. Lecture Notes in Computer Science","volume":"2769","author":"Pant G.","unstructured":"Pant , G. , Bradshaw , S. , and Menczer , F . 2003. Search engine - crawler symbiosis . In Proceedings of the 7th European Conference on Research and Advanced Technology for Digital Libraries (ECDL), T. Koch and I. Solvberg, Eds. Lecture Notes in Computer Science , Vol. 2769 . Springer Verlag, Berlin. Pant, G., Bradshaw, S., and Menczer, F. 2003. Search engine - crawler symbiosis. In Proceedings of the 7th European Conference on Research and Advanced Technology for Digital Libraries (ECDL), T. Koch and I. Solvberg, Eds. Lecture Notes in Computer Science, Vol. 2769. Springer Verlag, Berlin."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1014853428272"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 7th European Conference on Research and Advanced Technology for Digital Libraries (ECDL), T. Koch and I. Solvberg, Eds. Lecture Notes in Computer Science","volume":"2769","author":"Pant G.","unstructured":"Pant , G. and Menczer , F . 2003. Topical crawling for business intelligence . In Proceedings of the 7th European Conference on Research and Advanced Technology for Digital Libraries (ECDL), T. Koch and I. Solvberg, Eds. Lecture Notes in Computer Science , Vol. 2769 . Berlin. Pant, G. and Menczer, F. 2003. Topical crawling for business intelligence. In Proceedings of the 7th European Conference on Research and Advanced Technology for Digital Libraries (ECDL), T. Koch and I. Solvberg, Eds. Lecture Notes in Computer Science, Vol. 2769. Berlin."},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the WWW-02 Workshop on Web Dynamics.","author":"Pant G.","unstructured":"Pant , G. , Srinivasan , P. , and Menczer , F . 2002. Exploration versus exploitation in topic driven crawlers . In Proceedings of the WWW-02 Workshop on Web Dynamics. Pant, G., Srinivasan, P., and Menczer, F. 2002. Exploration versus exploitation in topic driven crawlers. In Proceedings of the WWW-02 Workshop on Web Dynamics."},{"key":"e_1_2_1_38_1","volume-title":"Proceedings of the 2nd International World Wide Web Conference","author":"Pinkerton B.","year":"1994","unstructured":"Pinkerton , B. 1994 . Finding what people want: Experiences with the WebCrawler . In Proceedings of the 2nd International World Wide Web Conference ( Chicago). Pinkerton, B. 1994. Finding what people want: Experiences with the WebCrawler. In Proceedings of the 2nd International World Wide Web Conference (Chicago)."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb046814"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the 16th International Conference on Machine Learning. Morgan Kaufmann","author":"Rennie J.","year":"1999","unstructured":"Rennie , J. and McCallum , A. 1999 . Using reinforcement learning to spider the Web efficiently . In Proceedings of the 16th International Conference on Machine Learning. Morgan Kaufmann , San Francisco, CA, 335--343. Rennie, J. and McCallum, A. 1999. Using reinforcement learning to spider the Web efficiently. In Proceedings of the 16th International Conference on Machine Learning. Morgan Kaufmann, San Francisco, CA, 335--343."},{"key":"e_1_2_1_41_1","first-page":"318","article-title":"Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, D. Rumelhart and J. McClelland, Eds. Vol. 1. Bradford Books (MIT Press), Cambridge, MA","volume":"8","author":"Rumelhart D.","year":"1986","unstructured":"Rumelhart , D. , Hinton , G. , and Williams , R. 1986 . Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, D. Rumelhart and J. McClelland, Eds. Vol. 1. Bradford Books (MIT Press), Cambridge, MA , Chapter 8 , 318 -- 362 . Rumelhart, D., Hinton, G., and Williams, R. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, D. Rumelhart and J. McClelland, Eds. Vol. 1. Bradford Books (MIT Press), Cambridge, MA, Chapter 8, 318--362.","journal-title":"Chapter"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-005-6993-5"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 8th International World Wide Web Conference","author":"Wills C.","unstructured":"Wills , C. and Mikhailov , M . 1999. Towards a better understanding of Web resources and server responses for improved caching . In Proceedings of the 8th International World Wide Web Conference ( Toronto). Wills, C. and Mikhailov, M. 1999. Towards a better understanding of Web resources and server responses for improved caching. In Proceedings of the 8th International World Wide Web Conference (Toronto)."}],"container-title":["ACM Transactions on Internet Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1031114.1031117","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1031114.1031117","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T16:30:56Z","timestamp":1750264256000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1031114.1031117"}},"subtitle":["Evaluating adaptive algorithms"],"short-title":[],"issued":{"date-parts":[[2004,11]]},"references-count":43,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2004,11]]}},"alternative-id":["10.1145\/1031114.1031117"],"URL":"https:\/\/doi.org\/10.1145\/1031114.1031117","relation":{},"ISSN":["1533-5399","1557-6051"],"issn-type":[{"value":"1533-5399","type":"print"},{"value":"1557-6051","type":"electronic"}],"subject":[],"published":{"date-parts":[[2004,11]]},"assertion":[{"value":"2004-11-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}