{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T18:53:46Z","timestamp":1772909626792,"version":"3.50.1"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2018,9,27]],"date-time":"2018-09-27T00:00:00Z","timestamp":1538006400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Web"],"published-print":{"date-parts":[[2018,11,30]]},"abstract":"<jats:p>With the proliferation of web spam and infinite autogenerated web content, large-scale web crawlers require low-complexity ranking methods to effectively budget their limited resources and allocate bandwidth to reputable sites. In this work, we assume crawls that produce frontiers orders of magnitude larger than RAM, where sorting of pending URLs is infeasible in real time. Under these constraints, the main objective is to quickly compute domain budgets and decide which of them can be massively crawled. Those ranked at the top of the list receive aggressive crawling allowances, while all other domains are visited at some small default rate. To shed light on Internet-wide spam avoidance, we study topology-based ranking algorithms on domain-level graphs from the two largest academic crawls: a 6.3B-page IRLbot dataset and a 1B-page ClueWeb09 exploration. We first propose a new methodology for comparing the various rankings and then show that in-degree BFS-based techniques decisively outperform classic PageRank-style methods, including TrustRank. However, since BFS requires several orders of magnitude higher overhead and is generally infeasible for real-time use, we propose a fast, accurate, and scalable estimation method called TSE that can achieve much better crawl prioritization in practice. It is especially beneficial in applications with limited hardware resources.<\/jats:p>","DOI":"10.1145\/3182180","type":"journal-article","created":{"date-parts":[[2018,9,28]],"date-time":"2018-09-28T17:47:33Z","timestamp":1538156853000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Unsupervised Domain Ranking in Large-Scale Web Crawls"],"prefix":"10.1145","volume":"12","author":[{"given":"Yi","family":"Cui","sequence":"first","affiliation":[{"name":"Texas A8M University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Clint","family":"Sparkman","sequence":"additional","affiliation":[{"name":"United States Air Force Academy, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hsin-Tsang","family":"Lee","sequence":"additional","affiliation":[{"name":"Microsoft Corp., USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dmitri","family":"Loguinov","sequence":"additional","affiliation":[{"name":"Texas A8M University, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,9,27]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1451983.1451994"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/775152.775192"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM.2015.7218539"},{"key":"e_1_2_1_4_1","unstructured":"alexa.com. Retrieved in 2015 from http:\/\/www.alexa.com\/.  alexa.com. Retrieved in 2015 from http:\/\/www.alexa.com\/."},{"key":"e_1_2_1_5_1","volume-title":"We Knew the Web Was Big... (July","author":"Alpert Jesse","year":"2008"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505668"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1062745.1062768"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063592"},{"key":"e_1_2_1_9_1","volume-title":"Workshop on Adversarial Information Retrieval on the Web.","author":"Becchetti Luca","year":"2006"},{"key":"e_1_2_1_10_1","volume-title":"Workshop on Web Mining and Web Usage Analysis (WebKDD).","author":"Becchetti Luca","year":"2006"},{"key":"e_1_2_1_11_1","volume-title":"Workshop on Adversarial Information Retrieval on the Web.","author":"Benczur Andras","year":"2006"},{"key":"e_1_2_1_12_1","volume-title":"Workshop on Adversarial Information Retrieval on the Web.","author":"Benczur Andras A.","year":"2005"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1002\/spe.587"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5555\/297805.297827"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.5555\/1032657.1034011"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1281100.1281124"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2007.370233"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1992.10475194"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/584792.584882"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1031171.1031248"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1149121.1149124"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/297805.297835"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1531914.1531917"},{"key":"e_1_2_1_24_1","unstructured":"ClueWeb09 Dataset. Retrieved in 2015 from http:\/\/www.lemurproject.org\/clueweb09\/.  ClueWeb09 Dataset. Retrieved in 2015 from http:\/\/www.lemurproject.org\/clueweb09\/."},{"key":"e_1_2_1_25_1","unstructured":"CLuE cluster. Retrieved from http:\/\/www.nsf.gov\/cise\/clue\/index.jsp.  CLuE cluster. Retrieved from http:\/\/www.nsf.gov\/cise\/clue\/index.jsp."},{"key":"e_1_2_1_26_1","unstructured":"ClueWeb09 documentation. Retrieved in 2015 from http:\/\/boston.lti.cs.cmu.edu\/Data\/web08-bst\/planning.html.  ClueWeb09 documentation. Retrieved in 2015 from http:\/\/boston.lti.cs.cmu.edu\/Data\/web08-bst\/planning.html."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/11564096_14"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/863955.863992"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/371920.371960"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(94)90151-1"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/988672.988714"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148187"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1017074.1017077"},{"key":"e_1_2_1_34_1","unstructured":"Google Toolbar. Retrieved in 2008 from http:\/\/toolbar.google.com.  Google Toolbar. Retrieved in 2008 from http:\/\/toolbar.google.com."},{"key":"e_1_2_1_35_1","unstructured":"IRLbot PLD graph. Retrieved in 2018 from http:\/\/irl.cs.tamu.edu\/projects\/web\/.  IRLbot PLD graph. Retrieved in 2018 from http:\/\/irl.cs.tamu.edu\/projects\/web\/."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1147\/sj.431.0064"},{"key":"e_1_2_1_37_1","volume-title":"International Conference on Very Large Data Bases (VLDB). 439--450","author":"Gy\u00f6ngyi Zolt\u00e1n","year":"2006"},{"key":"e_1_2_1_38_1","volume-title":"International Conference on Very Large Data Bases (VLDB). 517--528","author":"Gy\u00f6ngyi Zolt\u00e1n","year":"2005"},{"key":"e_1_2_1_39_1","volume-title":"Workshop on Adversarial Information Retrieval on the Web. 39--47","author":"Gy\u00f6ngyi Zolt\u00e1n","year":"2005"},{"key":"e_1_2_1_40_1","volume-title":"International Conference on Very Large Data Bases (VLDB). 576--587","author":"Gy\u00f6ngyi Zolt\u00e1n","year":"2004"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1026711.1026760"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1019213109274"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.5555\/347319.346288"},{"key":"e_1_2_1_44_1","unstructured":"Internet Archive. Retrieved in 2018 from http:\/\/archive.org\/.  Internet Archive. Retrieved in 2018 from http:\/\/archive.org\/."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/503376.503442"},{"key":"e_1_2_1_46_1","volume-title":"IEEE International Symposium on Communications and Information Technologies (ISCIT).","author":"Surasak Sanguanpong Kasom","year":"2002"},{"key":"e_1_2_1_47_1","unstructured":"Laboratory for Web Algorithmics. Retrieved in 2018 from http:\/\/law.dsi.unimi.it\/.  Laboratory for Web Algorithmics. Retrieved in 2018 from http:\/\/law.dsi.unimi.it\/."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1541822.1541823"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/382979.383041"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/4236.707687"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390412"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(94)90149-X"},{"key":"e_1_2_1_53_1","volume-title":"Workshop on Adversarial Information Retrieval on the Web.","author":"Mishne Gilad","year":"2005"},{"key":"e_1_2_1_54_1","volume-title":"Public Suffix List. (May","year":"2013"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/371920.371965"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1060745.1060828"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/1135777.1135794"},{"key":"e_1_2_1_59_1","unstructured":"Nutch. Retrieved in 2018 from http:\/\/nutch.apache.org\/.  Nutch. Retrieved in 2018 from http:\/\/nutch.apache.org\/."},{"key":"e_1_2_1_60_1","volume-title":"Introduction to Grid Computing via Map-Reduce 8 Hadoop. (Dec","author":"Padmanabhan Jothi","year":"2010"},{"key":"e_1_2_1_61_1","volume-title":"International Conference on World Wide Web (WWW).","author":"Pinkerton Brian","year":"1994"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/1451983.1451990"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/1135777.1135881"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1244408.1244417"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.5555\/876875.879017"},{"key":"e_1_2_1_67_1","volume-title":"Breakfast with Google\u2019s Search Team. (Aug","author":"Signhal Amit","year":"2012"},{"key":"e_1_2_1_68_1","volume-title":"SIGIR Workshop on Distributed Information Retrieval. 126--142","author":"Singh Aameek","year":"2003"},{"key":"e_1_2_1_69_1","volume-title":"SpamAssassin Domain Blacklist. (Aug","author":"Stearns William","year":"2008"},{"key":"e_1_2_1_70_1","volume-title":"International Workshop on the Web and Databases (WebDB). 67--72","author":"Suel T."},{"key":"e_1_2_1_71_1","unstructured":"The Stanford WebBase Project. 2008. WebBase Archive. (Sept. 2008). Retrieved from http:\/\/dbpubs.stanford.edu:8091\/&sim;testbed\/doc2\/WebBase\/.  The Stanford WebBase Project. 2008. WebBase Archive. (Sept. 2008). Retrieved from http:\/\/dbpubs.stanford.edu:8091\/&sim;testbed\/doc2\/WebBase\/."},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/775152.775202"},{"key":"e_1_2_1_73_1","volume-title":"8th Australasian Document Computing Symposium.","author":"Upstill Trystan","year":"2003"},{"key":"e_1_2_1_74_1","volume-title":"IEEE Conference on Computer Communications (INFOCOM). 1808--1816","author":"Wang X."},{"key":"e_1_2_1_75_1","volume-title":"Conference on Email and Anti-Spam (CEAS).","author":"Webb Steve","year":"2007"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458129"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/1244408.1244416"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/1062745.1062762"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/1135777.1135792"},{"key":"e_1_2_1_80_1","doi-asserted-by":"crossref","unstructured":"J. Wu and K. Aberer. 2004. Using Siterank for decentralized computation of web document ranking. In Adaptive Hypermedia. 265--274.  J. Wu and K. Aberer. 2004. Using Siterank for decentralized computation of web document ranking. In Adaptive Hypermedia. 265--274.","DOI":"10.1007\/978-3-540-27780-4_30"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076068"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/1879141.1879148"},{"key":"e_1_2_1_83_1","volume-title":"Datasets for Research on Web Spam Detection. (Sept","author":"Barcelona Yahoo Research","year":"2008"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0164-1212(01)00091-7"}],"container-title":["ACM Transactions on the Web"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3182180","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3182180","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T21:41:20Z","timestamp":1750282880000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3182180"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,9,27]]},"references-count":82,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2018,11,30]]}},"alternative-id":["10.1145\/3182180"],"URL":"https:\/\/doi.org\/10.1145\/3182180","relation":{},"ISSN":["1559-1131","1559-114X"],"issn-type":[{"value":"1559-1131","type":"print"},{"value":"1559-114X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,9,27]]},"assertion":[{"value":"2017-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-09-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}