{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T16:41:54Z","timestamp":1783615314663,"version":"3.55.0"},"reference-count":38,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2018,5,31]],"date-time":"2018-05-31T00:00:00Z","timestamp":1527724800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"EU, EU-FET and NADINE","award":["GA 288956"],"award-info":[{"award-number":["GA 288956"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Web"],"published-print":{"date-parts":[[2018,5,31]]},"abstract":"<jats:p>Although web crawlers have been around for twenty years by now, there is virtually no freely available, open-source crawling software that guarantees high throughput, overcomes the limits of single-machine systems, and, at the same time, scales linearly with the amount of resources available. This article aims at filling this gap, through the description of BUbiNG, our next-generation web crawler built upon the authors\u2019 experience with UbiCrawler\u00a0[9] and on the last ten years of research on the topic. BUbiNG is an open-source Java fully distributed crawler; a single BUbiNG agent, using sizeable hardware, can crawl several thousand pages per second respecting strict politeness constraints, both host- and IP-based. Unlike existing open-source distributed crawlers that rely on batch techniques (like MapReduce), BUbiNG job distribution is based on modern high-speed protocols to achieve very high throughput.<\/jats:p>","DOI":"10.1145\/3160017","type":"journal-article","created":{"date-parts":[[2018,6,4]],"date-time":"2018-06-04T13:41:34Z","timestamp":1528119694000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":32,"title":["BUbiNG"],"prefix":"10.1145","volume":"12","author":[{"given":"Paolo","family":"Boldi","sequence":"first","affiliation":[{"name":"Dipartimento di Informatica, Universit\u00e0 degli Studi di Milano, Milano, MI, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Marino","sequence":"additional","affiliation":[{"name":"Dipartimento di Informatica, Universit\u00e0 degli Studi di Milano, Milano, MI, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Massimo","family":"Santini","sequence":"additional","affiliation":[{"name":"Dipartimento di Informatica, Universit\u00e0 degli Studi di Milano, Milano, MI, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3257-651X","authenticated-orcid":false,"given":"Sebastiano","family":"Vigna","sequence":"additional","affiliation":[{"name":"Dipartimento di Informatica, Universit\u00e0 degli Studi di Milano, Milano, MI, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,6]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Retrieved","author":"Internet Archive","year":"1996","unstructured":"Internet Archive website. 1996 . Homepage . Retrieved May 16, 2018 from http:\/\/archive.org\/web\/web.php. Internet Archive website. 1996. Homepage. Retrieved May 16, 2018 from http:\/\/archive.org\/web\/web.php."},{"key":"e_1_2_1_2_1","volume-title":"Retrieved","author":"Site Heritrix Web","year":"2003","unstructured":"Heritrix Web Site . 2003 . Homepage . Retrieved May 16, 2018 from https:\/\/webarchive.jira.com\/wiki\/display\/Heritrix\/. Heritrix Web Site. 2003. Homepage. Retrieved May 16, 2018 from https:\/\/webarchive.jira.com\/wiki\/display\/Heritrix\/."},{"key":"e_1_2_1_3_1","volume-title":"Retrieved","author":"Dataset The","year":"2009","unstructured":"The ClueWeb09 Dataset . 2009 . Homepage . Retrieved May 16, 2018 from http:\/\/lemurproject.org\/clueweb09\/. The ClueWeb09 Dataset. 2009. Homepage. Retrieved May 16, 2018 from http:\/\/lemurproject.org\/clueweb09\/."},{"key":"e_1_2_1_4_1","volume-title":"Retrieved","author":"ISO","year":"2018","unstructured":"ISO 28500:2009 , Information and documentation\u2014WARC file format . Retrieved May 16, 2018 from https:\/\/www.iso.org\/standard\/44717.html. ISO 28500:2009, Information and documentation\u2014WARC file format. Retrieved May 16, 2018 from https:\/\/www.iso.org\/standard\/44717.html."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1538902.1538905"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM.2015.7218539"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.17487\/rfc3986"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/362686.362692"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1002\/spe.587"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2567948.2577304"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDMW.2013.10"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1080\/15427951.2013.865686"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(98)00110-X"},{"key":"e_1_2_1_14_1","volume-title":"Selected Papers from the 6th International Conference on World Wide Web","author":"Broder Andrei Z.","unstructured":"Andrei Z. Broder , Steven C. Glassman , Mark S. Manasse , and Geoffrey Zweig . 1997. Syntactic clustering of the web . In Selected Papers from the 6th International Conference on World Wide Web . Elsevier Science Publishers Ltd ., Essex, UK, 1157--1166. Andrei Z. Broder, Steven C. Glassman, Mark S. Manasse, and Geoffrey Zweig. 1997. Syntactic clustering of the web. In Selected Papers from the 6th International Conference on World Wide Web. Elsevier Science Publishers Ltd., Essex, UK, 1157--1166."},{"key":"e_1_2_1_15_1","volume-title":"Crawling towards eternity: Building an archive of the world wide web. Web Techniques 2, 5","author":"Burner M.","year":"1997","unstructured":"M. Burner . 1997. Crawling towards eternity: Building an archive of the world wide web. Web Techniques 2, 5 ( 1997 ). M. Burner. 1997. Crawling towards eternity: Building an archive of the world wide web. Web Techniques 2, 5 (1997)."},{"key":"e_1_2_1_16_1","unstructured":"Jamie Callan. 2012. The Lemur Project and its ClueWeb12 Dataset. Invited talk at the SIGIR 2012 Workshop on Open-Source Information Retrieval. (2012).  Jamie Callan. 2012. The Lemur Project and its ClueWeb12 Dataset. Invited talk at the SIGIR 2012 Workshop on Open-Source Information Retrieval. (2012)."},{"key":"e_1_2_1_17_1","unstructured":"Soumen Chakrabarti. 2003. Mining the Web\u2014Discovering Knowledge from Hypertext Data. Morgan Kaufmann. I--XVIII 1--345 pages.   Soumen Chakrabarti. 2003. Mining the Web\u2014Discovering Knowledge from Hypertext Data. Morgan Kaufmann. I--XVIII 1--345 pages."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/509907.509965"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1137\/070710111"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/371920.371960"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(94)90151-1"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/321812.321820"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/775152.775246"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.5555\/197754.195683"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1019213109274"},{"key":"e_1_2_1_26_1","volume-title":"Nutch: A flexible and scalable open-source web search engine. CommerceNet Labs Technical Report 04-04.","author":"Khare Rohit","year":"2004","unstructured":"Rohit Khare , Doug Cutting , Kragen Sitaker , and Adam Rifkin . 2004 . Nutch: A flexible and scalable open-source web search engine. CommerceNet Labs Technical Report 04-04. Rohit Khare, Doug Cutting, Kragen Sitaker, and Adam Rifkin. 2004. Nutch: A flexible and scalable open-source web search engine. CommerceNet Labs Technical Report 04-04."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1541822.1541823"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242592"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-7552(94)90149-X"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1561\/106.00000003"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/248052.248106"},{"key":"e_1_2_1_32_1","volume-title":"Salman Hooshmand, Gregor V. Bochmann, Guy-Vincent Jourdan, and Iosif-Viorel Onut.","author":"Mirtaheri Seyed M.","year":"2013","unstructured":"Seyed M. Mirtaheri , Mustafa Emre Dincturk , Salman Hooshmand, Gregor V. Bochmann, Guy-Vincent Jourdan, and Iosif-Viorel Onut. 2013 . A brief history of web crawlers. In CASCON. Seyed M. Mirtaheri, Mustafa Emre Dincturk, Salman Hooshmand, Gregor V. Bochmann, Guy-Vincent Jourdan, and Iosif-Viorel Onut. 2013. A brief history of web crawlers. In CASCON."},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the 4th International Web Archiving Workshop (IWAW\u201904)","author":"Mohr Gordon","year":"2004","unstructured":"Gordon Mohr , Michele Kimpton , Micheal Stack , and Igor Ranitovic . 2004 . Introduction to Heritrix, an archival quality web crawler . In Proceedings of the 4th International Web Archiving Workshop (IWAW\u201904) . Gordon Mohr, Michele Kimpton, Micheal Stack, and Igor Ranitovic. 2004. Introduction to Heritrix, an archival quality web crawler. In Proceedings of the 4th International Web Archiving Workshop (IWAW\u201904)."},{"key":"e_1_2_1_35_1","volume-title":"Handbook of Massive Data Sets, James Abello, Panos M","author":"Najork Marc","unstructured":"Marc Najork and Allan Heydon . 2002. High-performance web crawling . In Handbook of Massive Data Sets, James Abello, Panos M . Pardalos, and Mauricio G. C. Resende (Eds.). Kluwer Academic Publishers , 25--45. Marc Najork and Allan Heydon. 2002. High-performance web crawling. In Handbook of Massive Data Sets, James Abello, Panos M. Pardalos, and Mauricio G. C. Resende (Eds.). Kluwer Academic Publishers, 25--45."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000017"},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the 2nd International World Wide Web (Online 8 CDROM review: the international journal of), Anonymous (Ed.)","author":"Pinkerton Brian","year":"1994","unstructured":"Brian Pinkerton . 1994 . Finding what people want: Experiences with the webcrawler . In Proceedings of the 2nd International World Wide Web (Online 8 CDROM review: the international journal of), Anonymous (Ed.) , Vol. 18(6). Learned Information, Medford, NJ. Brian Pinkerton. 1994. Finding what people want: Experiences with the webcrawler. In Proceedings of the 2nd International World Wide Web (Online 8 CDROM review: the international journal of), Anonymous (Ed.), Vol. 18(6). Learned Information, Medford, NJ."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/876875.879017"},{"key":"e_1_2_1_39_1","volume-title":"Fibonacci binning. CoRR abs\/1312.3749","author":"Vigna Sebastiano","year":"2013","unstructured":"Sebastiano Vigna . 2013. Fibonacci binning. CoRR abs\/1312.3749 ( 2013 ). Sebastiano Vigna. 2013. Fibonacci binning. CoRR abs\/1312.3749 (2013)."}],"container-title":["ACM Transactions on the Web"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3160017","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3160017","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:26:37Z","timestamp":1750213597000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3160017"}},"subtitle":["Massive Crawling for the Masses"],"short-title":[],"issued":{"date-parts":[[2018,5,31]]},"references-count":38,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2018,5,31]]}},"alternative-id":["10.1145\/3160017"],"URL":"https:\/\/doi.org\/10.1145\/3160017","relation":{},"ISSN":["1559-1131","1559-114X"],"issn-type":[{"value":"1559-1131","type":"print"},{"value":"1559-114X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,5,31]]},"assertion":[{"value":"2016-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-06-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}