{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T16:42:05Z","timestamp":1778776925250,"version":"3.51.4"},"reference-count":40,"publisher":"Cambridge University Press (CUP)","issue":"1","license":[{"start":{"date-parts":[[2023,2,21]],"date-time":"2023-02-21T00:00:00Z","timestamp":1676937600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2024,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The availability of large web-based corpora has led to significant advances in a wide range of technologies, including massive retrieval systems or deep neural networks. However, leveraging this data is challenging, since web content is plagued by the so-called boilerplate: ads, incomplete or noisy text and rests of the navigation structure, such as menus or navigation bars. In this work, we present a novel and efficient approach to extract useful and well-formed content from web-scraped data. Our approach takes advantage of Language Models and their implicit knowledge about correctly formed text, and we demonstrate here that perplexity is a valuable artefact that can contribute in terms of effectiveness and efficiency. As a matter of fact, the removal of noisy parts leads to lighter AI or search solutions that are effective and entail important reductions in resources spent. We exemplify here the usefulness of our method with two downstream tasks, search and classification, and a cleaning task. We also provide a Python package with pre-trained models and a web demo demonstrating the capabilities of our approach.<\/jats:p>","DOI":"10.1017\/s1351324923000049","type":"journal-article","created":{"date-parts":[[2023,2,21]],"date-time":"2023-02-21T09:21:04Z","timestamp":1676971264000},"page":"132-149","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":8,"title":["An unsupervised perplexity-based method for boilerplate removal"],"prefix":"10.1017","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6560-9832","authenticated-orcid":false,"given":"Marcos","family":"Fern\u00e1ndez-Pichel","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2731-079X","authenticated-orcid":false,"given":"Manuel","family":"Prada-Corral","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8823-7501","authenticated-orcid":false,"given":"David E.","family":"Losada","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9505-6493","authenticated-orcid":false,"given":"Juan C.","family":"Pichel","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5819-2469","authenticated-orcid":false,"given":"Pablo","family":"Gamallo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2023,2,21]]},"reference":[{"key":"S1351324923000049_ref8","first-page":"146","article-title":"I-divergence geometry of probability distributions and minimization problems","author":"Csisz\u00e1r","year":"1975","journal-title":"The Annals of Probability"},{"key":"S1351324923000049_ref3","unstructured":"Bauer, D. , Degen, J. , Deng, X. , Herger, P. , Gasthaus, J. , Giesbrecht, E. , Jansen, L. , Kalina, C. , Kr\u00e4ger, T. , M\u00e4rtin, R. , et al. (2007). Fiasco: Filtering the internet by automatic subtree classification, osnabruck. In Building and Exploring Web Corpora: Proceedings of the 3rd Web as Corpus Workshop, Incorporating Cleaneval, volume 4, pp. 111\u2013121. Presses Univ. de Louvain."},{"key":"S1351324923000049_ref14","first-page":"1","article-title":"Towards the systematic reporting of the energy and carbon footprints of machine learning","volume":"21","author":"Henderson","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"S1351324923000049_ref12","doi-asserted-by":"publisher","DOI":"10.1016\/j.physa.2017.05.011"},{"key":"S1351324923000049_ref10","unstructured":"Duhart, C. , Dublon, G. , Mayton, B. , Davenport, G. and Paradiso, J.A. (2019). Deep learning for wildlife conservation and restoration efforts. In 36th International Conference on Machine Learning, Long Beach, volume 5."},{"key":"S1351324923000049_ref18","unstructured":"Lacoste, A. , Luccioni, A. , Schmidt, V. and Dandres, T. (2019). Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700."},{"key":"S1351324923000049_ref28","doi-asserted-by":"publisher","DOI":"10.1145\/3381831"},{"key":"S1351324923000049_ref25","doi-asserted-by":"publisher","DOI":"10.1002\/asi.24203"},{"key":"S1351324923000049_ref39","doi-asserted-by":"publisher","DOI":"10.1145\/3239571"},{"key":"S1351324923000049_ref34","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183654"},{"key":"S1351324923000049_ref9","unstructured":"Devlin, J. , Chang, M.-W. , Lee, K. and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805."},{"key":"S1351324923000049_ref32","doi-asserted-by":"publisher","DOI":"10.1145\/584931.584952"},{"key":"S1351324923000049_ref11","unstructured":"Finn, A. , Kushmerick, N. and Smyth, B. (2001). Fact or fiction: Content classification for digital libraries. In DELOS."},{"key":"S1351324923000049_ref36","unstructured":"Wang, L.L. , Lo, K. , Chandrasekhar, Y. , Reas, R. , Yang, J. , Burdick, D. , Eide, D. , Funk, K. , Katsis, Y. , Kinney, R.M. , Li, Y. , Liu, Z. , Merrill, W. , Mooney, P. , Murdick, D.A. , Rishi, D. , Sheehan, J. , Shen, Z. , Stilson, B. , Wade, A.D. , Wang, K. , Wang, N.X.R. , Wilhelm, C. , Xie, B. , Raymond, D.M. , Weld, D.S. , Etzioni, O. and Kohlmeier, S. (2020). CORD-19: The COVID-19 open research dataset. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, Online. Association for Computational Linguistics."},{"key":"S1351324923000049_ref21","unstructured":"Liu, Y. , Ott, M. , Goyal, N. , Du, J. , Joshi, M. , Chen, D. , Levy, O. , Lewis, M. , Zettlemoyer, L. and Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs\/1907.11692."},{"key":"S1351324923000049_ref20","doi-asserted-by":"publisher","DOI":"10.1145\/3366424.3383547"},{"key":"S1351324923000049_ref7","volume-title":"Search Engines: Information Retrieval in Practice","volume":"520","author":"Croft","year":"2010"},{"key":"S1351324923000049_ref22","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1007\/s10791-007-9040-x","article-title":"An analysis on document length retrieval trends in language modeling smoothing","volume":"11","author":"Losada","year":"2008","journal-title":"Information Retrieval"},{"key":"S1351324923000049_ref31","unstructured":"Spousta, M. , Marek, M. and Pecina, P. (2008). Victor: the web-page cleaning tool. In 4th Web as Corpus Workshop (WAC4)-Can We Beat Google, pp. 12\u201317."},{"key":"S1351324923000049_ref33","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"key":"S1351324923000049_ref37","unstructured":"Wenzek, G. , Lachaux, M.-A. , Conneau, A. , Chaudhary, V. , Guzm\u00e1n, F. , Joulin, A. and Grave, E. (2019). Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359."},{"key":"S1351324923000049_ref5","unstructured":"Collins-Thompson, K. , Bennett, P.N. , Diaz, F. , Clarke, C. and Voorhees, E.M. (2013). Overview of the TREC 2013 web track. In Proceedings of the 22nd Text REtrieval Conference (TREC \u201913), Gaithesburg, Maryland, USA. National Institute of Standards and Technology (NIST)."},{"key":"S1351324923000049_ref17","doi-asserted-by":"crossref","unstructured":"Kohlsch\u00fctter, C. , Fankhauser, P. and Nejdl, W. (2010). Boilerplate detection using shallow text features. In Proceedings of the Third ACM International Conference on Web Search and Data Mining, New York, USA, pp. 441\u2013450.","DOI":"10.1145\/1718487.1718542"},{"key":"S1351324923000049_ref16","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526909"},{"key":"S1351324923000049_ref38","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324919000585"},{"key":"S1351324923000049_ref24","doi-asserted-by":"publisher","DOI":"10.1145\/3412841.3441945"},{"key":"S1351324923000049_ref13","unstructured":"Gonz\u00e1lez, M. (2015). An analysis of twitter corpora and the differences between formal and colloquial tweets. In Proceedings of the Tweet Translation Workshop 2015, pp. 1\u20137, Alicante, Spain."},{"key":"S1351324923000049_ref15","first-page":"7","article-title":"Preprocessing techniques for text mining","volume":"5","author":"Kannan","year":"2014","journal-title":"International Journal of Computer Science and Communication Networks"},{"key":"S1351324923000049_ref23","doi-asserted-by":"publisher","DOI":"10.1016\/j.trd.2017.07.012"},{"key":"S1351324923000049_ref26","unstructured":"Pomik\u00e1lek, J. (2011). Removing boilerplate and duplicate content from web corpora. Disertacn pr\u00e1ce, Masarykova univerzita, Fakulta informatiky."},{"key":"S1351324923000049_ref29","unstructured":"Sennrich, R. (2012). Perplexity minimization for translation model domain adaptation in statistical machine translation. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL \u201912, pp. 539\u2013549, Association for Computational Linguistics: Stroudsburg, PA, USA."},{"key":"S1351324923000049_ref27","doi-asserted-by":"publisher","DOI":"10.1145\/3130348.3130368"},{"key":"S1351324923000049_ref4","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324919000378"},{"key":"S1351324923000049_ref40","doi-asserted-by":"publisher","DOI":"10.1145\/956750.956785"},{"key":"S1351324923000049_ref2","unstructured":"Baroni, M. , Chantree, F. , Kilgarriff, A. and Sharoff, S. (2008). Cleaneval: A competition for cleaning web pages. In Lrec, Marrakech."},{"key":"S1351324923000049_ref19","unstructured":"Lee, N. , Bang, Y. , Madotto, A. and Fung, P. (2020). Misinformation has high perplexity. arXiv preprint arXiv:2006.04666."},{"key":"S1351324923000049_ref6","volume-title":"British National Corpus","author":"Consortium","year":"2007"},{"key":"S1351324923000049_ref35","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-76941-7_13"},{"key":"S1351324923000049_ref30","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324910000252"},{"key":"S1351324923000049_ref1","unstructured":"Abualsaud, M. , Lioma, C. , Maistro, M. , Smucker, M.D. and Zuccon, G. (2019). Overview of the TREC 2019 decision track. In Proceedings of the 28th Text REtrieval Conference, (TREC \u201919), Gaithesburg, Maryland, USA. National Institute of Standards and Technology (NIST)."}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324923000049","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,25]],"date-time":"2024-04-25T12:50:32Z","timestamp":1714049432000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324923000049\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,21]]},"references-count":40,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,1]]}},"alternative-id":["S1351324923000049"],"URL":"https:\/\/doi.org\/10.1017\/s1351324923000049","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,2,21]]},"assertion":[{"value":"\u00a9 The Author(s), 2023. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (https:\/\/creativecommons.org\/licenses\/by\/4.0\/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.","name":"license","label":"License","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}