{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T05:37:30Z","timestamp":1781501850645,"version":"3.54.1"},"reference-count":38,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2021,6,28]],"date-time":"2021-06-28T00:00:00Z","timestamp":1624838400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"EU"},{"name":"Spanish MCI\/AEI","award":["TIN2016-76843-C4-1-R and PID2019-104735RB-C41"],"award-info":[{"award-number":["TIN2016-76843-C4-1-R and PID2019-104735RB-C41"]}]},{"DOI":"10.13039\/501100003359","name":"Generalitat Valenciana","doi-asserted-by":"crossref","award":["Prometeo\/2019\/098 (DeepTrust)"],"award-info":[{"award-number":["Prometeo\/2019\/098 (DeepTrust)"]}],"id":[{"id":"10.13039\/501100003359","id-type":"DOI","asserted-by":"crossref"}]},{"name":"TAILOR"},{"name":"EU Horizon 2020 research and innovation programme under GA","award":["952215"],"award-info":[{"award-number":["952215"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2021,6,28]]},"abstract":"<jats:p>The main content of a webpage is often surrounded by other boilerplate elements related to the template, such as menus, advertisements, copyright notices, and comments. For crawlers and indexers, isolating the main content from the template and other noisy information is an essential task, because processing and storing noisy information produce a waste of resources such as bandwidth, storage space, and computing time. Besides, the detection and extraction of the main content is useful in different areas, such as data mining, web summarization, and content adaptation to low resolutions. This work introduces a new technique for main content extraction. In contrast to most techniques, this technique not only extracts text, but also other types of content, such as images, and animations. It is a Document Object Model-based page-level technique, thus it only needs to load one single webpage to extract the main content. As a consequence, it is efficient enough as to be used online (in real-time). We have empirically evaluated the technique using a suite of real heterogeneous benchmarks producing very good results compared with other well-known content extraction techniques.<\/jats:p>","DOI":"10.1145\/3451168","type":"journal-article","created":{"date-parts":[[2021,6,28]],"date-time":"2021-06-28T14:02:52Z","timestamp":1624888972000},"page":"1-105","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Page-Level Main Content Extraction From Heterogeneous Webpages"],"prefix":"10.1145","volume":"15","author":[{"given":"Juli\u00e1n","family":"Alarte","sequence":"first","affiliation":[{"name":"Universitat Polit\u00e8cnica de Val\u00e8ncia, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Josep","family":"Silva","sequence":"additional","affiliation":[{"name":"Universitat Polit\u00e8cnica de Val\u00e8ncia, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,6,28]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/WAINA.2009.60"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-51963-0_32"},{"key":"e_1_2_1_3_1","volume-title":"PSI 2015","author":"Alarte Juli\u00e1n","year":"2015","unstructured":"Juli\u00e1n Alarte , David Insa , Josep Silva , and Salvador Tamarit . 2015 . Site-level web template extraction based on DOM analysis. In Perspectives of System Informatics - 10th International Andrei Ershov Informatics Conference , PSI 2015 , in Memory of Helmut Veith, Kazan and Innopolis, Russia , August 24-27, 2015, Revised Selected Papers(Lecture Notes in Computer Science, Vol. 9609), Manuel Mazzara and Andrei Voronkov (Eds.). Springer, 36\u201349. DOI:https:\/\/doi.org\/10.1007\/978-3-319-41579-6_4 10.1007\/978-3-319-41579-6_4 Juli\u00e1n Alarte, David Insa, Josep Silva, and Salvador Tamarit. 2015. Site-level web template extraction based on DOM analysis. In Perspectives of System Informatics - 10th International Andrei Ershov Informatics Conference, PSI 2015, in Memory of Helmut Veith, Kazan and Innopolis, Russia, August 24-27, 2015, Revised Selected Papers(Lecture Notes in Computer Science, Vol. 9609), Manuel Mazzara and Andrei Voronkov (Eds.). Springer, 36\u201349. DOI:https:\/\/doi.org\/10.1007\/978-3-319-41579-6_4"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the Web Information Systems Engineering, Hakim Hacid, Wojciech Cellary, Hua Wang, Hye-Young Paik","author":"Alarte Julian","unstructured":"Julian Alarte , David Insa , Josep Silva , and Salvador Tamarit . 2018. Main content extraction from heterogeneous webpages . In Proceedings of the Web Information Systems Engineering, Hakim Hacid, Wojciech Cellary, Hua Wang, Hye-Young Paik , and Rui Zhou (Eds.). Springer International Publishing , Cham , 393\u2013407. Julian Alarte, David Insa, Josep Silva, and Salvador Tamarit. 2018. Main content extraction from heterogeneous webpages. In Proceedings of the Web Information Systems Engineering, Hakim Hacid, Wojciech Cellary, Hua Wang, Hye-Young Paik, and Rui Zhou (Eds.). Springer International Publishing, Cham, 393\u2013407."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5120\/869-1219"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/511446.511522"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the International Conference on Language Resources and Evaluation (Marrakech, Morocco). European Language Resources Association, 638\u2013643","author":"Baroni Marco","year":"2008","unstructured":"Marco Baroni , Francis Chantree , Adam Kilgarriff , and Serge Sharoff . 2008 . Cleaneval: A competition for cleaning web pages . In Proceedings of the International Conference on Language Resources and Evaluation (Marrakech, Morocco). European Language Resources Association, 638\u2013643 . Marco Baroni, Francis Chantree, Adam Kilgarriff, and Serge Sharoff. 2008. Cleaneval: A competition for cleaning web pages. In Proceedings of the International Conference on Language Resources and Evaluation (Marrakech, Morocco). European Language Resources Association, 638\u2013643."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACIIDS.2009.71"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2034691.2034720"},{"key":"e_1_2_1_10_1","unstructured":"W3C Consortium. 1997. Document Object Model (DOM). Available from URL: http:\/\/www.w3.org\/DOM\/.  W3C Consortium. 1997. Document Object Model (DOM). Available from URL: http:\/\/www.w3.org\/DOM\/."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1066677.1067065"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/11425274_30"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 4th Web as Corpus Workshop. 47\u201354","author":"Ferraresi Adriano","year":"2008","unstructured":"Adriano Ferraresi , Eros Zanchetta , Marco Baroni , and Silvia Bernardini . 2008 . Introducing and evaluating ukWaC, a very large web-derived corpus of english . In Proceedings of the 4th Web as Corpus Workshop. 47\u201354 . Adriano Ferraresi, Eros Zanchetta, Marco Baroni, and Silvia Bernardini. 2008. Introducing and evaluating ukWaC, a very large web-derived corpus of english. In Proceedings of the 4th Web as Corpus Workshop. 47\u201354."},{"key":"e_1_2_1_14_1","volume-title":"DELOS Workshop: Personalisation and Recommender Systems in Digital Libraries","author":"Finn Aidan","year":"2001","unstructured":"Aidan Finn , Nicholas Kushmerick , and Barry Smyth . 2001 . Fact or fiction: Content classification for digital libraries . In DELOS Workshop: Personalisation and Recommender Systems in Digital Libraries ( Dublin (Ireland)). 1\u20136. http:\/\/www.ercim.org\/publication\/ws-proceedings\/DelNoe02\/AidanFinn.pdf. Aidan Finn, Nicholas Kushmerick, and Barry Smyth. 2001. Fact or fiction: Content classification for digital libraries. In DELOS Workshop: Personalisation and Recommender Systems in Digital Libraries (Dublin (Ireland)). 1\u20136. http:\/\/www.ercim.org\/publication\/ws-proceedings\/DelNoe02\/AidanFinn.pdf."},{"key":"e_1_2_1_15_1","volume-title":"Unfluff - An automatic web page content extractor for node.js!","author":"Geitgey Albert","year":"2014","unstructured":"Albert Geitgey . 2014. Unfluff - An automatic web page content extractor for node.js! ( 2014 ). Albert Geitgey. 2014. Unfluff - An automatic web page content extractor for node.js! (2014)."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1062745.1062763"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/DEXA.2008.43"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/DEXA.2008.43"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jlap.2013.01.002"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526909"},{"key":"e_1_2_1_21_1","volume-title":"Boilerpipe\u2013boilerplate removal and fulltext extraction from HTML pages. Google Code","author":"\u00a0al Christian Kohlsch\u00fctter","year":"2010","unstructured":"Christian Kohlsch\u00fctter et \u00a0al . 2010. Boilerpipe\u2013boilerplate removal and fulltext extraction from HTML pages. Google Code ( 2010 ). Christian Kohlsch\u00fctter et\u00a0al. 2010. Boilerpipe\u2013boilerplate removal and fulltext extraction from HTML pages. Google Code (2010)."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1718487.1718542"},{"key":"e_1_2_1_23_1","volume-title":"Wee Keong Ng, and Aixin Sun","author":"Li Zhao","year":"2005","unstructured":"Zhao Li , Wee Keong Ng, and Aixin Sun . 2005 . Web data extraction based on structural similarity. Knowledge and Information Systems 8, 4 (01 Nov 2005), 438\u2013461. DOI:https:\/\/doi.org\/10.1007\/s10115-004-0188-z 10.1007\/s10115-004-0188-z Zhao Li, Wee Keong Ng, and Aixin Sun. 2005. Web data extraction based on structural similarity. Knowledge and Information Systems 8, 4 (01 Nov 2005), 438\u2013461. DOI:https:\/\/doi.org\/10.1007\/s10115-004-0188-z"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.5555\/1215470"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1083356.1083384"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1526709.1526840"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/544220.544228"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcss.2011.10.012"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-013-0687-x"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 4th Web as Corpus Workshop-Can we beat Google. 12\u201317","author":"Spousta Miroslav","year":"2008","unstructured":"Miroslav Spousta , Michal Marek , and Pavel Pecina . 2008 . Victor: The web-page cleaning tool . In Proceedings of the 4th Web as Corpus Workshop-Can we beat Google. 12\u201317 . Miroslav Spousta, Michal Marek, and Pavel Pecina. 2008. Victor: The web-page cleaning tool. In Proceedings of the 4th Web as Corpus Workshop-Can we beat Google. 12\u201317."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009952"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11280-009-0059-3"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183654"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-76941-7_13"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1367497.1367749"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772789"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741659"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/956750.956785"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3451168","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3451168","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T17:49:25Z","timestamp":1750268965000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3451168"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,28]]},"references-count":38,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2021,6,28]]}},"alternative-id":["10.1145\/3451168"],"URL":"https:\/\/doi.org\/10.1145\/3451168","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,28]]},"assertion":[{"value":"2020-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-06-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}