{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T17:28:51Z","timestamp":1786728531612,"version":"build-2736575974"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T00:00:00Z","timestamp":1786665600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Web"],"published-print":{"date-parts":[[2026,8,31]]},"abstract":"<jats:p>The history-based content of Wikipedia has long served as a valuable resource for a wide range of research efforts in Natural Language Processing (NLP). However, due to Wikipedia\u2019s dynamic and ever-evolving nature, conducting large-scale temporal analyses requires the efficient retrieval and processing of historical article versions. One of the major challenges in this domain arises from the lack of robust and scalable tools capable of managing the vast volume of revision data that Wikipedia provides. This limitation often acts as a significant bottleneck in both data management and downstream applications.<\/jats:p>\n                  <jats:p>\n                    In this work,\n                    <jats:xref ref-type=\"fn\">\n                      <jats:sup>1<\/jats:sup>\n                    <\/jats:xref>\n                    we present a comprehensive analysis of online algorithms designed to enable efficient compression and retrieval of Wikipedia\u2019s revision history. Our methods are supported by theoretical guarantees, demonstrating their ability to achieve high compression rates while maintaining favorable time\u2013space tradeoffs. The analysis emphasizes the practicality and scalability of our approach, particularly in environments constrained by memory or processing time.\n                  <\/jats:p>\n                  <jats:p>To operationalize these theoretical findings, we develop and release WikECD (Wikipedia Efficient Compression and Decompression), an open-source Python library that implements our proposed algorithms for large-scale Wikipedia revision compression, retrieval, and analysis. Experimental evaluations on sampled Wikipedia articles show that the method can reduce dataset size to as little as 6% of its original volume, achieving up to 94% compression while preserving reconstructive accuracy.<\/jats:p>\n                  <jats:p>To the best of our knowledge, this work constitutes the first detailed and rigorous study to analyze and quantify the compression of Wikipedia\u2019s complete revision history dataset, paving the way for scalable and reproducible NLP research based on historical web data.<\/jats:p>","DOI":"10.1145\/3787449","type":"journal-article","created":{"date-parts":[[2026,1,26]],"date-time":"2026-01-26T12:01:02Z","timestamp":1769428862000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Scalable Methods for Storing and Retrieving Wikipedia Revision Histories for Large-Scale Analysis"],"prefix":"10.1145","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2392-0198","authenticated-orcid":false,"given":"Amit","family":"Verma","sequence":"first","affiliation":[{"name":"Indian Institute of Technology Ropar","place":["Rupnagar, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-0786-4100","authenticated-orcid":false,"given":"Simran","family":"Setia","sequence":"additional","affiliation":[{"name":"Thapar Institute of Engineering and Technology (Deemed to be University)","place":["Indra Puri Colony, India"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7311-3693","authenticated-orcid":false,"given":"Gianluca","family":"Demartini","sequence":"additional","affiliation":[{"name":"The University of Queensland","place":["Brisbane, Australia"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,8,14]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"crossref","unstructured":"Ofer Arazy Oded Nov Raymond Patterson and Lisa Yeo. 2011. Information quality in wikipedia: The effects of group composition and task conflict. Journal of Management Information Systems 27 4 (2011) 71\u201398.","DOI":"10.2753\/MIS0742-1222270403"},{"key":"e_1_3_3_3_2","doi-asserted-by":"crossref","unstructured":"Alberto Berenguer Jose-Norberto Maz\u00f3n and David Tom\u00e1s. 2024. Word embeddings for retrieving tabular data from research publications. Machine Learning 113 4 (2024) 2227\u20132248.","DOI":"10.1007\/s10994-023-06472-0"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/1367497.1367673"},{"key":"e_1_3_3_5_2","doi-asserted-by":"crossref","unstructured":"Piotr Bojanowski Edouard Grave Armand Joulin and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5 (2017) 135\u2013146.","DOI":"10.1162\/tacl_a_00051"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/VAST.2010.5652896"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-6111"},{"key":"e_1_3_3_8_2","unstructured":"Anamika Chhabra and SRS Iyengar. 2017. How does knowledge come by? arXiv:1705.06946. Retrieved from https:\/\/arxiv.org\/abs\/1705.06946"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/1772938.1772942"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v13i01.3257"},{"key":"e_1_3_3_11_2","first-page":"97","volume-title":"Proceedings of the ACL","author":"Ferschke Oliver","year":"2011","unstructured":"Oliver Ferschke, Torsten Zesch, and Iryna Gurevych. 2011. Wikipedia revision toolkit: Efficiently accessing wikipedia\u2019s edit history. In Proceedings of the ACL. Association for Computational Linguistics, 97\u2013102."},{"key":"e_1_3_3_12_2","first-page":"1301","volume-title":"Proceedings of the AAAI","author":"Gabrilovich Evgeniy","year":"2006","unstructured":"Evgeniy Gabrilovich and Shaul Markovitch. 2006. Overcoming the brittleness bottleneck using Wikipedia: Enhancing text categorization with encyclopedic knowledge. In Proceedings of the AAAI. 1301\u20131306."},{"key":"e_1_3_3_13_2","first-page":"1606","volume-title":"Proceedings of the IJcAI","author":"Gabrilovich Evgeniy","year":"2007","unstructured":"Evgeniy Gabrilovich, Shaul Markovitch, et\u00a0al. 2007. Computing semantic relatedness using wikipedia-based explicit semantic analysis.. In Proceedings of the IJcAI. 1606\u20131611."},{"key":"e_1_3_3_14_2","doi-asserted-by":"crossref","unstructured":"Jana Gallus and Sudeep Bhatia. 2020. Gender power and emotions in the collaborative production of knowledge: A large-scale analysis of wikipedia editor conversations. Organizational Behavior and Human Decision Processes 160 (2020) 115\u2013130.","DOI":"10.1016\/j.obhdp.2020.02.003"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2022\/574"},{"key":"e_1_3_3_16_2","volume-title":"MediaWiki Utilities","author":"Halfaker Aaron","year":"2020","unstructured":"Aaron Halfaker. 2020. MediaWiki Utilities. Retrieved April 4, 2024 from https:\/\/pythonhosted.org\/mediawiki-utilities\/"},{"key":"e_1_3_3_17_2","volume-title":"MediaWiki XML Processing","author":"Halfaker Aaron","year":"2020","unstructured":"Aaron Halfaker. 2020. MediaWiki XML Processing. Retrieved September 5, 2020 from https:\/\/pythonhosted.org\/mwxml\/"},{"key":"e_1_3_3_18_2","doi-asserted-by":"crossref","unstructured":"Paul Heckel. 1978. A technique for isolating differences between files. Communications of the ACM 21 4 (1978) 264\u2013268.","DOI":"10.1145\/359460.359467"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/2467696.2467699"},{"key":"e_1_3_3_20_2","volume-title":"An Algorithm for Differential File Comparison","author":"Hunt James Wayne","year":"1976","unstructured":"James Wayne Hunt and M. Douglas MacIlroy. 1976. An Algorithm for Differential File Comparison. Bell Laboratories Murray Hill."},{"key":"e_1_3_3_21_2","unstructured":"Abraham Israeli David Jurgens and Daniel Romero. 2024. A test of time: Predicting the sustainable success of online collaboration in wikipedia. arXiv:2410.19150. Retrieved from https:\/\/arxiv.org\/abs\/2410.19150"},{"key":"e_1_3_3_22_2","doi-asserted-by":"crossref","unstructured":"Joel Jang Seonghyeon Ye Changho Lee Sohee Yang Joongbo Shin Janghoon Han Gyeonghun Kim and Minjoon Seo. 2022. Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models. arXiv:2204.14211. Retrieved from https:\/\/arxiv.org\/abs\/2204.14211","DOI":"10.18653\/v1\/2022.emnlp-main.418"},{"key":"e_1_3_3_23_2","first-page":"162","volume-title":"Proceedings of the International AAAI Conference on Web and Social Media","author":"Jurgens David","year":"2012","unstructured":"David Jurgens and Tsai-Ching Lu. 2012. Temporal motifs reveal the dynamics of editor interactions in wikipedia. In Proceedings of the International AAAI Conference on Web and Social Media. 162\u2013169."},{"key":"e_1_3_3_24_2","doi-asserted-by":"crossref","unstructured":"Brian C. Keegan. 2019. The dynamics of peer-produced political information during the 2016 US presidential campaign. Proceedings of the ACM on Human-Computer Interaction 3 CSCW (2019) 1\u201320.","DOI":"10.1145\/3359135"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/1460563.1460572"},{"key":"e_1_3_3_26_2","volume-title":"File System Support for Delta Compression","author":"MacDonald Josh","year":"2000","unstructured":"Josh MacDonald. 2000. File System Support for Delta Compression. Ph. D. Dissertation. Masters thesis. Department of Electrical Engineering and Computer Science \u2026."},{"key":"e_1_3_3_27_2","unstructured":"Sayed-Amir Marashi Seyed Mohammad Amin Hosseini-Nami Khadijeh Alishah Mahdieh Hadi Ali Karimi Saeedeh Hosseinian Rouhallah Ramezanifard Reihaneh Sadat Mirhassani Zhaleh Hosseini and Zahra Shojaie. 2013. Impact of wikipedia on citation trends. EXCLI Journal 12 (2013) 15."},{"key":"e_1_3_3_28_2","unstructured":"Mediawiki. 2024. Size of Wikipedia. Retrieved October 10 2023 from https:\/\/en.wikipedia.org\/wiki\/Wikipedia:Size_of_Wikipedia"},{"key":"e_1_3_3_29_2","unstructured":"Mediawiki. 2024. Size of Wikipedia. Retrieved October 10 2023 from https:\/\/en.wikipedia.org\/wiki\/Special:MostRevisionsRetrieved October 10 2023 from"},{"key":"e_1_3_3_30_2","doi-asserted-by":"crossref","unstructured":"M\u00e1rton Mesty\u00e1n Taha Yasseri and J\u00e1nos Kert\u00e9sz. 2013. Early prediction of movie box office success based on wikipedia activity big data. PloS One 8 8 (2013) e71226.","DOI":"10.1371\/journal.pone.0071226"},{"key":"e_1_3_3_31_2","unstructured":"Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv:1301.3781. Retrieved from https:\/\/arxiv.org\/abs\/1301.3781"},{"key":"e_1_3_3_32_2","doi-asserted-by":"crossref","unstructured":"David Milne and Ian H. Witten. 2013. An open-source toolkit for mining wikipedia. Artificial Intelligence 194 (2013) 222\u2013239.","DOI":"10.1016\/j.artint.2012.06.007"},{"key":"e_1_3_3_33_2","doi-asserted-by":"crossref","unstructured":"Eugene W. Myers. 1986. AnO (ND) difference algorithm and its variations. Algorithmica 1 1-4 (1986) 251\u2013266.","DOI":"10.1007\/BF01840446"},{"key":"e_1_3_3_34_2","unstructured":"Neil Fraser. 2006. Fuzzy Patch. Retrieved April 1 2024 from https:\/\/neil.fraser.name\/writing\/patch\/"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3450136"},{"key":"e_1_3_3_36_2","doi-asserted-by":"crossref","unstructured":"Adam Roberts Colin Raffel and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? arXiv:2002.08910. Retrieved from https:\/\/arxiv.org\/abs\/2002.08910","DOI":"10.18653\/v1\/2020.emnlp-main.437"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2641580.2641615"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-88113-9_14"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/2487788.2488049"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i17.29869"},{"key":"e_1_3_3_41_2","unstructured":"Tuan Tran and Tu Ngoc Nguyen. 2017. Hedera: Scalable indexing and exploring entities in wikipedia revision history. arXiv:1701.03937. Retrieved from https:\/\/arxiv.org\/abs\/1701.03937"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998232"},{"key":"e_1_3_3_43_2","unstructured":"Amit Arjun Verma SRS Iyengar Simran Setia and Neeru Dubey. [n. d.]. KDAP: An open source toolkit to accelerate knowledge building research. ([n. d.])."},{"key":"e_1_3_3_44_2","doi-asserted-by":"crossref","unstructured":"Amit Arjun Verma SRS Iyengar Simran Setia and Neeru Dubey. 2021. An open source toolkit to parse and analyze online crowdsourced portals. (2021).","DOI":"10.21203\/rs.3.rs-154658\/v1"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174140"},{"key":"e_1_3_3_46_2","unstructured":"Wikipedia contributors. 2024. Knapsack problem \u2014 Wikipedia The Free Encyclopedia. Retrieved April 18 2021 from https:\/\/en.wikipedia.org\/w\/index.php?title=Knapsack_problem&oldid=1014657879"},{"key":"e_1_3_3_47_2","unstructured":"Wikipedia contributors. 2024. Wikipedia:Pruning article revisions \u2014 Wikipedia The Free Encyclopedia. Retrieved April 4 2024 from https:\/\/en.wikipedia.org\/wiki\/Wikipedia:Pruning_article_revisions"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v29i1.9496"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1213"},{"key":"e_1_3_3_50_2","first-page":"446","volume-title":"Proceedings of the ICWSM","author":"Yang Diyi","year":"2016","unstructured":"Diyi Yang, Aaron Halfaker, Robert E. Kraut, and Eduard H. Hovy. 2016. Who did what: Editor role identification in wikipedia. In Proceedings of the ICWSM. 446\u2013455."},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.63317\/273wtukpj6kd"},{"key":"e_1_3_3_52_2","doi-asserted-by":"crossref","unstructured":"Ark Fangzhou Zhang Danielle Livneh Ceren Budak Lionel P. Robert Jr and Daniel M. Romero. 2017. Crowd development: The interplay between crowd evaluation and collaborative dynamics in wikipedia. Proceedings of the ACM on Human-Computer Interaction 1 CSCW (2017) 1\u201321.","DOI":"10.1145\/3134754"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557213"}],"container-title":["ACM Transactions on the Web"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3787449","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,14]],"date-time":"2026-08-14T16:28:20Z","timestamp":1786724900000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3787449"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,14]]},"references-count":52,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,8,31]]}},"alternative-id":["10.1145\/3787449"],"URL":"https:\/\/doi.org\/10.1145\/3787449","relation":{},"ISSN":["1559-1131","1559-114X"],"issn-type":[{"value":"1559-1131","type":"print"},{"value":"1559-114X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,14]]},"assertion":[{"value":"2025-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-20","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-14","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}