{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,4]],"date-time":"2026-09-04T21:12:53Z","timestamp":1788556373408,"version":"build-2803163510"},"publisher-location":"New York, NY, USA","reference-count":38,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,8,8]],"date-time":"2026-08-08T00:00:00Z","timestamp":1786147200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,8,9]]},"DOI":"10.1145\/3770855.3818161","type":"proceedings-article","created":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T17:15:47Z","timestamp":1786036547000},"page":"4333-4344","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["SEDD: Scalable and Efficient Dataset Deduplication with GPUs"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-1530-8845","authenticated-orcid":false,"given":"Youngjun","family":"Son","sequence":"first","affiliation":[{"name":"Graduate School of Data Science, Seoul National University, Seoul, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-9042-0435","authenticated-orcid":false,"given":"Chaewon","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Seoul National University, Seoul, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4638-8170","authenticated-orcid":false,"given":"Jaejin","family":"Lee","sequence":"additional","affiliation":[{"name":"Graduate School of Data Science, Seoul National University, Seoul, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,8,8]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al.","author":"Albalak Alon","year":"2024","unstructured":"Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al., 2024. A survey on data selection for language models. arXiv preprint arXiv:2402.16827 (2024)."},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359591.3359735"},{"key":"e_1_3_2_2_3_1","volume-title":"URL: https:\/\/github.com\/aappleby\/smhasher\/blob\/master\/src\/MurmurHash3. cpp","author":"Appleby Austin","year":"2012","unstructured":"Austin Appleby. 2012. MurmurHash3, 2012. URL: https:\/\/github.com\/aappleby\/smhasher\/blob\/master\/src\/MurmurHash3. cpp (2012)."},{"key":"e_1_3_2_2_4_1","first-page":"21","volume-title":"Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171)","author":"Broder Andrei Z","year":"1997","unstructured":"Andrei Z Broder. 1997. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171). IEEE, 21-29."},{"key":"e_1_3_2_2_5_1","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems Vol. 33 (2020) 1877-1901."},{"key":"e_1_3_2_2_6_1","volume-title":"Professional CUDA c programming","author":"Cheng John","unstructured":"John Cheng, Max Grossman, and Ty McKercher. 2014. Professional CUDA c programming. John Wiley & Sons."},{"key":"e_1_3_2_2_7_1","volume-title":"Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457","author":"Clark Peter","year":"2018","unstructured":"Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 (2018)."},{"key":"e_1_3_2_2_8_1","unstructured":"Yann Collet. 2012. xxHash. GitHub repository. https:\/\/github.com\/Cyan4973\/xxHash."},{"key":"e_1_3_2_2_9_1","unstructured":"Together Computer. 2023. RedPajama: an Open Dataset for Training Large Language Models. https:\/\/github.com\/togethercomputer\/RedPajama-Data"},{"key":"e_1_3_2_2_10_1","unstructured":"Dask. 2024. Dask: a Python library for parallel and distributed computing. https:\/\/github.com\/dask\/dask"},{"key":"e_1_3_2_2_11_1","unstructured":"Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Amy Yang Angela Fan et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)."},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"crossref","unstructured":"D. Eastlake 3rd and P. Jones. 2001. US Secure Hash Algorithm 1 (SHA1). RFC 3174 (Informational). http:\/\/www.ietf.org\/rfc\/rfc3174.txt Updated by RFC 4634.","DOI":"10.17487\/rfc3174"},{"key":"e_1_3_2_2_13_1","unstructured":"Yanai Elazar Akshita Bhagia Ian Magnusson Abhilasha Ravichander Dustin Schwenk Alane Suhr Pete Walsh Dirk Groeneveld Luca Soldaini Sameer Singh Hanna Hajishirzi Noah A. Smith and Jesse Dodge. 2024. What's In My Big Data? arXiv:2310.20707 [cs.CL] https:\/\/arxiv.org\/abs\/2310.20707"},{"key":"e_1_3_2_2_14_1","volume-title":"Accelerating Large-Scale Inference with Anisotropic Vector Quantization. In International Conference on Machine Learning. https:\/\/arxiv.org\/abs\/1908","author":"Guo Ruiqi","year":"2020","unstructured":"Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020. Accelerating Large-Scale Inference with Anisotropic Vector Quantization. In International Conference on Machine Learning. https:\/\/arxiv.org\/abs\/1908.10396"},{"key":"e_1_3_2_2_15_1","volume-title":"Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300","author":"Hendrycks Dan","year":"2020","unstructured":"Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 (2020)."},{"key":"e_1_3_2_2_16_1","volume-title":"Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.","author":"Hoffmann Jordan","year":"2022","unstructured":"Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al., 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022)."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/276698.276876"},{"key":"e_1_3_2_2_18_1","unstructured":"Joseph Jennings Mostofa Patwary Sandeep Subramanian Shrimai Prabhumoye Ayush Dattagupta Vibhu Jawa Jiwei Liu Ryan Wolf Sarah Yurick and Varun Singh. 2024. NeMo-Curator: a toolkit for data curation. https:\/\/github.com\/NVIDIA\/NeMo-Curator"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TBDATA.2019.2921572"},{"key":"e_1_3_2_2_20_1","volume-title":"Efficient randomized pattern-matching algorithms. IBM journal of research and development","author":"Karp Richard M","year":"1987","unstructured":"Richard M Karp and Michael O Rabin. 1987. Efficient randomized pattern-matching algorithms. IBM journal of research and development, Vol. 31, 2 (1987), 249-260."},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.577"},{"key":"e_1_3_2_2_22_1","volume-title":"Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, et al.","author":"Magnusson Ian","year":"2023","unstructured":"Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, et al., 2023. Paloma: A benchmark for evaluating language model fit. arXiv preprint arXiv:2312.10523 (2023)."},{"key":"e_1_3_2_2_23_1","volume-title":"Nemotron-cc-math: A 133 billion-token-scale high quality math pretraining dataset. arXiv preprint arXiv:2508.15096","author":"Mahabadi Rabeeh Karimi","year":"2025","unstructured":"Rabeeh Karimi Mahabadi, Sanjeev Satheesh, Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2025. Nemotron-cc-math: A 133 billion-token-scale high quality math pretraining dataset. arXiv preprint arXiv:2508.15096 (2025)."},{"key":"e_1_3_2_2_24_1","unstructured":"Geoff Pike. 2014. Introducing FarmHash. Google Open Source Blog. https:\/\/opensource.googleblog.com\/2014\/03\/introducing-farmhash.html."},{"key":"e_1_3_2_2_25_1","unstructured":"Jack W Rae Sebastian Borgeaud Trevor Cai Katie Millican Jordan Hoffmann Francis Song John Aslanides Sarah Henderson Roman Ring Susannah Young et al. 2021. Scaling language models: Methods analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 (2021)."},{"key":"e_1_3_2_2_26_1","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, Vol. 21, 140 (2020), 1-67.","journal-title":"Journal of machine learning research"},{"key":"e_1_3_2_2_27_1","unstructured":"RAPIDS. 2024. cuDF - GPU DataFrames. https:\/\/github.com\/rapidsai\/cudf"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"crossref","unstructured":"Ronald Rivest. 1992. RFC1321: The MD5 message-digest algorithm.","DOI":"10.17487\/rfc1321"},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474381"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/322217.322225"},{"key":"e_1_3_2_2_31_1","volume-title":"Slimpajama-dc: Understanding data combinations for llm training. arXiv preprint arXiv:2309.10818","author":"Shen Zhiqiang","year":"2023","unstructured":"Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. 2023. Slimpajama-dc: Understanding data combinations for llm training. arXiv preprint arXiv:2309.10818 (2023)."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1419"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2348"},{"key":"e_1_3_2_2_34_1","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave and Guillaume Lample. 2023a. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL]"},{"key":"e_1_3_2_2_35_1","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)."},{"key":"e_1_3_2_2_36_1","volume-title":"Attention is all you need. Advances in neural information processing systems","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, Vol. 30 (2017). https:\/\/arxiv.org\/abs\/1706.03762"},{"key":"e_1_3_2_2_37_1","volume-title":"Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830","author":"Zellers Rowan","year":"2019","unstructured":"Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019a. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 (2019)."},{"key":"e_1_3_2_2_38_1","volume-title":"Defending against neural fake news. Advances in neural information processing systems","author":"Zellers Rowan","year":"2019","unstructured":"Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019b. Defending against neural fake news. Advances in neural information processing systems, Vol. 32 (2019)."}],"event":{"name":"KDD '26: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining","location":"Jeju Island Republic of Korea","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"]},"container-title":["Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3770855.3818161","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,4]],"date-time":"2026-09-04T20:24:33Z","timestamp":1788553473000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3770855.3818161"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,8]]},"references-count":38,"alternative-id":["10.1145\/3770855.3818161","10.1145\/3770855"],"URL":"https:\/\/doi.org\/10.1145\/3770855.3818161","relation":{},"subject":[],"published":{"date-parts":[[2026,8,8]]},"assertion":[{"value":"2026-08-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}