{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T12:17:31Z","timestamp":1775737051976,"version":"3.50.1"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2020,6,13]],"date-time":"2020-06-13T00:00:00Z","timestamp":1592006400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Data and Information Quality"],"published-print":{"date-parts":[[2020,9,30]]},"abstract":"<jats:p>Data errors represent a major issue in most application workflows. Before any important task can take place, a certain data quality has to be guaranteed by eliminating a number of different errors that may appear in data. Typically, most of these errors are fixed with data preparation methods, such as whitespace removal. However, the particular error of duplicate records, where multiple records refer to the same entity, is usually eliminated independently with specialized techniques. Our work is the first to bring these two areas together by applying data preparation operations under a systematic approach prior to performing duplicate detection.<\/jats:p>\n          <jats:p>Our process workflow can be summarized as follows: It begins with the user providing as input a sample of the gold standard, the actual dataset, and optionally some constraints to domain-specific data preparations, such as address normalization. The preparation selection operates in two consecutive phases. First, to vastly reduce the search space of ineffective data preparations, decisions are made based on the improvement or worsening of pair similarities. Second, using the remaining data preparations an iterative leave-one-out classification process removes preparations one by one and determines the redundant preparations based on the achieved area under the precision-recall curve (AUC-PR). Using this workflow, we manage to improve the results of duplicate detection up to 19% in AUC-PR.<\/jats:p>","DOI":"10.1145\/3377878","type":"journal-article","created":{"date-parts":[[2020,6,13]],"date-time":"2020-06-13T22:18:11Z","timestamp":1592086691000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":15,"title":["Data Preparation for Duplicate Detection"],"prefix":"10.1145","volume":"12","author":[{"given":"Ioannis","family":"Koumarelas","sequence":"first","affiliation":[{"name":"Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lan","family":"Jiang","sequence":"additional","affiliation":[{"name":"Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Felix","family":"Naumann","sequence":"additional","affiliation":[{"name":"Hasso Plattner Institute, University of Potsdam, 14482 Potsdam, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,6,13]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994518"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the Conference on Innovative Data Systems Research (CIDR\u201915)","author":"Abedjan Ziawasch","year":"2015","unstructured":"Ziawasch Abedjan, John Morcos, Michael N. Gubanov, Ihab F. Ilyas, Michael Stonebraker, Paolo Papotti, and Mourad Ouzzani. 2015. DataXFormer: Leveraging the web for semantic transformations. In Proceedings of the Conference on Innovative Data Systems Research (CIDR\u201915)."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.5555\/1105926.1106227"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the International Conference on Knowledge Discovery and Data Mining (SIGKDD\u201903)","volume":"3","author":"Baxter Rohan","year":"2003","unstructured":"Rohan Baxter, Peter Christen, and Tim Churches. 2003. A comparison of fast blocking methods for record linkage. In Proceedings of the International Conference on Knowledge Discovery and Data Mining (SIGKDD\u201903), Vol. 3. 25--27."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-008-0098-x"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1247480.1247521"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401913"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the Australasian Workshop on Health Data and Knowledge Management (HDKM\u201908)","author":"Christen Peter","year":"2008","unstructured":"Peter Christen. 2008. Febrl: A freely available record linkage system with a graphical user interface. In Proceedings of the Australasian Workshop on Health Data and Knowledge Management (HDKM\u201908). 17--25."},{"key":"e_1_2_1_9_1","volume-title":"Entity Resolution, and Duplicate Detection","author":"Christen Peter","unstructured":"Peter Christen. 2012. Data Matching\u2014Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection. Springer Data-Centric Systems and Applications."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1186\/1472-6947-2-9"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0020-0255(00)00070-0"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2007.250581"},{"key":"e_1_2_1_13_1","volume-title":"The Elements of Statistical Learning","author":"Friedman Jerome","unstructured":"Jerome Friedman, Trevor Hastie, and Robert Tibshirani. 2001. The Elements of Statistical Learning (1st ed.). Springer-Verlag, Berlin.","edition":"1"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1926385.1926423"},{"key":"e_1_2_1_15_1","volume-title":"Data Mining: Concepts and Techniques","author":"Han Jiawei","year":"2011","unstructured":"Jiawei Han, Jian Pei, and Micheline Kamber. 2011. Data Mining: Concepts and Techniques. Elsevier."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11222-017-9746-6"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.14778\/3231751.3231766"},{"key":"e_1_2_1_18_1","volume-title":"Winkler","author":"Herzog Thomas N.","year":"2007","unstructured":"Thomas N. Herzog, Fritz J. Scheuren, and William E. Winkler. 2007. Data Quality and Record Linkage Techniques. Springer-Verlag, New York."},{"key":"e_1_2_1_19_1","unstructured":"James Inman. 1849. Navigation and Nautical Astronomy for the Use of British Seamen. F. 8 J. Rivington."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the International Conference on Management of Data (SIGMOD\u201917)","author":"Jin Zhongjun","unstructured":"Zhongjun Jin, Michael R. Anderson, Michael Cafarella, and H. V. Jagadish. 2017. Foofah: Transforming data by example. In Proceedings of the International Conference on Management of Data (SIGMOD\u201917). 683--698."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1177\/1473871611415994"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920904"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.14778\/3151106.3151110"},{"key":"e_1_2_1_24_1","volume-title":"Wang","author":"Lee Yang W.","year":"2009","unstructured":"Yang W. Lee, Leo L. Pipino, James D. Funk, and Richard Y. Wang. 2009. Journey to Data Quality. The MIT Press."},{"key":"e_1_2_1_25_1","volume-title":"Mining of Massive Datasets","author":"Leskovec Jure","unstructured":"Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. 2014. Mining of Massive Datasets. Cambridge University Press."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.14778\/3402707.3402733"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.14778\/2947618.2947620"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the International Conference on Knowledge Discovery and Data Mining (SIGKDD\u201996)","author":"Alvaro","unstructured":"Alvaro E. Monge and Charles P. Elkan. 1996. The field matching problem: Algorithms and applications. In Proceedings of the International Conference on Knowledge Discovery and Data Mining (SIGKDD\u201996). 267--270."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196926"},{"key":"e_1_2_1_30_1","doi-asserted-by":"crossref","unstructured":"Felix Naumann and Melanie Herschel. 2010. An Introduction to Duplicate Detection. Morgan 8 Claypool Publishers.","DOI":"10.1007\/978-3-031-01835-0"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the International Workshop on Data and Information Quality. 219--233","author":"Oliveira Paulo","year":"2005","unstructured":"Paulo Oliveira, F\u00e1tima Rodrigues, Pedro Henriques, and Helena Galhardas. 2005. A taxonomy of data quality problems. In Proceedings of the International Workshop on Data and Information Quality. 219--233."},{"key":"e_1_2_1_32_1","volume-title":"Data Preparation for Data Mining","author":"Pyle Dorian","unstructured":"Dorian Pyle. 1999. Data Preparation for Data Mining. Vol. 1. Morgan Kaufmann."},{"key":"e_1_2_1_33_1","first-page":"3","article-title":"Data cleaning: Problems and current approaches","volume":"23","author":"Rahm Erhard","year":"2000","unstructured":"Erhard Rahm and Hong Hai Do. 2000. Data cleaning: Problems and current approaches. IEEE Data Eng. Bull. 23, 4 (2000), 3--13.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_34_1","volume-title":"Data Quality: The Field Guide","author":"Redman Thomas C.","year":"2001","unstructured":"Thomas C. Redman. 2001. Data Quality: The Field Guide. Digital Press."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063737"},{"key":"e_1_2_1_36_1","volume-title":"Handbook of Big Data Technologies","author":"Vatsalan Dinusha","unstructured":"Dinusha Vatsalan, Ziad Sehili, Peter Christen, and Erhard Rahm. 2017. Privacy-preserving record linkage for big data: Current approaches and research challenges. In Handbook of Big Data Technologies. Springer, 851--895."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/2021017.2021020"},{"key":"e_1_2_1_38_1","unstructured":"Y. Y. R. Wang R. Y. Wang M. Ziad and Y. W. Lee. 2001. Data Quality. Springer US."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.14778\/1454159.1454165"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824055"}],"container-title":["Journal of Data and Information Quality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3377878","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3377878","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:38:52Z","timestamp":1750199932000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3377878"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,13]]},"references-count":40,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2020,9,30]]}},"alternative-id":["10.1145\/3377878"],"URL":"https:\/\/doi.org\/10.1145\/3377878","relation":{},"ISSN":["1936-1955","1936-1963"],"issn-type":[{"value":"1936-1955","type":"print"},{"value":"1936-1963","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,13]]},"assertion":[{"value":"2019-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-06-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}