{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,21]],"date-time":"2026-01-21T04:09:48Z","timestamp":1768968588805,"version":"3.49.0"},"reference-count":29,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2024,3,1]],"date-time":"2024-03-01T00:00:00Z","timestamp":1709251200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Big Data"],"abstract":"<jats:p>Traditional data curation processes typically depend on human intervention. As data volume and variety grow exponentially, organizations are striving to increase efficiency of their data processes by automating manual processes and making them as unsupervised as possible. An additional challenge is to make these unsupervised processes scalable to meet the demands of increased data volume. This paper describes the parallelization of an unsupervised entity resolution (ER) process. ER is a component of many different data curation processes because it clusters records from multiple data sources that refer to the same real-world entity, such as the same customer, patient, or product. The ability to scale ER processes is particularly important because the computation effort of ER increases quadratically with data volume. The Data Washing Machine (DWM) is an already proposed unsupervised ER system which clusters references from diverse data sources. This work aims at solving the single-threaded nature of the DWM by adopting the parallelization nature of Hadoop MapReduce. However, the proposed parallelization method can be applied to both supervised systems, where matching rules are created by experts, and unsupervised systems, where expert intervention is not required. The DWM uses an entropy measure to self-evaluate the quality of record clustering. The current single-threaded implementations of the DWM in Python and Java are not scalable beyond a few 1,000 records and rely on large, shared memory. The objective of this research is to solve the major two shortcomings of the current design of the DWM which are the creation and usage of shared memory and lack of scalability by leveraging on the power of Hadoop MapReduce. We propose Hadoop Data Washing Machine (HDWM), a MapReduce implementation of the legacy DWM. The scalability of the proposed system is displayed using publicly available ER datasets. Based on results from our experiment, we conclude that HDWM can cluster from 1,000's to millions of equivalent references using multiple computational nodes with independent RAM and CPU cores.<\/jats:p>","DOI":"10.3389\/fdata.2024.1296552","type":"journal-article","created":{"date-parts":[[2024,3,1]],"date-time":"2024-03-01T13:48:43Z","timestamp":1709300923000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["A scalable MapReduce-based design of an unsupervised entity resolution system"],"prefix":"10.3389","volume":"7","author":[{"given":"Nicholas Kofi Akortia","family":"Hagan","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John R.","family":"Talburt","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kris E.","family":"Anderson","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Deasia","family":"Hagan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2024,3,1]]},"reference":[{"key":"B1","first-page":"19","article-title":"A scalable, hybrid entity resolution process for unstandardized entity references","volume":"35","author":"Al Sarkhi","year":"2020","journal-title":"J. Comput. Sci. Coll"},{"key":"B2","first-page":"12","article-title":"Estimating the parameters for linking unstandardized references with the matrix comparator","volume":"10","author":"Al Sarkhi","year":"2018","journal-title":"J. Inform. Technol. Manag"},{"key":"B3","first-page":"64","article-title":"An analysis of the effect of stop words on the performance of the matrix comparator for entity resolution","volume":"34","author":"Al Sarkhi","year":"2019","journal-title":"J. Comput. Sci. Coll"},{"key":"B4","article-title":"\u201cOptimal starting parameters for unsupervised data clustering and cleaning in the data washing machine,\u201d","author":"Anderson","year":"2023","journal-title":"Proceeding: Future Technologies Conference (FTC'23)."},{"key":"B5","doi-asserted-by":"publisher","first-page":"1537","DOI":"10.1109\/TKDE.2011.127","article-title":"A survey of indexing techniques for scalable record linkage and deduplication","volume":"24","author":"Christen","year":"2012","journal-title":"IEEE Trans. Knowl. Data Eng"},{"key":"B6","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1145\/1327452.1327492","article-title":"MapReduce: simplified data processing on large clusters","volume":"51","author":"Dean","year":"2008","journal-title":"Commun. ACM"},{"key":"B7","doi-asserted-by":"publisher","first-page":"901","DOI":"10.14569\/IJACSA.2022.0130901","article-title":"ModER: graph-based unsupervised entity resolution using composite modularity optimization and locality sensitive hashing","volume":"13","author":"Ebeid","year":"2022","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"B8","doi-asserted-by":"publisher","first-page":"895","DOI":"10.1007\/s11704-016-5346-4","article-title":"MapReduce-based entity matching with multiple blocking functions","volume":"11","author":"Jin","year":"2017","journal-title":"Front. Comput. Sci"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1145\/2064085.2064087","article-title":"\u201cLearning-based entity resolution with MapReduce,\u201d","author":"Kolb","year":"2011","journal-title":"Proceedings of the Third International Workshop on Cloud Data Management"},{"key":"B10","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1007\/s13222-014-0154-1","article-title":"Iterative computation of connected graph components with MapReduce","volume":"14","author":"Kolb","year":"2014","journal-title":"Datenbank-Spektrum"},{"key":"B11","doi-asserted-by":"publisher","first-page":"1878","DOI":"10.14778\/2367502.2367527","article-title":"Dedoop: efficient deduplication with Hadoop","volume":"5","author":"Kolb","year":"","journal-title":"Proc. VLDB Endowm"},{"key":"B12","first-page":"618","article-title":"\u201cLoad balancing for MapReduce-based entity resolution,\u201d","volume-title":"2012 IEEE 28th International Conference on Data Engineering","author":"Kolb","year":""},{"key":"B13","first-page":"1","article-title":"\u201cDon't match twice: redundancy-free similarity computation with MapReduce,\u201d","author":"Kolb","year":"2013","journal-title":"Proceedings of the Second Workshop on Data Analytics in the Cloud"},{"key":"B14","doi-asserted-by":"publisher","first-page":"484","DOI":"10.14778\/1920841.1920904","article-title":"Evaluation of entity resolution approaches on real-world match problems","volume":"3","author":"K\u00f6pcke","year":"2010","journal-title":"Proc. VLDB Endowm."},{"key":"B15","doi-asserted-by":"crossref","first-page":"1087","DOI":"10.1109\/CSCI46756.2018.00211","article-title":"\u201cScoring matrix for unstandardized data in entity resolution,\u201d","volume-title":"2018 International Conference on Computational Science and Computational Intelligence (CSCI)","author":"Li","year":"2018"},{"key":"B16","doi-asserted-by":"crossref","first-page":"192","DOI":"10.1007\/978-3-031-09469-9_17","article-title":"\u201cEfficient entity resolution for bibliographic data using MapReduce,\u201d","author":"Mittal","year":"2022","journal-title":"Advanced Informatics for Computing Research"},{"key":"B17","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1016\/j.jpdc.2020.05.003","article-title":"Estimating record linkage costs in distributed environments","volume":"143","author":"Nascimento","year":"","journal-title":"J. Parallel Distribut. Comput"},{"key":"B18","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1007\/s10115-019-01347-0","article-title":"Exploiting block co-occurrence to control block sizes for entity resolution","volume":"62","author":"Nascimento","year":"","journal-title":"Knowl. Inform. Syst"},{"key":"B19","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3377455","article-title":"Blocking and filtering techniques for entity resolution: a survey","volume":"53","author":"Papadakis","year":"2020","journal-title":"ACM Comput. Surveys"},{"key":"B20","doi-asserted-by":"crossref","first-page":"415","DOI":"10.1002\/9781119357056.ch17","article-title":"\u201cAn automated regression testing framework for a hadoop-based entity resolution system,\u201d","author":"Pullen","year":"2018","journal-title":"Analytic Methods in Systems and Software Testing, 1st Edn"},{"key":"B21","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1007\/978-3-319-66917-5_19","article-title":"Comparative evaluation of distributed clustering schemes for multi-source entity resolution","volume":"19","author":"Saeedi","year":"2017","journal-title":"Adv. Datab. Inform. Syst."},{"key":"B22","first-page":"458","article-title":"\u201cCC-MR\u2014finding connected components in huge graphs with MapReduce,\u201d","author":"Seidl","year":"2012","journal-title":"Proceedings of the 2012th European Conference on Machine Learning and Knowledge Discovery in Databases - Volume Part I"},{"key":"B23","doi-asserted-by":"publisher","first-page":"1173","DOI":"10.14778\/2994509.2994533","article-title":"BLAST: a loosely schema-aware meta-blocking approach for entity resolution","volume":"9","author":"Simonini","year":"2016","journal-title":"Proc. VLDB Endowm"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.14569\/IJACSA.2020.0111279","article-title":"An iterative, self-assessing entity resolution system: first steps toward a data washing machine","author":"Talburt","year":"2020","journal-title":"Int. J. Adv. Comput. Sci. Appl."},{"key":"B25","doi-asserted-by":"publisher","first-page":"1148331","DOI":"10.3389\/fdata.2023.1148331","article-title":"Editorial: Automated data curation and data governance automation","volume":"6","author":"Talburt","year":"2023","journal-title":"Front. Big Data"},{"key":"B26","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1007\/978-3-030-03643-0_14","article-title":"\u201cEvaluating and improving data fusion accuracy,\u201d","author":"Talburt","year":"2019","journal-title":"Information Quality in Information Fusion and Decision Making"},{"key":"B27","volume-title":"Entity Information Life Cycle for Big Data: Master Data Management and Information Integration, 1st Edn","author":"Talburt","year":"2015"},{"key":"B28","first-page":"91","article-title":"SOG: a synthetic occupancy generator to support entity resolution instruction and research","volume":"9","author":"Talburt","year":"2009","journal-title":"ICIQ"},{"key":"B29","doi-asserted-by":"crossref","first-page":"1233","DOI":"10.1109\/CCGrid.2015.24","article-title":"\u201cEliminating the redundancy in MapReduce-based entity resolution,\u201d","volume-title":"2015 15th IEEE\/ACM International Symposium on Cluster, Cloud and Grid Computing","author":"Yan","year":"2015"}],"container-title":["Frontiers in Big Data"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdata.2024.1296552\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,1]],"date-time":"2024-03-01T13:49:02Z","timestamp":1709300942000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdata.2024.1296552\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,3,1]]},"references-count":29,"alternative-id":["10.3389\/fdata.2024.1296552"],"URL":"https:\/\/doi.org\/10.3389\/fdata.2024.1296552","relation":{},"ISSN":["2624-909X"],"issn-type":[{"value":"2624-909X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,3,1]]},"article-number":"1296552"}}