{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:11:21Z","timestamp":1750306281695,"version":"3.41.0"},"reference-count":37,"publisher":"Association for Computing Machinery (ACM)","issue":"1-2","license":[{"start":{"date-parts":[[2016,4,19]],"date-time":"2016-04-19T00:00:00Z","timestamp":1461024000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"publisher","award":["LP100200079"],"award-info":[{"award-number":["LP100200079"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Data and Information Quality"],"published-print":{"date-parts":[[2016,6,6]]},"abstract":"<jats:p>Textual databases are ubiquitous in many application domains. Examples of textual data range from names and addresses of customers to social media posts and bibliographic records. With online services, individuals are increasingly required to enter their personal details for example when purchasing products online or registering for government services, while many social network and e-commerce sites allow users to post short comments. Many online sites leave open the possibility for people to enter unintended or malicious abnormal values, such as names with errors, bogus values, profane comments, or random character sequences. In other applications, such as online bibliographic databases or comparative online shopping sites, databases are increasingly populated in (semi-) automatic ways through Web crawls. This practice can result in low quality data being added automatically into a database. In this article, we develop three techniques to automatically discover abnormal (unexpected or unusual) values in large textual databases. Following recent work in categorical outlier detection, our assumption is that \u201cnormal\u201d values are those that occur frequently in a database, while an individual abnormal value is rare. Our techniques are unsupervised and address the challenge of discovering abnormal values as an outlier detection problem. Our first technique is a basic but efficient q-gram set based technique, the second is based on a probabilistic language model, and the third employs morphological word features to train a one-class support vector machine classifier. Our aim is to investigate and develop techniques that are fast, efficient, and automatic. The output of our techniques can help in the development of rule-based data cleaning and information extraction systems, or be used as training data for further supervised data cleaning procedures. We evaluate our techniques on four large real-world datasets from different domains: two US voter registration databases containing personal details, the 2013 KDD Cup dataset of bibliographic records, and the SNAP Memetracker dataset of phrases from social networking sites. Our results show that our techniques can efficiently and automatically discover abnormal textual values, allowing an organization to conduct efficient data exploration, and improve the quality of their textual databases without the need of requiring explicit training data.<\/jats:p>","DOI":"10.1145\/2889311","type":"journal-article","created":{"date-parts":[[2016,4,22]],"date-time":"2016-04-22T13:53:06Z","timestamp":1461333186000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Automatic Discovery of Abnormal Values in Large Textual Databases"],"prefix":"10.1145","volume":"7","author":[{"given":"Peter","family":"Christen","sequence":"first","affiliation":[{"name":"The Australian National University, Acton, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ross W.","family":"Gayler","sequence":"additional","affiliation":[{"name":"Veda"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Khoi-Nguyen","family":"Tran","sequence":"additional","affiliation":[{"name":"The Australian National University, Acton, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeffrey","family":"Fisher","sequence":"additional","affiliation":[{"name":"The Australian National University, Acton, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dinusha","family":"Vatsalan","sequence":"additional","affiliation":[{"name":"The Australian National University, Acton, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,4,19]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063954"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1089\/10665270050081397"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1007558221122"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/376284.375682"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1541880.1541882"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2010.235"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1961189.1961199"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1006\/csla.1999.0128"},{"volume-title":"Entity Resolution, and Duplicate Detection","author":"Christen Peter","key":"e_1_2_1_9_1","unstructured":"Peter Christen . 2012. Data Matching\u2014Concepts and Techniques for Record Linkage , Entity Resolution, and Duplicate Detection . Springer , Berlin . Peter Christen. 2012. Data Matching\u2014Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection. Springer, Berlin."},{"volume-title":"Application of clustering and association methods in data cleaning","author":"Ciszak Lukasz","key":"e_1_2_1_11_1","unstructured":"Lukasz Ciszak . 2008. Application of clustering and association methods in data cleaning . In IMCSIT. IEEE , 97--103. Lukasz Ciszak. 2008. Application of clustering and association methods in data cleaning. In IMCSIT. IEEE, 97--103."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1281192.1281219"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401915"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.14778\/3402755.3402774"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1080\/09296179508590051"},{"key":"e_1_2_1_16_1","unstructured":"David Guthrie Louise Guthrie Ben Allison and Yorick Wilks. 2007. Unsupervised anomaly detection. In IJCAI. Hyderabad 1624--1628. David Guthrie Louise Guthrie Ben Allison and Yorick Wilks. 2007. Unsupervised anomaly detection. In IJCAI. Hyderabad 1624--1628."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1023\/B:AIRE.0000045502.10941.a9"},{"key":"e_1_2_1_18_1","volume-title":"Martin","author":"Jurafsky Daniel","year":"2008","unstructured":"Daniel Jurafsky and James H . Martin . 2008 . Speech and Language Processing (2nd ed.). Prentice Hall , Englewood Cliffs. Daniel Jurafsky and James H. Martin. 2008. Speech and Language Processing (2nd ed.). Prentice Hall, Englewood Cliffs."},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Michail Kazimianec and Nikolaus Augsten. 2011. PG-skip: Proximity graph based clustering of long strings. In DASFAA. 31--46. Michail Kazimianec and Nikolaus Augsten. 2011. PG-skip: Proximity graph based clustering of long strings. In DASFAA. 31--46.","DOI":"10.1007\/978-3-642-20152-3_3"},{"key":"e_1_2_1_20_1","series-title":"Lecture Notes in Computer Science","volume-title":"Madden","author":"Khan Shehroz S.","year":"2010","unstructured":"Shehroz S. Khan and Michael G . Madden . 2010 . A survey of recent trends in one class classification. In AICS, Lecture Notes in Computer Science , Vol. 6206 . Springer , 188--197. Shehroz S. Khan and Michael G. Madden. 2010. A survey of recent trends in one class classification. In AICS, Lecture Notes in Computer Science, Vol. 6206. Springer, 188--197."},{"key":"e_1_2_1_21_1","volume-title":"SNAP Datasets: Stanford Large Network Dataset Collection. (June","author":"Leskovec Jure","year":"2014","unstructured":"Jure Leskovec and Andrej Krevl . 2014. SNAP Datasets: Stanford Large Network Dataset Collection. (June 2014 ). http:\/\/snap.stanford.edu\/data. Jure Leskovec and Andrej Krevl. 2014. SNAP Datasets: Stanford Large Network Dataset Collection. (June 2014). http:\/\/snap.stanford.edu\/data."},{"key":"e_1_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Yunyao Li Rajasekar Krishnamurthy Sriram Raghavan Shivakumar Vaithyanathan and H. V. Jagadish. 2008. Regular expression learning for information extraction. In EMNLP. ACM 21--30. Yunyao Li Rajasekar Krishnamurthy Sriram Raghavan Shivakumar Vaithyanathan and H. V. Jagadish. 2008. Regular expression learning for information extraction. In EMNLP. ACM 21--30.","DOI":"10.3115\/1613715.1613719"},{"key":"e_1_2_1_23_1","unstructured":"Yunyao Li Frederick R. Reiss and Laura Chiticariu. 2011. SystemT: A declarative information extraction system. In ACL-HLT Systems Demonstrations. 109--114. Yunyao Li Frederick R. Reiss and Laura Chiticariu. 2011. SystemT: A declarative information extraction system. In ACL-HLT Systems Demonstrations. 109--114."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920916"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/944790.944808"},{"volume-title":"Foundations of Statistical Natural Language Processing","author":"Manning Christopher","key":"e_1_2_1_26_1","unstructured":"Christopher Manning and Hinrich Sch\u00fctze . 1999. Foundations of Statistical Natural Language Processing . MIT Press . Christopher Manning and Hinrich Sch\u00fctze. 1999. Foundations of Statistical Natural Language Processing. MIT Press."},{"key":"e_1_2_1_27_1","unstructured":"Arturas Mazeika and Michael B\u00f6hlen. 2006. Cleansing databases of misspelled proper nouns. In CleanDB. VLDB. Arturas Mazeika and Michael B\u00f6hlen. 2006. Cleansing databases of misspelled proper nouns. In CleanDB. VLDB."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/2567709.2567713"},{"key":"e_1_2_1_29_1","unstructured":"Thomas M\u00fcller and Hinrich Sch\u00fctze. 2011. Improved modeling of out-of-vocabulary words using morphological classes. In ACL. 524--528. Thomas M\u00fcller and Hinrich Sch\u00fctze. 2011. Improved modeling of out-of-vocabulary words using morphological classes. In ACL. 524--528."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022643204877"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1162\/089976601750264965"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1162\/089976600300015565"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661837"},{"key":"e_1_2_1_35_1","volume-title":"Bell","author":"Witten Ian H.","year":"1999","unstructured":"Ian H. Witten , Alistair Moffat , and Timothy C . Bell . 1999 . Managing Gigabytes (2nd ed.). Morgan Kaufmann , San Francisco. Ian H. Witten, Alistair Moffat, and Timothy C. Bell. 1999. Managing Gigabytes (2nd ed.). Morgan Kaufmann, San Francisco."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2011.261"},{"key":"e_1_2_1_37_1","volume-title":"Lyu","author":"Yang Haiqin","year":"2010","unstructured":"Haiqin Yang , Irwin King , and Michael R . Lyu . 2010 . Multi-task learning for one-class classification. In WCCI. IEEE , 2820--2827. Haiqin Yang, Irwin King, and Michael R. Lyu. 2010. Multi-task learning for one-class classification. In WCCI. IEEE, 2820--2827."},{"key":"e_1_2_1_38_1","unstructured":"Congle Zhang Tyler Baldwin Howard Ho Benny Kimelfeld and Yunyao Li. 2013. Adaptive parser-centric text normalization. In ACL. 1159--1168. Congle Zhang Tyler Baldwin Howard Ho Benny Kimelfeld and Yunyao Li. 2013. Adaptive parser-centric text normalization. In ACL. 1159--1168."}],"container-title":["Journal of Data and Information Quality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2889311","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2889311","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:38:54Z","timestamp":1750221534000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2889311"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,4,19]]},"references-count":37,"journal-issue":{"issue":"1-2","published-print":{"date-parts":[[2016,6,6]]}},"alternative-id":["10.1145\/2889311"],"URL":"https:\/\/doi.org\/10.1145\/2889311","relation":{},"ISSN":["1936-1955","1936-1963"],"issn-type":[{"type":"print","value":"1936-1955"},{"type":"electronic","value":"1936-1963"}],"subject":[],"published":{"date-parts":[[2016,4,19]]},"assertion":[{"value":"2015-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-04-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}