{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,11]],"date-time":"2026-10-11T00:02:01Z","timestamp":1791676921422,"version":"4.3.4"},"reference-count":47,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:00:00Z","timestamp":1687132800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:00:00Z","timestamp":1687132800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Big Data"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Fraud datasets often times lack consistent and accurate labels, and are characterized by having high class imbalance where the number of fraudulent examples are far fewer than those of normal ones. Machine learning designed for effectively detecting fraud is an important task since fraudulent behavior can have significant financial or health consequences, but is presented with significant challenges due to the class imbalance and availability of reliable labels. This paper presents an unsupervised fraud detection method that uses an iterative cleaning process for effective fraud detection. We measure our method performance using a newly created Medicare fraud big dataset and a widely used credit card fraud dataset. Additionally, we detail the process of creating the highly-imbalanced Medicare dataset from multiple publicly available sources, how additional trainable features were added, and how fraudulent labels were assigned for final model performance measurements. The results are compared with two popular unsupervised learners and show that our method outperforms both models in both datasets. Our work achieves a higher AUPRC with relatively few iterations across both domains.<\/jats:p>","DOI":"10.1186\/s40537-023-00750-3","type":"journal-article","created":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T14:02:23Z","timestamp":1687183343000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["Iterative cleaning and learning of big highly-imbalanced fraud data using unsupervised learning"],"prefix":"10.1186","volume":"10","author":[{"given":"Robert K. L.","family":"Kennedy","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zahra","family":"Salekshahrezaee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Flavio","family":"Villanustre","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Taghi M.","family":"Khoshgoftaar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,6,19]]},"reference":[{"issue":"5","key":"750_CR1","doi-asserted-by":"publisher","first-page":"1351","DOI":"10.1377\/hlthaff.28.5.1351","volume":"28","author":"L Morris","year":"2009","unstructured":"Morris L. Combating fraud in health care: an essential component of any cost containment strategy. Health Aff. 2009;28(5):1351\u20136.","journal-title":"Health Aff"},{"key":"750_CR2","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1007\/s10742-016-0154-8","volume":"17","author":"RA Bauder","year":"2017","unstructured":"Bauder RA, Khoshgoftaar TM, Seliya N. A survey on the state of healthcare upcoding fraud analysis and detection. Health Serv Outcomes Res Methodol. 2017;17:31\u201355.","journal-title":"Health Serv Outcomes Res Methodol"},{"issue":"1","key":"750_CR3","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1186\/s40537-018-0151-6","volume":"5","author":"JL Leevy","year":"2018","unstructured":"Leevy JL, Khoshgoftaar TM, Bauder RA, Seliya N. A survey on addressing high-class imbalance in big data. J Big Data. 2018;5(1):42.","journal-title":"J Big Data"},{"key":"750_CR4","doi-asserted-by":"crossref","unstructured":"Johnson JM, Khoshgoftaar TM. Encoding techniques for high-cardinality features and ensemble learners. In: 2021 IEEE 22nd international conference on information reuse and integration for data science (IRI). IEEE; 2021. p. 355\u201361.","DOI":"10.1109\/IRI51335.2021.00055"},{"key":"750_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s13755-018-0051-3","volume":"6","author":"RA Bauder","year":"2018","unstructured":"Bauder RA, Khoshgoftaar TM. The effects of varying class distribution on learner behavior for medicare fraud detection with imbalanced big data. Health Inf Sci Syst. 2018;6:1\u201314.","journal-title":"Health Inf Sci Syst"},{"issue":"4","key":"750_CR6","doi-asserted-by":"publisher","first-page":"449","DOI":"10.1007\/s11280-012-0178-0","volume":"16","author":"W Wei","year":"2013","unstructured":"Wei W, Li J, Cao L, Ou Y, Chen J. Effective detection of sophisticated online banking fraud on extremely imbalanced data. World Wide Web. 2013;16(4):449\u201375.","journal-title":"World Wide Web"},{"issue":"2","key":"750_CR7","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1023\/A:1007452223027","volume":"30","author":"M Kubat","year":"1998","unstructured":"Kubat M, Holte RC, Matwin S. Machine learning for the detection of oil spills in satellite radar images. Mach Learn. 1998;30(2):195\u2013215.","journal-title":"Mach Learn"},{"key":"750_CR8","unstructured":"Cieslak DA, Chawla NV, Striegel A. Combating imbalance in network intrusion datasets. In: GrC; 2006. p. 732\u20137."},{"key":"750_CR9","doi-asserted-by":"crossref","unstructured":"Davis J, Goadrich M. The relationship between precision-recall and roc curves. In: Proceedings of the 23rd international conference on machine learning. 2006. p. 233\u201340.","DOI":"10.1145\/1143844.1143874"},{"issue":"1","key":"750_CR10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-023-00724-5","volume":"10","author":"JT Hancock","year":"2023","unstructured":"Hancock JT, Khoshgoftaar TM, Johnson JM. Evaluating classifier performance with highly imbalanced big data. J Big Data. 2023;10(1):1\u201331.","journal-title":"J Big Data"},{"key":"750_CR11","doi-asserted-by":"crossref","unstructured":"Kennedy RK, Johnson JM, Khoshgoftaar TM. The effects of class label noise on highly-imbalanced big data. In: 2021 IEEE 33rd international conference on tools with artificial intelligence (ICTAI). IEEE; 2021. p. 1427\u201333.","DOI":"10.1109\/ICTAI52525.2021.00227"},{"key":"750_CR12","doi-asserted-by":"crossref","unstructured":"Salekshahrezaee Z, Leevy JL, Khoshgoftaar TM. A class-imbalanced study with feature extraction via PCA and convolutional autoencoder. In: 2022 IEEE 23rd international conference on information reuse and integration for data science (IRI). IEEE; 2022. p. 63\u20138.","DOI":"10.1109\/IRI54793.2022.00026"},{"issue":"1","key":"750_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-019-0231-2","volume":"6","author":"T Hasanin","year":"2019","unstructured":"Hasanin T, Khoshgoftaar TM, Leevy JL, Seliya N. Examining characteristics of predictive models with imbalanced big data. J Big Data. 2019;6(1):1\u201321.","journal-title":"J Big Data"},{"key":"750_CR14","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1016\/j.ins.2019.05.042","volume":"557","author":"F Carcillo","year":"2021","unstructured":"Carcillo F, Le Borgne Y-A, Caelen O, Kessaci Y, Obl\u00e9 F, Bontempi G. Combining unsupervised and supervised learning in credit card fraud detection. Inf Sci. 2021;557:317\u201331.","journal-title":"Inf Sci"},{"issue":"1","key":"750_CR15","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1109\/TDSC.2007.70228","volume":"5","author":"A Srivastava","year":"2008","unstructured":"Srivastava A, Kundu A, Sural S, Majumdar A. Credit card fraud detection using hidden Markov model. IEEE Trans Dependable Secure Comput. 2008;5(1):37\u201348.","journal-title":"IEEE Trans Dependable Secure Comput"},{"key":"750_CR16","unstructured":"Lee C-Y, Li C-L, Yoon J, Sohn K, Arik S, Pfister T. Self-supervise, refine, repeat: improving unsupervised anomaly detection. arXiv preprint. 2022. arXiv:2106.06115."},{"issue":"2","key":"750_CR17","first-page":"539","volume":"39","author":"X-Y Liu","year":"2008","unstructured":"Liu X-Y, Wu J, Zhou Z-H. Exploratory undersampling for class-imbalance learning. IEEE Trans Syst Man Cybern B Cybern. 2008;39(2):539\u201350.","journal-title":"IEEE Trans Syst Man Cybern B Cybern"},{"key":"750_CR18","doi-asserted-by":"crossref","unstructured":"Liu T-Y. Easyensemble and feature selection for imbalance data sets. In: 2009 International joint conference on bioinformatics, systems biology and intelligent computing. IEEE; 2009. p. 517\u201320.","DOI":"10.1109\/IJCBS.2009.22"},{"key":"750_CR19","unstructured":"Zong B, Song Q, Min MR, Cheng W, Lumezanu C, Cho D, Chen H. Deep autoencoding Gaussian mixture model for unsupervised anomaly detection. In: International conference on learning representations. 2018."},{"issue":"2","key":"750_CR20","doi-asserted-by":"publisher","first-page":"146","DOI":"10.26599\/TST.2019.9010051","volume":"26","author":"G Pu","year":"2020","unstructured":"Pu G, Wang L, Shen J, Dong F. A hybrid unsupervised clustering-based anomaly detection method. Tsinghua Sci Technol. 2020;26(2):146\u201353.","journal-title":"Tsinghua Sci Technol"},{"key":"750_CR21","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2021.107443","volume":"108","author":"S Maleki","year":"2021","unstructured":"Maleki S, Maleki S, Jennings NR. Unsupervised anomaly detection with LSTM autoencoders using statistical data-filtering. Appl Soft Comput. 2021;108: 107443.","journal-title":"Appl Soft Comput"},{"key":"750_CR22","doi-asserted-by":"crossref","unstructured":"Pang G, Yan C, Shen C, Hengel AVD, Bai X. Self-trained deep ordinal regression for end-to-end video anomaly detection. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition. 2020. p. 12173\u201382.","DOI":"10.1109\/CVPR42600.2020.01219"},{"key":"750_CR23","doi-asserted-by":"crossref","unstructured":"Beggel L, Pfeiffer M, Bischl B. Robust anomaly detection in images using adversarial autoencoders. In: Machine learning and knowledge discovery in databases: European conference, ECML PKDD 2019, W\u00fcrzburg, Germany, September 16\u201320, 2019, Proceedings, Part I. Springer; 2020. p. 206\u201322.","DOI":"10.1007\/978-3-030-46150-8_13"},{"key":"750_CR24","unstructured":"Ng A, et al. Sparse autoencoder. CS294A Lecture notes. 2011;72(2011):1\u201319."},{"key":"750_CR25","unstructured":"Chollet F, et al. Keras. 2015. https:\/\/keras.io."},{"issue":"4","key":"750_CR26","doi-asserted-by":"publisher","first-page":"1119","DOI":"10.1109\/TSMCB.2012.2187280","volume":"42","author":"S Wang","year":"2012","unstructured":"Wang S, Yao X. Multiclass imbalance problems: analysis and potential solutions. IEEE Trans Syst Man Cybern B Cybern. 2012;42(4):1119\u201330.","journal-title":"IEEE Trans Syst Man Cybern B Cybern"},{"issue":"4","key":"750_CR27","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1007\/s13748-016-0094-0","volume":"5","author":"B Krawczyk","year":"2016","unstructured":"Krawczyk B. Learning from imbalanced data: open challenges and future directions. Progr Artif Intell. 2016;5(4):221\u201332.","journal-title":"Progr Artif Intell"},{"key":"750_CR28","doi-asserted-by":"crossref","unstructured":"Khoshgoftaar TM, Seiffert C, Van\u00a0Hulse J, Napolitano A, Folleco A. Learning with limited minority class data. In: Sixth international conference on machine learning and applications (ICMLA 2007). IEEE; 2007. p. 348\u201353.","DOI":"10.1109\/ICMLA.2007.76"},{"key":"750_CR29","unstructured":"Japkowicz N. The class imbalance problem: significance and strategies. In: Proc. of the Int\u2019l Conf. on artificial intelligence, vol. 56; 2000. p. 111\u20137."},{"key":"750_CR30","unstructured":"Fang J, Xia S, Lin J, Xia Z, Liu X, Jiang Y. Alpha discovery neural network based on prior knowledge. 2019. arXiv preprint. arXiv:1912.11761."},{"key":"750_CR31","doi-asserted-by":"crossref","unstructured":"Clark J, Liu Z, Japkowicz N. Adaptive threshold for outlier detection on data streams. In: 2018 IEEE 5th international conference on data science and advanced analytics (DSAA). IEEE; 2018. p. 41\u20139.","DOI":"10.1109\/DSAA.2018.00014"},{"key":"750_CR32","doi-asserted-by":"crossref","unstructured":"Li Z, Zhao Y, Botta N, Ionescu C, Hu X. COPOD: copula-based outlier detection. In: 2020 IEEE international conference on data mining (ICDM). IEEE; 2020. p. 1118\u201323.","DOI":"10.1109\/ICDM50108.2020.00135"},{"issue":"96","key":"750_CR33","first-page":"1","volume":"20","author":"Y Zhao","year":"2019","unstructured":"Zhao Y, Nasrullah Z, Li Z. Pyod: a python toolbox for scalable outlier detection. J Mach Learn Res. 2019;20(96):1\u20137.","journal-title":"J Mach Learn Res"},{"key":"750_CR34","doi-asserted-by":"crossref","unstructured":"Liu FT, Ting KM, Zhou Z-H. Isolation forest. In: 2008 eighth IEEE international conference on data mining. IEEE; 2008. p. 413\u201322.","DOI":"10.1109\/ICDM.2008.17"},{"key":"750_CR35","doi-asserted-by":"crossref","unstructured":"Bauder RA, da Rosa R, Khoshgoftaar TM. Identifying medicare provider fraud with unsupervised machine learning. In: 2018 IEEE international conference on information reuse and integration (IRI). 2018; IEEE. p. 285\u201392.","DOI":"10.1109\/IRI.2018.00051"},{"key":"750_CR36","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011;12:2825\u201330.","journal-title":"J Mach Learn Res"},{"key":"750_CR37","doi-asserted-by":"crossref","unstructured":"Johnson JM, Khoshgoftaar TM. Hcpcs2vec: healthcare procedure embeddings for medicare fraud prediction. In: 2020 IEEE 6th international conference on collaboration and internet computing (CIC). IEEE; 2020. p. 145\u201352.","DOI":"10.1109\/CIC50333.2020.00026"},{"key":"750_CR38","unstructured":"U.S. Government. US Centers for Medicare & Medicaid Services: the official U.S. government site for medicare. https:\/\/www.medicare.gov\/."},{"key":"750_CR39","unstructured":"Centers for Medicare & Medicaid Services: trustees report & trust funds. https:\/\/www.cms.gov\/Research-Statistics-Data-and-Systems\/Statistics-Trends-and-Reports\/ReportsTrustFunds\/index.html."},{"key":"750_CR40","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1016\/j.accinf.2016.04.001","volume":"21","author":"G van Capelleveen","year":"2016","unstructured":"van Capelleveen G, Poel M, Mueller RM, Thornton D, van Hillegersberg J. Outlier detection in healthcare fraud: a case study in the medicaid dental domain. Int J Acc Inf Syst. 2016;21:18\u201331.","journal-title":"Int J Acc Inf Syst"},{"key":"750_CR41","unstructured":"U.S. Department of Health and Human Services Office of Inspector General: LEIE downloadable databases. https:\/\/oig.hhs.gov\/exclusions\/exclusions_list.asp."},{"key":"750_CR42","unstructured":"U.S. Department of Health and Human Services Office of Inspector General: exclusion authorities. https:\/\/oig.hhs.gov\/exclusions\/authorities.asp."},{"key":"750_CR43","doi-asserted-by":"crossref","unstructured":"Bauder RA, Khoshgoftaar TM. A novel method for fraudulent medicare claims detection from expected payment deviations (application paper). In: 2016 IEEE 17th international conference on information reuse and integration (IRI). 2016; IEEE. p. 11\u201319.","DOI":"10.1109\/IRI.2016.11"},{"key":"750_CR44","doi-asserted-by":"crossref","unstructured":"Dal\u00a0Pozzolo A, Caelen O, Johnson RA, Bontempi G. Calibrating probability with undersampling for unbalanced classification. In: 2015 IEEE symposium series on computational intelligence. IEEE; 2015. p. 159\u201366.","DOI":"10.1109\/SSCI.2015.33"},{"key":"750_CR45","unstructured":"Provost FJ, Fawcett T, et al. Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions. In: KDD, vol. 97; 1997. p. 43\u20138."},{"key":"750_CR46","first-page":"1","volume":"3","author":"H Abdi","year":"2010","unstructured":"Abdi H, Williams LJ. Tukey\u2019s honestly significant difference (HSD) test. Encycl Res Des. 2010;3:1\u20135.","journal-title":"Encycl Res Des"},{"key":"750_CR47","volume-title":"Intermediate statistical methods and applications: a computer package approach","author":"M Berenson","year":"1983","unstructured":"Berenson M, Levine D, Goldstein M. Intermediate statistical methods and applications: a computer package approach. Englewood Cliffs: Prentice-Hall; 1983."}],"container-title":["Journal of Big Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-023-00750-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s40537-023-00750-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-023-00750-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T14:05:59Z","timestamp":1687183559000},"score":1,"resource":{"primary":{"URL":"https:\/\/journalofbigdata.springeropen.com\/articles\/10.1186\/s40537-023-00750-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,19]]},"references-count":47,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["750"],"URL":"https:\/\/doi.org\/10.1186\/s40537-023-00750-3","relation":{},"ISSN":["2196-1115"],"issn-type":[{"value":"2196-1115","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,19]]},"assertion":[{"value":"4 April 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 May 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 June 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"106"}}