{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,22]],"date-time":"2026-09-22T12:17:31Z","timestamp":1790079451062,"version":"4.0.1"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,6,19]],"date-time":"2024-06-19T00:00:00Z","timestamp":1718755200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,6,19]],"date-time":"2024-06-19T00:00:00Z","timestamp":1718755200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R56AI165448"],"award-info":[{"award-number":["R56AI165448"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Compared to traditional supervised machine learning approaches employing fully labeled samples, positive-unlabeled (PU) learning techniques aim to classify \u201cunlabeled\u201d samples based on a smaller proportion of known positive examples. This more challenging modeling goal reflects many real-world scenarios in which negative examples are not available\u2014posing direct challenges to defining prediction accuracy and robustness. While several studies have evaluated predictions learned from only definitive positive examples, few have investigated whether correct classification of a high proportion of known positives (KP) samples from among unlabeled samples can act as a surrogate to indicate model quality.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>In this study, we report a novel methodology combining multiple established PU learning-based strategies with permutation testing to evaluate the potential of KP samples to accurately classify unlabeled samples without using \u201cground truth\u201d positive and negative labels for validation. Multivariate synthetic and real-world high-dimensional benchmark datasets were employed to demonstrate the suitability of the proposed pipeline to provide evidence of model robustness across varied underlying ground truth class label compositions among the unlabeled set and with different proportions of KP examples. Comparisons between model performance with actual and permuted labels could be used to distinguish reliable from unreliable models.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>As in fully supervised machine learning, permutation testing offers a means to set a baseline \u201cno-information rate\u201d benchmark in the context of semi-supervised PU learning inference tasks\u2014providing a standard against which model performance can be compared.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12859-024-05834-2","type":"journal-article","created":{"date-parts":[[2024,6,19]],"date-time":"2024-06-19T02:02:14Z","timestamp":1718762534000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Leveraging permutation testing to assess confidence in positive-unlabeled learning applied to high-dimensional biological datasets"],"prefix":"10.1186","volume":"25","author":[{"given":"Shiwei","family":"Xu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Margaret E.","family":"Ackerman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,6,19]]},"reference":[{"key":"5834_CR1","unstructured":"K\u00f6ppen M. The curse of dimensionality. In: 5th online world conference on soft computing in industrial applications (WSC5). 2000."},{"key":"5834_CR2","volume-title":"Permutation tests: a practical guide to resampling methods for testing hypotheses","author":"P Good","year":"2013","unstructured":"Good P. Permutation tests: a practical guide to resampling methods for testing hypotheses. Berlin: Springer; 2013."},{"issue":"6","key":"5834_CR3","first-page":"1833","volume":"11","author":"M Ojala","year":"2010","unstructured":"Ojala M, Garriga GC. Permutation tests for studying classifier performance. J Mach Learn Res. 2010;11(6):1833\u201363.","journal-title":"J Mach Learn Res"},{"issue":"1","key":"5834_CR4","doi-asserted-by":"publisher","first-page":"bbab461","DOI":"10.1093\/bib\/bbab461","volume":"23","author":"F Li","year":"2022","unstructured":"Li F, et al. Positive-unlabeled learning in bioinformatics and computational biology: a brief review. Brief Bioinform. 2022;23(1):bbab461.","journal-title":"Brief Bioinform"},{"key":"5834_CR5","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1016\/j.patrec.2013.06.010","volume":"37","author":"F Mordelet","year":"2014","unstructured":"Mordelet F, Vert JP. A bagging SVM to learn from positive and unlabeled examples. Pattern Recogn Lett. 2014;37:201\u20139.","journal-title":"Pattern Recogn Lett"},{"issue":"1","key":"5834_CR6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/1471-2105-11-1","volume":"11","author":"N Bhardwaj","year":"2010","unstructured":"Bhardwaj N, Gerstein M, Lu H. Genome-wide sequence-based prediction of peripheral proteins using a novel semi-supervised learning technique. BMC Bioinform. 2010;11(1):1\u20138.","journal-title":"BMC Bioinform"},{"issue":"03","key":"5834_CR7","doi-asserted-by":"publisher","first-page":"1541005","DOI":"10.1142\/S021972001541005X","volume":"13","author":"Z Cheng","year":"2015","unstructured":"Cheng Z, Zhou S, Guan J. Computationally predicting protein-RNA interactions using only positive and unlabeled examples. J Bioinform Comput Biol. 2015;13(03):1541005.","journal-title":"J Bioinform Comput Biol"},{"key":"5834_CR8","unstructured":"Liu B, et al. Partially supervised classification of text documents. In ICML. Sydney; 2002."},{"issue":"4","key":"5834_CR9","doi-asserted-by":"publisher","first-page":"719","DOI":"10.1007\/s10994-020-05877-5","volume":"109","author":"J Bekker","year":"2020","unstructured":"Bekker J, Davis J. Learning from positive and unlabeled data: a survey. Mach Learn. 2020;109(4):719\u201360.","journal-title":"Mach Learn"},{"key":"5834_CR10","doi-asserted-by":"crossref","unstructured":"Ren Y, Ji D, Zhang H. Positive unlabeled learning for deceptive reviews detection. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 2014.","DOI":"10.3115\/v1\/D14-1055"},{"key":"5834_CR11","doi-asserted-by":"crossref","unstructured":"Li H, et al. Spotting fake reviews via collective positive-unlabeled learning. In: 2014 IEEE international conference on data mining. 2014.","DOI":"10.1109\/ICDM.2014.47"},{"issue":"1","key":"5834_CR12","doi-asserted-by":"publisher","first-page":"70","DOI":"10.1109\/TKDE.2004.1264823","volume":"16","author":"H Yu","year":"2004","unstructured":"Yu H, Han J, Chang K-C. PEBL: Web page classification without negative examples. IEEE Trans Knowl Data Eng. 2004;16(1):70\u201381.","journal-title":"IEEE Trans Knowl Data Eng"},{"issue":"1","key":"5834_CR13","doi-asserted-by":"publisher","first-page":"228","DOI":"10.1186\/1471-2105-11-228","volume":"11","author":"L Cerulo","year":"2010","unstructured":"Cerulo L, Elkan C, Ceccarelli M. Learning gene regulatory networks from only positive and unlabeled data. BMC Bioinform. 2010;11(1):228.","journal-title":"BMC Bioinform"},{"issue":"2","key":"5834_CR14","doi-asserted-by":"publisher","first-page":"554","DOI":"10.1111\/j.1541-0420.2008.01116.x","volume":"65","author":"G Ward","year":"2009","unstructured":"Ward G, et al. Presence-only data and the EM algorithm. Biometrics. 2009;65(2):554\u201363.","journal-title":"Biometrics"},{"issue":"2","key":"5834_CR15","doi-asserted-by":"publisher","first-page":"320","DOI":"10.1109\/TMI.2021.3126169","volume":"41","author":"M Nagaya","year":"2022","unstructured":"Nagaya M, Ukita N. Embryo grading with unreliable labels due to chromosome abnormalities by regularized PU learning with ranking. IEEE Trans Med Imaging. 2022;41(2):320\u201331.","journal-title":"IEEE Trans Med Imaging"},{"issue":"3","key":"5834_CR16","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1007\/s13721-012-0012-8","volume":"1","author":"C K\u0131l\u0131\u00e7","year":"2012","unstructured":"K\u0131l\u0131\u00e7 C, Tan M. Positive unlabeled learning for deriving protein interaction networks. Netw Model Anal Health Inform Bioinform. 2012;1(3):87\u2013102.","journal-title":"Netw Model Anal Health Inform Bioinform"},{"issue":"3","key":"5834_CR17","doi-asserted-by":"publisher","first-page":"109086","DOI":"10.1016\/j.isci.2024.109086","volume":"27","author":"S Xu","year":"2024","unstructured":"Xu S, Kelkar NS, Ackerman ME. Positive-unlabeled learning to infer protection status and identify correlates in vaccine efficacy field trials. iScience. 2024;27(3):109086.","journal-title":"iScience"},{"key":"5834_CR18","doi-asserted-by":"crossref","unstructured":"Street WN, Wolberg WH, Mangasarian OL. Nuclear feature extraction for breast tumor diagnosis. In: Biomedical image processing and biomedical visualization. SPIE. 1993.","DOI":"10.1117\/12.148698"},{"issue":"10","key":"5834_CR19","doi-asserted-by":"publisher","first-page":"1113","DOI":"10.1038\/ng.2764","volume":"45","author":"JN Weinstein","year":"2013","unstructured":"Weinstein JN, Collisson EA, Mills GB, Shaw KR, Ozenberger BA, Ellrott K, Shmulevich I, Sander C, Stuart JM. The cancer genome atlas pan-cancer analysis project. Nat Genet. 2013;45(10):1113\u201320.","journal-title":"Nat Genet"},{"key":"5834_CR20","doi-asserted-by":"publisher","first-page":"788619","DOI":"10.3389\/fimmu.2022.788619","volume":"13","author":"SK Lakhashe","year":"2022","unstructured":"Lakhashe SK, et al. Cooperation between systemic and mucosal antibodies induced by virosomal vaccines targeting HIV-1 Env: protection of Indian rhesus macaques against low-dose intravaginal SHIV challenges. Front Immunol. 2022;13:788619.","journal-title":"Front Immunol"},{"key":"5834_CR21","unstructured":"Li X, Liu B. Learning to classify texts using positive and unlabeled data. In: IJCAI. Citeseer. 2003."},{"key":"5834_CR22","doi-asserted-by":"publisher","DOI":"10.1002\/9781118548387","volume-title":"Applied logistic regression","author":"DW Hosmer Jr","year":"2013","unstructured":"Hosmer DW Jr, Lemeshow S, Sturdivant RX. Applied logistic regression, vol. 398. New York: Wiley; 2013."},{"key":"5834_CR23","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, et al. Scikit-learn: machine learning in python. J Mach Learn Res. 2011;12:2825\u201330.","journal-title":"J Mach Learn Res"},{"issue":"7825","key":"5834_CR24","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1038\/s41586-020-2649-2","volume":"585","author":"CR Harris","year":"2020","unstructured":"Harris CR, et al. Array programming with NumPy. Nature. 2020;585(7825):357\u201362.","journal-title":"Nature"},{"key":"5834_CR25","unstructured":"Liu B, et al. Building text classifiers using positive and unlabeled examples. In: Third IEEE international conference on data mining. IEEE. 2003."},{"key":"5834_CR26","unstructured":"Kassambara A, Kassambara MA. Package \u2018ggpubr\u2019. R package version 0.1, 2020. 6(0)."},{"key":"5834_CR27","unstructured":"Inc. P.T. Collaborative data science. 2015. https:\/\/plot.ly."}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-024-05834-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12859-024-05834-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-024-05834-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,19]],"date-time":"2024-06-19T13:02:25Z","timestamp":1718802145000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-024-05834-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,19]]},"references-count":27,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["5834"],"URL":"https:\/\/doi.org\/10.1186\/s12859-024-05834-2","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.07.06.548028","asserted-by":"object"}]},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,19]]},"assertion":[{"value":"6 July 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 June 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 June 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethic approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"218"}}