{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,5]],"date-time":"2025-12-05T09:03:25Z","timestamp":1764925405302,"version":"3.46.0"},"reference-count":58,"publisher":"Society for Industrial & Applied Mathematics (SIAM)","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SIAM Journal on Mathematics of Data Science"],"published-print":{"date-parts":[[2025,12,31]]},"DOI":"10.1137\/24m1715283","type":"journal-article","created":{"date-parts":[[2025,12,5]],"date-time":"2025-12-05T09:00:18Z","timestamp":1764925218000},"page":"1904-1927","source":"Crossref","is-referenced-by-count":0,"title":["Tight PAC-Bayesian Risk Certificates for Contrastive Learning"],"prefix":"10.1137","volume":"7","author":[{"given":"Anna","family":"van Elst","sequence":"first","affiliation":[{"name":"LTCI, T\u00e9l\u00e9com Paris, Institut Polytechnique de Paris, Paris, France."}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0202-7007","authenticated-orcid":true,"given":"Debarghya","family":"Ghoshdastidar","sequence":"additional","affiliation":[{"name":"School of Computation Information and Technology, Technical University of Munich, Munich, Germany."}]}],"member":"351","published-online":{"date-parts":[[2025,12,5]]},"reference":[{"key":"ref1","doi-asserted-by":"crossref","unstructured":"M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. G. Rabbat, Y. LeCun, and N. Ballas, Self-supervised learning from images with a joint-embedding predictive architecture, in IEEE\/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, Canada, 2023, IEEE, Piscataway, NJ, 2023, pp. 15619\u201315629, https:\/\/doi.org\/10.1109\/CVPR52729.2023.01499.","DOI":"10.1109\/CVPR52729.2023.01499"},{"key":"ref2","unstructured":"H. Bao, Y. Nagano, and K. Nozawa, On the surrogate gap between contrastive and supervised losses, Proc. Mach. Learn. Res. (PMLR), 1621 (2022), pp. 1585\u20131606."},{"key":"ref3","unstructured":"A. Bardes, J. Ponce, and Y. LeCun, Vicreg: Variance-invariance-covariance regularization for self-supervised learning, in International Conference on Learning Representations, 2022, https:\/\/openreview.net\/forum?id=xm6YD62D1Ub."},{"key":"ref4","unstructured":"V. Cabannes, B. T. Kiani, R. Balestriero, Y. LeCun, and A. Bietti, The SSL interplay: Augmentations, inductive bias, and generalization, Proc. Mach. Learn. Res. (PMLR), 202 (2023), pp. 3252\u20133298."},{"key":"ref5","volume-title":"PAC-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning","author":"Catoni O.","year":"2007"},{"key":"ref6","unstructured":"T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, A simple framework for contrastive learning of visual representations, Proc. Mach. Learn. Res. (PMLR), 119 (2020), pp. 1597\u20131607."},{"key":"ref7","volume":"22","author":"Chen W.","year":"2009","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref8","doi-asserted-by":"crossref","unstructured":"X. Chen and K. He, Exploring simple siamese representation learning, in IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, IEEE Computer Society, Los Alamitos, CA, 2021, pp. 15750\u201315758.","DOI":"10.1109\/CVPR46437.2021.01549"},{"key":"ref9","unstructured":"B.E. Ch\u00e9rief-Abdellatif, Y. Shi, A. Doucet, and B. Guedj, On PAC-Bayesian reconstruction guarantees for VAEs, Proc. Mach. Learn. Res. (PMLR), 151 (2022), pp. 3066\u20133079."},{"key":"ref10","unstructured":"G. K. Dziugaite and D. M. Roy, Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data, in Proceedings of the 33rd Annual Conference on Uncertainty in Artificial Intelligence (UAI), 2017, https:\/\/www.auai.org\/uai2017\/proceedings\/papers\/173.pdf."},{"key":"ref11","doi-asserted-by":"crossref","unstructured":"B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, CLAP learning audio concepts from natural language supervision, in ICASSP 2023, 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, Piscataway, NJ, 2023, pp. 1\u20135.","DOI":"10.1109\/ICASSP49357.2023.10095889"},{"key":"ref12","first-page":"2173","volume-title":"Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021","author":"Farid A.","year":"2021"},{"key":"ref13","doi-asserted-by":"crossref","unstructured":"T. Gao, X. Yao, and D. Chen, SimCSE: Simple contrastive learning of sentence embeddings, in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Kerrville, TX, 2021.","DOI":"10.18653\/v1\/2021.emnlp-main.552"},{"key":"ref14","doi-asserted-by":"crossref","unstructured":"P. Germain, A. Lacasse, F. Laviolette, and M. Marchand, PAC-Bayesian learning of linear classifiers, in Proceedings of the 26th Annual International Conference on Machine Learning, ACM, New York, 2009, pp. 353\u2013360.","DOI":"10.1145\/1553374.1553419"},{"key":"ref15","unstructured":"F. Graf, C. D. Hofer, M. Niethammer, and R. Kwitt, Dissecting supervised constrastive learning, Proc. Mach. Learn. Res. (PMLR), 139 (2021), pp. 3821\u20133830."},{"key":"ref16","first-page":"21271","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020","author":"Grill J.","year":"2020"},{"key":"ref17","unstructured":"M. Gutmann and A. Hyv\u00e4rinen, Noise-contrastive estimation: A new estimation principle for unnormalized statistical models, JMLR Workshop Conf. Proc., 9 (2010), pp. 297\u2013304."},{"key":"ref18","unstructured":"J. Z. HaoChen and T. Ma, A theoretical study of inductive biases in contrastive learning, in The Eleventh International Conference on Learning Representations ICLR, 2023, https:\/\/openreview.net\/forum?id=AuEgNlEAmed."},{"key":"ref19","first-page":"5000","volume":"34","author":"HaoChen J. Z.","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref20","first-page":"26889","volume-title":"NeurIPS","author":"HaoChen J. Z.","year":"2022"},{"key":"ref21","doi-asserted-by":"crossref","unstructured":"K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, Momentum contrast for unsupervised visual representation learning, in Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, IEEE Computer Society, Los Alamitos, CA, 2020, pp. 9729\u20139738.","DOI":"10.1109\/CVPR42600.2020.00975"},{"key":"ref22","doi-asserted-by":"crossref","first-page":"409","DOI":"10.1007\/978-1-4612-0865-5_26","volume-title":"The Collected Works of Wassily Hoeffding","author":"Hoeffding W.","year":"1994"},{"key":"ref23","doi-asserted-by":"crossref","first-page":"128645","DOI":"10.1016\/j.neucom.2024.128645","volume":"610","author":"Hu H.","year":"2024","journal-title":"Neurocomputing"},{"key":"ref24","unstructured":"W. Huang, M. Yi, X. Zhao, and Z. Jiang, Towards the generalization of contrastive self-supervised learning, in The Eleventh International Conference on Learning Representations, ICLR, 2023, https:\/\/openreview.net\/pdf?id=XDJwuEYHhme."},{"key":"ref25","unstructured":"L. Jing, P. Vincent, Y. LeCun, and Y. Tian, Understanding dimensional collapse in contrastive self-supervised learning, in Proceedings of the 10th International Conference on Learning Representations, ICLR, 2022, https:\/\/openreview.net\/forum?id=YevsQ05DEN7."},{"key":"ref26","unstructured":"A. Krizhevsky and G. Hinton, Learning Multiple Layers of Features from Tiny Images, Technical report, University of Toronto, 2009, https:\/\/api.semanticscholar.org\/CorpusID:18268744."},{"key":"ref27","doi-asserted-by":"crossref","first-page":"193907","DOI":"10.1109\/ACCESS.2020.3031549","volume":"8","author":"Le-Khac P. H.","year":"2020","journal-title":"IEEE Access"},{"key":"ref28","author":"LeCun Y.","year":"2010","journal-title":"MNIST Handwritten Digit Database"},{"key":"ref29","unstructured":"Y. Lei, T. Yang, Y. Ying, and D. Zhou, Generalization analysis for contrastive representation learning Proc. Mach. Learn. Res. (PMLR), 202 (2023), pp. 19200\u201319227."},{"key":"ref30","doi-asserted-by":"crossref","unstructured":"S. Marcel and Y. Rodriguez, Torchvision the machine-vision package of torch, in Proceedings of the 18th ACM International Conference on Multimedia, ACM, New York, 2010, pp. 1485\u20131488.","DOI":"10.1145\/1873951.1874254"},{"key":"ref31","doi-asserted-by":"crossref","unstructured":"D. McAllester, Simplified PAC-Bayesian margin bounds, in Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT\/Kernel 2003, Washington, DC, 2003, Springer, Berlin, 2003, pp. 203\u2013215.","DOI":"10.1007\/978-3-540-45167-9_16"},{"key":"ref32","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1023\/A:1021840411064","volume":"51","author":"McAllester D. A.","year":"2003","journal-title":"Mach. Learn."},{"key":"ref33","first-page":"148","volume":"141","author":"McDiarmid C.","year":"1989","journal-title":"Surveys Combin."},{"key":"ref34","unstructured":"K. Nozawa, P. Germain, and B. Guedj, PAC-Bayesian contrastive unsupervised representation learning, Proc. Mach. Learn. Res. (PMLR), 124 (2020), pp. 21\u201330."},{"key":"ref35","first-page":"5784","volume-title":"Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021","author":"Nozawa K.","year":"2021"},{"key":"ref36","unstructured":"A. v. d. Oord, Y. Li, and O. Vinyals, Representation Learning with Contrastive Predictive Coding, preprint, arXiv:1807.03748, 2018."},{"key":"ref37","unstructured":"A. Parulekar, L. Collins, K. Shanmugam, A. Mokhtari, and S. Shakkottai, InfoNCE loss provably learns cluster-preserving representations, Proc Mach. Learn. Res. (PMLR), 195 (2023), pp. 1914\u20131961."},{"key":"ref38","unstructured":"A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, Automatic differentiation in PyTorch, 31st Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, 2017."},{"key":"ref39","unstructured":"M. Perez-Ortiz, O. Rivasplata, B. Guedj, M. Gleeson, J. Zhang, J. Shawe-Taylor, M. Bober, and J. Kittler, Learning PAC-Bayes Priors for Probabilistic Neural Networks, preprint, arXiv:2109.10304, 2021."},{"key":"ref40","first-page":"1","volume":"22","author":"Perez-Ortiz M.","year":"2021","journal-title":"J. Mach. Learn. Res."},{"key":"ref41","doi-asserted-by":"crossref","unstructured":"J. Qiu, Q. Chen, Y. Dong, J. Zhang, H. Yang, M. Ding, K. Wang, and J. Tang, GCC: Graph contrastive coding for graph neural network pre-training, in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, New York, 2020, pp. 1150\u20131160.","DOI":"10.1145\/3394486.3403168"},{"key":"ref42","unstructured":"A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, and J. Clark, G. Krueger, and I. Sutskever, Learning transferable visual models from natural language supervision, Proc. Mach. Learn. Res. (PMLR), 2021, pp. 8748\u20138763."},{"key":"ref43","unstructured":"N. Saunshi, O. Plevrakis, S. Arora, M. Khodak, and H. Khandeparkar, A theoretical analysis of contrastive unsupervised representation learning, Proc. Mach. Learn. Res. (PMLR), 2019, pp. 5628\u20135637."},{"key":"ref44","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1162\/153244303765208377","volume":"3","author":"Seeger M.","year":"2002","journal-title":"J. Mach. Learn. Res."},{"key":"ref45","first-page":"33965","volume-title":"in NIPS 2023","author":"Shwartz-Ziv R.","year":"2023"},{"key":"ref46","doi-asserted-by":"crossref","first-page":"252","DOI":"10.3390\/e26030252","volume":"26","author":"Shwartz-Ziv R.","year":"2024","journal-title":"Entropy"},{"key":"ref47","first-page":"1857","volume":"29","author":"Sohn K.","year":"2016","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref48","unstructured":"Z. Tan, Y. Zhang, J. Yang, and Y. Yuan, Contrastive learning is spectral clustering on similarity graph, in The Twelfth International Conference on Learning Representations, ICLR, 2024, https:\/\/openreview.net\/forum?id=hLZQTFGToA."},{"key":"ref49","unstructured":"C. Tosh, A. Krishnamurthy, and D. Hsu, Contrastive learning, multi-view redundancy, and linear models, Proc. Mach. Learn. Res. (PMLR), 132 (2021), pp. 1179\u20131206."},{"key":"ref50","unstructured":"Y. H. Tsai, Y. Wu, R. Salakhutdinov, and L. Morency, Self-supervised learning from a multi-view perspective, in 9th International Conference on Learning Representations, ICLR, 2021, https:\/\/openreview.net\/forum?id=-bdp_8Itjwp."},{"key":"ref51","unstructured":"Y. Wang, Q. Zhang, Y. Wang, J. Yang, and Z. Lin, Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap, in Proceedings of the 10th International Conference on Learning Representations, ICLR, 2022, https:\/\/openreview.net\/forum?id=ECvgmYVyeUz."},{"key":"ref52","unstructured":"C. Wei, K. Shen, Y. Chen, and T. Ma, Theoretical analysis of self-training with deep networks on unlabeled data, in 9th International Conference on Learning Representations, ICLR, 2021, https:\/\/openreview.net\/forum?id=rC8sJ4i6kaH."},{"key":"ref53","doi-asserted-by":"crossref","unstructured":"Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, Unsupervised feature learning via non-parametric instance discrimination, in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, IEEE Computer Society, Los Alamitos, CA, 2018, pp. 3733\u20133742.","DOI":"10.1109\/CVPR.2018.00393"},{"key":"ref54","first-page":"913","volume":"36","author":"Yu J.","year":"2023","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref55","unstructured":"J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, Barlow twins: Self-supervised learning via redundancy reduction, Proc. Mach. Learn. Res. (PMLR), 139 (2021), pp. 12310\u201312320."},{"key":"ref56","unstructured":"Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz, Contrastive learning of medical visual representations from paired images and text, Proc. Mach. Learn. Res. (PMLR), 182 (2022), pp. 2\u201325."},{"key":"ref57","volume-title":"ICLR","author":"Zhou W.","year":"2019"},{"key":"ref58","first-page":"1","volume":"24","author":"Zou X.","year":"2023","journal-title":"J. Mach. Learn. Res."}],"container-title":["SIAM Journal on Mathematics of Data Science"],"original-title":[],"language":"en","deposited":{"date-parts":[[2025,12,5]],"date-time":"2025-12-05T09:00:46Z","timestamp":1764925246000},"score":1,"resource":{"primary":{"URL":"https:\/\/epubs.siam.org\/doi\/10.1137\/24M1715283"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,5]]},"references-count":58,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,12,31]]}},"alternative-id":["10.1137\/24M1715283"],"URL":"https:\/\/doi.org\/10.1137\/24m1715283","relation":{},"ISSN":["2577-0187"],"issn-type":[{"value":"2577-0187","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,5]]}}}