{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,2]],"date-time":"2026-08-02T18:01:46Z","timestamp":1785693706465,"version":"3.56.0"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2025,2,24]],"date-time":"2025-02-24T00:00:00Z","timestamp":1740355200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,2,24]],"date-time":"2025-02-24T00:00:00Z","timestamp":1740355200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003790","name":"Hiroshima University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100003790","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Knowl Inf Syst"],"published-print":{"date-parts":[[2025,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>In our study, we propose a self-supervised neural topic model (NTM) that combines the power of NTMs and regularized self-supervised learning methods to improve performance. NTMs use neural networks to learn latent topics hidden behind the words in documents, enabling greater flexibility and the ability to estimate more coherent topics compared to traditional topic models. On the other hand, some self-supervised learning methods use a joint embedding architecture with two identical networks that produce similar representations for two augmented versions of the same input. Regularizations are applied to these representations to prevent collapse, which would otherwise result in the networks outputting constant or redundant representations for all inputs. Our model enhances topic quality by explicitly regularizing latent topic representations of anchor and positive samples. We also introduced an adversarial data augmentation method to replace the heuristic sampling method. We further developed several variation models including those on the basis of an NTM that incorporates contrastive learning with both positive and negative samples. Experimental results on three datasets showed that our models outperformed baselines and state-of-the-art models both quantitatively and qualitatively.<\/jats:p>","DOI":"10.1007\/s10115-025-02367-9","type":"journal-article","created":{"date-parts":[[2025,2,24]],"date-time":"2025-02-24T17:49:23Z","timestamp":1740419363000},"page":"5057-5075","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Self-supervised learning for neural topic models with variance\u2013invariance\u2013covariance regularization"],"prefix":"10.1007","volume":"67","author":[{"given":"Weiran","family":"Xu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kengo","family":"Hirami","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Koji","family":"Eguchi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,2,24]]},"reference":[{"key":"2367_CR1","doi-asserted-by":"crossref","unstructured":"Akiba T, Sano S, Yanase T et\u00a0al (2019) Optuna: A next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp 2623\u20132631","DOI":"10.1145\/3292500.3330701"},{"key":"2367_CR2","doi-asserted-by":"crossref","unstructured":"Bai H, Chen Z, Lyu MR et\u00a0al (2018) Neural relational topic models for scientific article analysis. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp 27\u201336","DOI":"10.1145\/3269206.3271696"},{"key":"2367_CR3","unstructured":"Bardes A, Ponce J, Lecun Y (2022) Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In: Proceedings of the 10th International Conference on Learning Representations"},{"key":"2367_CR4","doi-asserted-by":"crossref","unstructured":"Bianchi F, Terragni S, Hovy D (2021) Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp 759\u2013766","DOI":"10.18653\/v1\/2021.acl-short.96"},{"issue":"4","key":"2367_CR5","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1145\/2133806.2133826","volume":"55","author":"DM Blei","year":"2012","unstructured":"Blei DM (2012) Probabilistic topic models. Commun ACM 55(4):77\u201384","journal-title":"Commun ACM"},{"issue":"1","key":"2367_CR6","first-page":"993","volume":"3","author":"DM Blei","year":"2003","unstructured":"Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res 3(1):993\u20131022","journal-title":"J Mach Learn Res"},{"key":"2367_CR7","doi-asserted-by":"crossref","unstructured":"Card D, Tan C, Smith NA (2018) Neural models for documents with metadata. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 2031\u20132040","DOI":"10.18653\/v1\/P18-1189"},{"key":"2367_CR8","unstructured":"Devlin J, Chang MW, Lee K et\u00a0al (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (Long and Short Papers), pp 4171\u20134186"},{"key":"2367_CR9","first-page":"439","volume":"8","author":"AB Dieng","year":"2020","unstructured":"Dieng AB, Ruiz FJ, Blei DM (2020) Topic modeling in embedding spaces. Transa Assoc Comput Ling 8:439\u2013453","journal-title":"Transa Assoc Comput Ling"},{"key":"2367_CR10","doi-asserted-by":"crossref","unstructured":"Han S, Shin M, Park S et\u00a0al (2023) Unified neural topic model via contrastive learning and term weighting. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp 1802\u20131817","DOI":"10.18653\/v1\/2023.eacl-main.132"},{"key":"2367_CR11","doi-asserted-by":"crossref","unstructured":"Hoyle AM, Goel P, Resnik P (2020) Improving neural topic models using knowledge distillation. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp 1752\u20131771","DOI":"10.18653\/v1\/2020.emnlp-main.137"},{"key":"2367_CR12","doi-asserted-by":"crossref","unstructured":"Lang K (1995) Newsweeder: Learning to filter netnews. In: Proceedings of the 12th International Conference on International Conference on Machine Learning, pp 331\u2013339","DOI":"10.1016\/B978-1-55860-377-6.50048-7"},{"key":"2367_CR13","doi-asserted-by":"crossref","unstructured":"Lau JH, Newman D, Baldwin T (2014) Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In: Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp 530\u2013539","DOI":"10.3115\/v1\/E14-1056"},{"key":"2367_CR14","unstructured":"Maas A, Daly RE, Pham PT et\u00a0al (2011) Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp 142\u2013150"},{"key":"2367_CR15","first-page":"1280","volume":"20","author":"J Mcauliffe","year":"2007","unstructured":"Mcauliffe J, Blei D (2007) Supervised topic models. Adv Neural Inf Process Syst 20:1280\u20131287","journal-title":"Adv Neural Inf Process Syst"},{"key":"2367_CR16","unstructured":"Merity S, Xiong C, Bradbury J et\u00a0al (2017) Pointer sentinel mixture models. In: Proceedings of the 5th International Conference on Learning Representations"},{"key":"2367_CR17","unstructured":"Miao Y, Grefenstette E, Blunsom P (2017) Discovering discrete latent topics with neural variational inference. In: Proceedings of the 34th International Conference on Machine Learning, pp 2410\u20132419"},{"key":"2367_CR18","first-page":"11974","volume":"34","author":"T Nguyen","year":"2021","unstructured":"Nguyen T, Luu AT (2021) Contrastive learning for neural topic model. Adv Neural Inf Process Syst 34:11974\u201311986","journal-title":"Adv Neural Inf Process Syst"},{"key":"2367_CR19","unstructured":"Srivastava A, Sutton C (2017) Autoencoding variational inference for topic models. In: Proceedings of the 5th International Conference on Learning Representations"},{"key":"2367_CR20","doi-asserted-by":"crossref","unstructured":"Suzuki T (2022) Teachaugment: Data augmentation optimization using teacher knowledge. In: Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp 10904\u201310914","DOI":"10.1109\/CVPR52688.2022.01063"},{"key":"2367_CR21","first-page":"4460","volume":"2024","author":"YK Tang","year":"2024","unstructured":"Tang YK, Huang H, Shi X et al (2024) Beyond labels and topics: discovering causal relationships in neural topic modeling. Proc ACM Web Conf 2024:4460\u20134469","journal-title":"Proc ACM Web Conf"},{"key":"2367_CR22","doi-asserted-by":"crossref","unstructured":"Terragni S, Fersini E, Messina E (2021) Word embedding-based topic similarity measures. In: Natural Language Processing and Information Systems, pp 33\u201345","DOI":"10.1007\/978-3-030-80599-9_4"},{"key":"2367_CR23","doi-asserted-by":"crossref","unstructured":"Wang Y, Li X, Ouyang J (2021a) Layer-assisted neural topic modeling over document networks. In: Proceedings of the 30th International Joint Conference on Artificial Intelligence, pp 3148\u20133154","DOI":"10.24963\/ijcai.2021\/433"},{"key":"2367_CR24","first-page":"18","volume":"2021","author":"Y Wang","year":"2021","unstructured":"Wang Y, Li X, Zhou X et al (2021) Extracting topics with simultaneous word co-occurrence and semantic correlation graphs: neural topic modeling for short texts. Find Assoc Comput Ling: EMNLP 2021:18\u201327","journal-title":"Find Assoc Comput Ling: EMNLP"},{"key":"2367_CR25","unstructured":"Welling M, Kingma DP (2014) Auto-encoding variational bayes. In: Proceedings of the 2nd International Conference on Learning Representations"},{"key":"2367_CR26","doi-asserted-by":"crossref","unstructured":"Wu X, Luu AT, Dong X (2022) Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp 2748\u20132760","DOI":"10.18653\/v1\/2022.emnlp-main.176"},{"key":"2367_CR27","unstructured":"Wu X, Dong X, Nguyen T et\u00a0al (2023) Effective neural topic modeling with embedding clustering regularization. In: Proceedings of the 40th International Conference on Machine Learning, pp 37335\u201337357"},{"issue":"2","key":"2367_CR28","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1007\/s10462-023-10661-7","volume":"57","author":"X Wu","year":"2024","unstructured":"Wu X, Nguyen T, Luu AT (2024) A survey on neural topic models: methods, applications, and challenges. Artif Intell Rev 57(2):18","journal-title":"Artif Intell Rev"},{"key":"2367_CR29","unstructured":"Zbontar J, Jing L, Misra I et\u00a0al (2021) Barlow twins: Self-supervised learning via redundancy reduction. In: Proceedings of the 38th International Conference on Machine Learning, pp 12310\u201312320"},{"key":"2367_CR30","first-page":"29498","volume":"35","author":"DC Zhang","year":"2022","unstructured":"Zhang DC, Lauw H (2022) Meta-complementing the semantics of short texts in neural topic models. Adv Neural Inf Process Syst 35:29498\u201329511","journal-title":"Adv Neural Inf Process Syst"},{"key":"2367_CR31","doi-asserted-by":"crossref","unstructured":"Zhao H, Phung D, Huynh V et\u00a0al (2021) Topic modelling meets deep neural networks: A survey. In: Proceedings of the 30th International Joint Conference on Artificial Intelligence, pp 4713\u20134720","DOI":"10.24963\/ijcai.2021\/638"}],"container-title":["Knowledge and Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10115-025-02367-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10115-025-02367-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10115-025-02367-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,5,20]],"date-time":"2025-05-20T12:36:56Z","timestamp":1747744616000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10115-025-02367-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,24]]},"references-count":31,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,6]]}},"alternative-id":["2367"],"URL":"https:\/\/doi.org\/10.1007\/s10115-025-02367-9","relation":{},"ISSN":["0219-1377","0219-3116"],"issn-type":[{"value":"0219-1377","type":"print"},{"value":"0219-3116","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,24]]},"assertion":[{"value":"26 October 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 September 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 February 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 February 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"This paper does not contain any studies with human participants or animals performed by the authors.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}}]}}