{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T15:42:23Z","timestamp":1786981343852,"version":"build-2736575974"},"reference-count":41,"publisher":"Springer Science and Business Media LLC","issue":"12","license":[{"start":{"date-parts":[[2022,11,4]],"date-time":"2022-11-04T00:00:00Z","timestamp":1667520000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,11,4]],"date-time":"2022-11-04T00:00:00Z","timestamp":1667520000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002341","name":"Academy of Finland","doi-asserted-by":"publisher","award":["347707"],"award-info":[{"award-number":["347707"]}],"id":[{"id":"10.13039\/501100002341","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002666","name":"Aalto University","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002666","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Extreme Multilabel Text Classification (XMTC) is a text classification problem in which, (i) the output space is extremely large, (ii) each data point may have multiple positive labels, and (iii) the data follows a strongly imbalanced distribution. With applications in recommendation systems and automatic tagging of web-scale documents, the research on XMTC has been focused on improving prediction accuracy and dealing with imbalanced data. However, the robustness of deep learning based XMTC models against adversarial examples has been largely underexplored. In this paper, we investigate the behaviour of XMTC models under adversarial attacks. To this end, first, we define adversarial attacks in multilabel text classification problems. We categorize attacking multilabel text classifiers as (a) positive-to-negative, where the target positive label should fall out of top-k predicted labels, and (b) negative-to-positive, where the target negative label should be among the top-k predicted labels. Then, by experiments on APLC-XLNet and AttentionXML, we show that XMTC models are highly vulnerable to positive-to-negative attacks but more robust to negative-to-positive ones. Furthermore, our experiments show that the success rate of positive-to-negative adversarial attacks has an imbalanced distribution. More precisely, tail classes are highly vulnerable to adversarial attacks for which an attacker can generate adversarial samples with high similarity to the actual data-points. To overcome this problem, we explore the effect of rebalanced loss functions in XMTC where not only do they increase accuracy on tail classes, but they also improve the robustness of these classes against adversarial attacks. The code for our experiments is available at\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/xmc-aalto\/adv-xmtc\">https:\/\/github.com\/xmc-aalto\/adv-xmtc<\/jats:ext-link>\n                    .\n                  <\/jats:p>","DOI":"10.1007\/s10994-022-06263-z","type":"journal-article","created":{"date-parts":[[2022,11,4]],"date-time":"2022-11-04T17:03:56Z","timestamp":1667581436000},"page":"4539-4563","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Adversarial examples for extreme multilabel text classification"],"prefix":"10.1007","volume":"111","author":[{"given":"Mohammadreza","family":"Qaraei","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3787-8971","authenticated-orcid":false,"given":"Rohit","family":"Babbar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,11,4]]},"reference":[{"key":"6263_CR1","doi-asserted-by":"crossref","unstructured":"Agrawal, R., Gupta, A., Prabhu, Y., & Varma, M. (2013). Multi-label learning with millions of labels: Recommending advertiser bid phrases for web pages. In Proceedings of the 22nd international conference on World Wide Web, pp. 13\u201324.","DOI":"10.1145\/2488388.2488391"},{"issue":"1","key":"6263_CR2","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1145\/2674026.2674033","volume":"16","author":"R Babbar","year":"2014","unstructured":"Babbar, R., Metzig, C., Partalas, I., Gaussier, E., & Amini, M. R. (2014). On power law distributions in large-scale taxonomies. ACM SIGKDD Explorations Newsletter, 16(1), 47\u201356.","journal-title":"ACM SIGKDD Explorations Newsletter"},{"key":"6263_CR3","doi-asserted-by":"crossref","unstructured":"Babbar, R., & Sch\u00f6lkopf, B. (2017). Dismec: Distributed sparse machines for extreme multi-label classification. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, pp. 721\u2013729.","DOI":"10.1145\/3018661.3018741"},{"issue":"8","key":"6263_CR4","doi-asserted-by":"publisher","first-page":"1329","DOI":"10.1007\/s10994-019-05791-5","volume":"108","author":"R Babbar","year":"2019","unstructured":"Babbar, R., & Sch\u00f6lkopf, B. (2019). Data scarcity, robustness and extreme multi-label classification. Machine Learning, 108(8), 1329\u20131351.","journal-title":"Machine Learning"},{"key":"6263_CR5","unstructured":"Bhatia, K., Dahiya, K., Jain, H., Kar, P., Mittal, A., Prabhu, Y., & Varma, M. (2016). The extreme classification repository: Multi-label datasets and code."},{"key":"6263_CR6","unstructured":"Bhatia, K., Jain, H., Kar, P., Varma, M., & Jain, P. (2015). Sparse local embeddings for extreme multi-label classification. In NIPS."},{"key":"6263_CR7","unstructured":"Brama, H., Dery, L., & Grinshpoun, T. (2022). Evaluation of neural networks defenses and attacks using ndcg and reciprocal rank metrics. http:\/\/arxiv.org\/abs\/2201.05071."},{"key":"6263_CR8","doi-asserted-by":"crossref","unstructured":"Cer, D., Yang, Y., Kong, S.y., Hua, N., Limtiaco, N., John, R.S., Constant, N., Guajardo-C\u00e9spedes, M., Yuan, S., & Tar, C. et\u00a0al. (2018). Universal sentence encoder. http:\/\/arxiv.org\/abs\/1803.11175.","DOI":"10.18653\/v1\/D18-2029"},{"key":"6263_CR9","doi-asserted-by":"crossref","unstructured":"Cui, Y., Jia, M., Lin, T.Y., Song, Y., & Belongie, S. (2019). Class-balanced loss based on effective number of samples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9268\u20139277.","DOI":"10.1109\/CVPR.2019.00949"},{"key":"6263_CR10","unstructured":"Devlin, J., Chang, M.W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. http:\/\/arxiv.org\/abs\/1810.04805."},{"key":"6263_CR11","doi-asserted-by":"crossref","unstructured":"Gao, J., Lanchantin, J., Soffa, M.L., & Qi, Y. (2018). Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pp. 50\u201356. IEEE.","DOI":"10.1109\/SPW.2018.00016"},{"key":"6263_CR12","doi-asserted-by":"crossref","unstructured":"Garg, S., & Ramakrishnan, G. (2020). Bae: Bert-based adversarial examples for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6174\u20136181.","DOI":"10.18653\/v1\/2020.emnlp-main.498"},{"key":"6263_CR13","unstructured":"Goodfellow, I., Shlens, J., & Szegedy, C. (2015). Explaining and harnessing adversarial examples. In International Conference on Learning Representations."},{"key":"6263_CR14","doi-asserted-by":"crossref","unstructured":"Hu, S., Ke L., Wang, X., & Lyu, S. (2021). T$$_k$$ML-AP: Adversarial attacks to top-k multi-label learning. http:\/\/arxiv.org\/abs\/2108.00146.","DOI":"10.1109\/ICCV48922.2021.00755"},{"key":"6263_CR15","doi-asserted-by":"crossref","unstructured":"Jain, H., Balasubramanian, V., Chunduri, B., Varma, M. (2019). Slice: Scalable linear extreme classifiers trained on 100 million labels for related searches. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp. 528\u2013536.","DOI":"10.1145\/3289600.3290979"},{"key":"6263_CR16","doi-asserted-by":"crossref","unstructured":"Jain, H., Prabhu, Y., Varma, M. (2016), August. Extreme multi-label loss functions for recommendation, tagging, ranking and other missing label applications. In KDD.","DOI":"10.1145\/2939672.2939756"},{"key":"6263_CR17","doi-asserted-by":"crossref","unstructured":"Jiang, T., Wang, D., Sun, L., Yang, H., Zhao, Z., & Zhuang, F. (2021). Lightxml: Transformer with dynamic negative sampling for high-performance extreme multi-label text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, 35, 7987\u20137994.","DOI":"10.1609\/aaai.v35i9.16974"},{"key":"6263_CR18","doi-asserted-by":"crossref","unstructured":"Jin, D., Jin, Z., Zhou, J. T., & Szolovits, P. (2020). Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI Conference on Artificial Intelligence, 34, 8018\u20138025.","DOI":"10.1609\/aaai.v34i05.6311"},{"key":"6263_CR19","first-page":"1","volume":"87","author":"S Khandagale","year":"2020","unstructured":"Khandagale, S., Xiao, H., & Babbar, R. (2020). Bonsai: diverse and shallow trees for extreme multi-label classification. Machine Learning, 87, 1\u201321.","journal-title":"Machine Learning"},{"key":"6263_CR20","doi-asserted-by":"crossref","unstructured":"Li, J., S.\u00a0Ji, Du, T., Li, B., & Wang, T. (2018). Textbugger: Generating adversarial text against real-world applications. http:\/\/arxiv.org\/abs\/1812.05271.","DOI":"10.14722\/ndss.2019.23138"},{"key":"6263_CR21","doi-asserted-by":"crossref","unstructured":"Li, L., Ma, R., Guo, Q., Xue, X., & Qiu, X. (2020). Bert-attack: Adversarial attack against bert using bert. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6193\u20136202.","DOI":"10.18653\/v1\/2020.emnlp-main.500"},{"key":"6263_CR22","first-page":"13265","volume":"32","author":"TKR Medini","year":"2019","unstructured":"Medini, T. K. R., Huang, Q., Wang, Y., Mohan, V., & Shrivastava, A. (2019). Extreme classification in log memory using count-min sketch: A case study of amazon search with 50m products. Advances in Neural Information Processing Systems, 32, 13265\u201313275.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"6263_CR23","unstructured":"Melacci, S., Ciravegna, G., Sotgiu, A., Demontis, A., Biggio, B., Gori, M., & Roli, F. (2020). Can domain knowledge alleviate adversarial attacks in multi-label classifiers? http:\/\/arxiv.org\/abs\/2006.03833."},{"key":"6263_CR24","doi-asserted-by":"crossref","unstructured":"Mittal, A., Dahiya, K., Agrawal, S., Saini, D., Agarwal, S., Kar, P., & Varma, M. (2021). Decaf: Deep extreme classification with label features. Proceedings of the 14th ACM International Conference on Web Search and Data Mining .","DOI":"10.1145\/3437963.3441807"},{"key":"6263_CR25","doi-asserted-by":"crossref","unstructured":"Partalas, I., Kosmopoulos, A., Baskiotis, N., Artieres, T., Paliouras, G., Gaussier, E., Androutsopoulos, I., Amini, M.R., & Galinari, P. (2015). Lshtc: A benchmark for large-scale text classification. http:\/\/arxiv.org\/abs\/1503.08581.","DOI":"10.1145\/2556195.2556208"},{"key":"6263_CR26","doi-asserted-by":"crossref","unstructured":"Prabhu, Y., Kag, A., Harsola, S., Agrawal, R., & Varma, M. (2018). Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. In Proceedings of the 2018 World Wide Web Conference, pp. 993\u20131002.","DOI":"10.1145\/3178876.3185998"},{"key":"6263_CR27","first-page":"3711","volume":"2021","author":"M Qaraei","year":"2021","unstructured":"Qaraei, M., Schultheis, E., Gupta, P., & Babbar, R. (2021). Convex surrogates for unbiased loss functions in extreme classification with missing labels. In Proceedings of the Web Conference, 2021, 3711\u20133720.","journal-title":"In Proceedings of the Web Conference"},{"key":"6263_CR28","doi-asserted-by":"crossref","unstructured":"Ren, S., Deng, Y., He, K., & Che, W. (2019). Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 1085\u20131097.","DOI":"10.18653\/v1\/P19-1103"},{"key":"6263_CR29","doi-asserted-by":"crossref","unstructured":"Song, Q., Jin, H., Huang, X., & Hu, X. (2018). Multi-label adversarial perturbations. In 2018 IEEE International Conference on Data Mining (ICDM), pp. 1242\u20131247. IEEE.","DOI":"10.1109\/ICDM.2018.00166"},{"key":"6263_CR30","doi-asserted-by":"crossref","unstructured":"Song, Y., Liu, Z., & Zhang, C. (2021). Multi-label text classification and text adversarial attack. In 2021 International Conference on Intelligent Computing, Automation and Applications (ICAA), pp. 532\u2013536. IEEE.","DOI":"10.1109\/ICAA53760.2021.00098"},{"key":"6263_CR31","unstructured":"Sun, L., Hashimoto, K., Yin, W., Asai, A., Li, J., Yu, P., & Xiong, C. (2020). Adv-bert: Bert is not robust on misspellings! generating nature adversarial samples on bert. http:\/\/arxiv.org\/abs\/2003.04985."},{"key":"6263_CR32","unstructured":"Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2014). Intriguing properties of neural networks. In International Conference on Learning Representations."},{"key":"6263_CR33","unstructured":"Wang, W., Xu, H., Liu, X., Li, Y., Thuraisingham, B., Tang, J. (2021). Imbalanced adversarial training with reweighting. http:\/\/arxiv.org\/abs\/2107.13639."},{"key":"6263_CR34","doi-asserted-by":"crossref","unstructured":"Wu, T., Liu, Z., Huang, Q., Wang, Y., & Lin, D. (2021). Adversarial robustness under long-tailed distribution. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, pp. 8659\u20138668.","DOI":"10.1109\/CVPR46437.2021.00855"},{"key":"6263_CR35","doi-asserted-by":"crossref","unstructured":"Wu, Y., Bamman, D., & Russell, S. (2017). Adversarial training for relation extraction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1778\u20131783.","DOI":"10.18653\/v1\/D17-1187"},{"key":"6263_CR36","unstructured":"Xu, L., & Veeramachaneni, K. (2021). Attacking text classifiers via sentence rewriting sampler. http:\/\/arxiv.org\/abs\/2104.08453."},{"key":"6263_CR37","unstructured":"Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q.V. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. http:\/\/arxiv.org\/abs\/1906.08237."},{"key":"6263_CR38","unstructured":"Yang, Z., Han, Y., & Zhang, X. (2020). Characterizing the evasion attackability of multi-label classifiers. http:\/\/arxiv.org\/abs\/2012.09427."},{"key":"6263_CR39","unstructured":"Ye, H., Chen, Z., Wang, D.H., & Davison, B. (2020). Pretrained generalized autoregressive model with adaptive probabilistic label clusters for extreme multi-label text classification. In International Conference on Machine Learning, pp. 10809\u201310819. PMLR."},{"key":"6263_CR40","unstructured":"You, R., Zhang, Z., Wang, Z., Dai, S., Mamitsuka, H., & Zhu, S. (2019). Attentionxml: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. In NeurIPS, pp. 5812\u20135822."},{"issue":"3","key":"6263_CR41","first-page":"1","volume":"11","author":"WE Zhang","year":"2020","unstructured":"Zhang, W. E., Sheng, Q. Z., Alhazmi, A., & Li, C. (2020). Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3), 1\u201341.","journal-title":"ACM Transactions on Intelligent Systems and Technology (TIST)"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-022-06263-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-022-06263-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-022-06263-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,29]],"date-time":"2022-11-29T17:27:45Z","timestamp":1669742865000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-022-06263-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,4]]},"references-count":41,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["6263"],"URL":"https:\/\/doi.org\/10.1007\/s10994-022-06263-z","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,4]]},"assertion":[{"value":"15 December 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 July 2022","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 September 2022","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 November 2022","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflicts of interests to disclose relevant to this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"We give our consent for the publication of identifiable details in this paper, which can include figures, tables, and the results, in other scientific projects by mentioning the reference.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}}]}}