{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,31]],"date-time":"2025-12-31T18:45:10Z","timestamp":1767206710902,"version":"build-2238731810"},"reference-count":48,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,8,4]],"date-time":"2025-08-04T00:00:00Z","timestamp":1754265600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>Generalizing deep learning models to unseen target domains with low latency has motivated research into test-time training\/adaptation (TTT\/TTA). However, deploying TTT\/TTA in open-world environments is challenging due to the difficulty in distinguishing between strong out-of-distribution (OOD) samples and regular weak OOD samples. While emerging Open-World TTT (OWTTT) approaches address this challenge, they introduce a new vulnerability: test-time poisoning attacks. These attacks differ fundamentally from traditional poisoning attacks that occur during model training, as adversaries cannot intervene in the training process itself.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>In response to this threat, we design a novel test-time poisoning attack method specifically targeting OWTTT models. Capitalizing on the fact that model gradients dynamically change during testing, our method employs a single-step query-based approach to dynamically generate and update adversarial perturbations. These perturbations are then input into the OWTTT model during its adaptation phase.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We extensively test our attack method on an OWTTT model. The experimental results demonstrate a significant vulnerability, showing that the OWTTT model's performance can be effectively compromised by our test-time poisoning attack.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Our findings reveal that OWTTT algorithms lacking rigorous security assessment against such attacks are unsuitable for real-world deployment. Consequently, we strongly advocate for the integration of defenses against test-time poisoning attacks into the fundamental design of future open-world test-time training methodologies.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/frai.2025.1621025","type":"journal-article","created":{"date-parts":[[2025,8,4]],"date-time":"2025-08-04T05:26:56Z","timestamp":1754285216000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Research on the robustness of the open-world test-time training model"],"prefix":"10.3389","volume":"8","author":[{"given":"Shu","family":"Pi","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xin","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiatian","family":"Pi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2025,8,4]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1007\/s10994-009-5152-4","article-title":"A theory of learning from different domains","volume":"79","author":"Ben-David","year":"2010","journal-title":"Mach. Learn"},{"key":"B2","article-title":"\u201cPoisoning attacks against support vector machines,\u201d","author":"Biggio","year":"2012","journal-title":"International Conference on Machine Learning"},{"key":"B3","article-title":"\u201cDecision-based adversarial attacks:reliable attacks against black-box machine learning models,\u201d","author":"Brendel","year":"2018","journal-title":"International Conference on Learning Representations"},{"key":"B4","article-title":"\u201cPoisoning and backdooring contrastive learning,\u201d","author":"Carlini","year":"2022","journal-title":"International Conference on Learning Representations (ICLR)"},{"key":"B5","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1109\/SP.2017.49","article-title":"\u201cTowards evaluating the robustness of neural networks,\u201d","volume-title":"2017 IEEE Symposium on Security and Privacy (SP)","author":"Carlini","year":"2017"},{"key":"B6","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1109\/CVPR52688.2022.00039","article-title":"\u201cContrastive test-time adaptation,\u201d","author":"Chen","year":"2022","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B7","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1145\/3128572.3140448","article-title":"\u201cZoo: zeroth order optimization based black-box attacks to deep neural networks without training substitute models,\u201d","author":"Chen","year":"2017","journal-title":"Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security"},{"key":"B8","article-title":"Houdini: fooling deep structured prediction models","author":"Cisse","year":"2017","journal-title":"arXiv preprint arXiv:1707.05373"},{"key":"B9","doi-asserted-by":"crossref","first-page":"1306","DOI":"10.1109\/SP54263.2024.00072","article-title":"\u201cTest-time poisoning attacks against test-time adaptation models,\u201d","volume-title":"2024 IEEE Symposium on Security and Privacy (SP)","author":"Cong","year":"2024"},{"key":"B10","first-page":"4421","article-title":"\u201cEvaluating the adversarial robustness of adaptive test-time defenses,\u201d","author":"Croce","year":"2022","journal-title":"International Conference on Machine Learning"},{"key":"B11","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1109\/CVPR.2009.5206848","article-title":"\u201cImagenet: a large-scale hierarchical image database,\u201d","volume-title":"2009 IEEE Conference on Computer Vision and Pattern Recognition","author":"Deng","year":"2009"},{"key":"B12","first-page":"1180","article-title":"\u201cUnsupervised domain adaptation by backpropagation,\u201d","volume-title":"International Conference on Machine Learning","author":"Ganin","year":"2015"},{"key":"B13","article-title":"Explaining and harnessing adversarial examples","author":"Goodfellow","year":"2014","journal-title":"arXiv preprint arXiv:1412.6572"},{"key":"B14","first-page":"6204","article-title":"\u201cTest time adaptation via conjugate pseudo-labels,\u201d","author":"Goyal","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B15","article-title":"Machine learning as an adversarial service: learning black-box adversarial examples","author":"Hayes","year":"2017","journal-title":"arXiv preprint arXiv:1708.05207"},{"key":"B16","doi-asserted-by":"publisher","first-page":"770","DOI":"10.1109\/CVPR.2016.90","article-title":"\u201cDeep residual learning for image recognition,\u201d","author":"He","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B17","doi-asserted-by":"publisher","first-page":"8340","DOI":"10.1109\/ICCV48922.2021.00823","article-title":"\u201cThe many faces of robustness: a critical analysis of out-of-distribution generalization,\u201d","author":"Hendrycks","year":"2021","journal-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision"},{"key":"B18","article-title":"Benchmarking neural network robustness to common corruptions and perturbations","author":"Hendrycks","year":"2019","journal-title":"arXiv preprint arXiv:1903.12261"},{"key":"B19","article-title":"\u201cTest-time learning for large language models,\u201d","author":"Hu","year":"2025","journal-title":"International Conference on Machine Learning (ICML)"},{"key":"B20","volume-title":"Dataset Shift in Machine Learning","author":"Joaquin","year":"2008"},{"key":"B21","article-title":"Sita: single image test-time adaptation","author":"Khurana","year":"2021","journal-title":"arXiv preprint arXiv:2112.02355"},{"key":"B22","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1201\/9781351251389-8","article-title":"\u201cAdversarial examples in the physical world,\u201d","volume-title":"Artificial Intelligence Safety and Security","author":"Kurakin","year":"2018"},{"key":"B23","doi-asserted-by":"publisher","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"2002","journal-title":"Proc. IEEE"},{"key":"B24","article-title":"\u201cTest-time adaptation with binary feedback,\u201d","author":"Lee","year":"2025","journal-title":"International Conference on Machine Learning (ICML)"},{"key":"B25","doi-asserted-by":"publisher","first-page":"11836","DOI":"10.1109\/ICCV51070.2023.01087","article-title":"\u201cOn the robustness of open-world test-time training: self-training with dynamic prototype expansion,\u201d","author":"Li","year":"2023","journal-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision"},{"key":"B26","first-page":"6028","article-title":"\u201cDo we really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation,\u201d","volume-title":"International Conference on Machine Learning","author":"Liang","year":"2020"},{"key":"B27","article-title":"Delving into transferable adversarial examples and black-box attacks","author":"Liu","year":"2016","journal-title":"arXiv preprint arXiv:1611.02770"},{"key":"B28","first-page":"21808","article-title":"\u201cTTT++: when does self-supervised test-time training fail or thrive?\u201d","author":"Liu","year":"2021","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B29","doi-asserted-by":"publisher","first-page":"2574","DOI":"10.1109\/CVPR.2016.282","article-title":"\u201cDeepfool: a simple and accurate method to fool deep neural networks,\u201d","author":"Moosavi-Dezfooli","year":"2016","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"B30","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1145\/3128572.3140451","article-title":"\u201cTowards poisoning of deep learning algorithms with backgradient optimization,\u201d","author":"Mu noz-Gonz\u00e1lez","year":"2017","journal-title":"Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security"},{"key":"B31","article-title":"Simple black-box adversarial perturbations for deep networks","author":"Narodytska","year":"2016","journal-title":"arXiv preprint arXiv:1612.06299"},{"key":"B32","article-title":"Biologically inspired protection of deep networks from adversarial attacks","author":"Nayebi","year":"2017","journal-title":"arXiv preprint arXiv:1703.09202"},{"key":"B33","first-page":"4","article-title":"\u201cReading digits in natural images with unsupervised feature learning,\u201d","volume-title":"NIPS Workshop on Deep Learning and Unsupervised Feature Learning","author":"Netzer","year":"2011"},{"key":"B34","article-title":"Test-time model adaptation with only forward passes","author":"Niu","year":"2024","journal-title":"arXiv preprint arXiv:2404.01650"},{"key":"B35","article-title":"Trojanzoo: everything you ever wanted to know about neural backdoors (but were afraid to ask)","author":"Pang","year":"2020","journal-title":"arXiv preprint arXiv:2012.09302"},{"key":"B36","first-page":"2899","article-title":"\u201cAccumulative poisoning attacks on real-time data,\u201d","author":"Pang","year":"2021","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B37","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1109\/EuroSP.2016.36","article-title":"\u201cThe limitations of deep learning in adversarial settings,\u201d","volume-title":"2016 IEEE European Symposium on Security and Privacy (EuroS &P)","author":"Papernot","year":"2016"},{"key":"B38","unstructured":"Pouransari\n              H.\n            \n            \n              Ghili\n              S.\n            \n          \n          Tiny ImageNet Visual Recognition Challenge\n          \n          2014"},{"key":"B39","first-page":"13032","article-title":"\u201cDo cifar-10 classifiers generalize to cifar-10?\u201d","author":"Recht","year":"2019","journal-title":"Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"B40","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"Imagenet large scale visual recognition challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis"},{"key":"B41","first-page":"31","article-title":"\u201cPoison frogs! Targeted clean-label poisoning attacks on neural networks,\u201d","author":"Shafahi","year":"2018","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B42","first-page":"17543","article-title":"\u201cRevisiting realistic test-time training: sequential inference and adaptation by anchored clustering,\u201d","author":"Su","year":"2022","journal-title":"Advances in Neural Information Processing Systems"},{"key":"B43","first-page":"9229","article-title":"\u201cTest-time training with self-supervision for generalization under distribution shifts,\u201d","volume-title":"International Conference on Machine Learning","author":"Sun","year":"2020"},{"key":"B44","doi-asserted-by":"publisher","first-page":"5940","DOI":"10.1609\/aaai.v34i04.6054","article-title":"\u201cDiscriminative adversarial domain adaptation,\u201d","author":"Tang","year":"2020","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"B45","article-title":"Gradient-based data subversion attack against binary classifiers","author":"Vasu","year":"2021","journal-title":"arXiv preprint arXiv:2105.14803"},{"key":"B46","article-title":"\u201cTent: fully test-time adaptation by entropy minimization,\u201d","author":"Wang","year":"2021","journal-title":"International Conference on Learning Representations"},{"key":"B47","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1016\/j.neucom.2018.05.083","article-title":"Deep visual domain adaptation: a survey","volume":"312","author":"Wang","year":"2018","journal-title":"Neurocomputing"},{"key":"B48","article-title":"Generative poisoning attack method against neural networks","author":"Yang","year":"2017","journal-title":"arXiv preprint arXiv:1703.01340"}],"updated-by":[{"DOI":"10.3389\/frai.2025.1682908","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2025,8,29]],"date-time":"2025-08-29T00:00:00Z","timestamp":1756425600000}}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2025.1621025\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,29]],"date-time":"2025-08-29T12:37:09Z","timestamp":1756471029000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2025.1621025\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,4]]},"references-count":48,"alternative-id":["10.3389\/frai.2025.1621025"],"URL":"https:\/\/doi.org\/10.3389\/frai.2025.1621025","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,8,4]]},"article-number":"1621025"}}