{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T02:33:04Z","timestamp":1760149984514,"version":"build-2065373602"},"reference-count":39,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2023,9,28]],"date-time":"2023-09-28T00:00:00Z","timestamp":1695859200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Shenzhen Key Technical Projects","award":["202208313000248","202205173000112","JCYJ20220818101406014","62006183","62206271","2020AAA0105600","2020M683489"],"award-info":[{"award-number":["202208313000248","202205173000112","JCYJ20220818101406014","62006183","62206271","2020AAA0105600","2020M683489"]}]},{"name":"Key Program of the Natural Science Foundation of Shenzhen","award":["202208313000248","202205173000112","JCYJ20220818101406014","62006183","62206271","2020AAA0105600","2020M683489"],"award-info":[{"award-number":["202208313000248","202205173000112","JCYJ20220818101406014","62006183","62206271","2020AAA0105600","2020M683489"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["202208313000248","202205173000112","JCYJ20220818101406014","62006183","62206271","2020AAA0105600","2020M683489"],"award-info":[{"award-number":["202208313000248","202205173000112","JCYJ20220818101406014","62006183","62206271","2020AAA0105600","2020M683489"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Key Research and Development Project of China","award":["202208313000248","202205173000112","JCYJ20220818101406014","62006183","62206271","2020AAA0105600","2020M683489"],"award-info":[{"award-number":["202208313000248","202205173000112","JCYJ20220818101406014","62006183","62206271","2020AAA0105600","2020M683489"]}]},{"DOI":"10.13039\/501100002858","name":"China Postdoctoral Science Foundation","doi-asserted-by":"publisher","award":["202208313000248","202205173000112","JCYJ20220818101406014","62006183","62206271","2020AAA0105600","2020M683489"],"award-info":[{"award-number":["202208313000248","202205173000112","JCYJ20220818101406014","62006183","62206271","2020AAA0105600","2020M683489"]}],"id":[{"id":"10.13039\/501100002858","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The rapid growth in dataset sizes in modern deep learning has significantly increased data storage costs. Furthermore, the training and time costs for deep neural networks are generally proportional to the dataset size. Therefore, reducing the dataset size while maintaining model performance is an urgent research problem that needs to be addressed. Dataset condensation is a technique that aims to distill the original dataset into a much smaller synthetic dataset while maintaining downstream training performance on any agnostic neural network. Previous work has demonstrated that matching the training trajectory between the synthetic dataset and the original dataset is more effective than matching the instantaneous gradient, as it incorporates long-range information. Despite the effectiveness of trajectory matching, it suffers from complex gradient unrolling across iterations, which leads to significant memory and computation overhead. To address this issue, this paper proposes a novel approach called Expert Subspace Projection (ESP), which leverages long-range information while avoiding gradient unrolling. Instead of strictly enforcing the synthetic dataset\u2019s training trajectory to mimic that of the real dataset, ESP only constrains it to lie within the subspace spanned by the training trajectory of the real dataset. The memory-saving advantage offered by our method facilitates unbiased training on the complete set of synthetic images and seamless integration with other dataset condensation techniques. Through extensive experiments, we have demonstrated the effectiveness of our approach. Our method outperforms the trajectory matching method on CIFAR10 by 16.7% in the setting of 1 Image\/Class, surpassing the previous state-of-the-art method by 3.2%.<\/jats:p>","DOI":"10.3390\/s23198148","type":"journal-article","created":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T07:42:08Z","timestamp":1695973328000},"page":"8148","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Dataset Condensation via Expert Subspace Projection"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0034-2065","authenticated-orcid":false,"given":"Zhiheng","family":"Ma","sequence":"first","affiliation":[{"name":"Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dezheng","family":"Gao","sequence":"additional","affiliation":[{"name":"Institute of Artificial Intelligence and Robotics, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shaolei","family":"Yang","sequence":"additional","affiliation":[{"name":"School of Software Engineering, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5025-3941","authenticated-orcid":false,"given":"Xing","family":"Wei","sequence":"additional","affiliation":[{"name":"School of Software Engineering, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1793-5836","authenticated-orcid":false,"given":"Yihong","family":"Gong","sequence":"additional","affiliation":[{"name":"Institute of Artificial Intelligence and Robotics, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"},{"name":"School of Software Engineering, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,9,28]]},"reference":[{"unstructured":"Lee, H.B., Lee, D.B., and Hwang, S.J. (2022). Dataset Condensation with Latent Space Knowledge Factorization and Sharing. arXiv.","key":"ref_1"},{"key":"ref_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Annu. Conf. Neural Inf. Process. Syst. (NeurIPS)"},{"key":"ref_3","first-page":"810","article-title":"DC-BENCH: Dataset condensation benchmark","volume":"35","author":"Cui","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"unstructured":"Wang, T., Zhu, J.Y., Torralba, A., and Efros, A.A. (2018). Dataset distillation. arXiv.","key":"ref_4"},{"unstructured":"Zhao, B., and Bilen, H. (2021). Dataset Condensation with Distribution Matching. arXiv.","key":"ref_5"},{"key":"ref_6","first-page":"3","article-title":"Dataset Condensation with Gradient Matching","volume":"1","author":"Zhao","year":"2021","journal-title":"Int. Conf. Learn. Represent. (ICLR)"},{"unstructured":"Zhao, B., and Bilen, H. (2021, January 18\u201324). Dataset condensation with differentiable siamese augmentation. Proceedings of the International Conference on Machine Learning (ICML), PMLR, Virtual Event.","key":"ref_7"},{"unstructured":"Finn, C., Abbeel, P., and Levine, S. (2017, January 6\u201311). Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the International Conference on Machine Learning (ICML), PMLR, Sydney, Australia.","key":"ref_8"},{"unstructured":"Nichol, A., Achiam, J., and Schulman, J. (2018). On first-order meta-learning algorithms. arXiv.","key":"ref_9"},{"doi-asserted-by":"crossref","unstructured":"Cazenavette, G., Wang, T., Torralba, A., Efros, A.A., and Zhu, J.Y. (2022, January 18\u201324). Dataset distillation by matching training trajectories. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.","key":"ref_10","DOI":"10.1109\/CVPR52688.2022.01045"},{"unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.","key":"ref_11"},{"unstructured":"Turc, I., Chang, M.W., Lee, K., and Toutanova, K. (2019). Well-read students learn better: On the importance of pre-training compact models. arXiv.","key":"ref_12"},{"unstructured":"Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv.","key":"ref_13"},{"doi-asserted-by":"crossref","unstructured":"Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. (2019). Tinybert: Distilling bert for natural language understanding. arXiv.","key":"ref_14","DOI":"10.18653\/v1\/2020.findings-emnlp.372"},{"unstructured":"Gur-Ari, G., Roberts, D.A., and Dyer, E. (2018). Gradient descent happens in a tiny subspace. arXiv.","key":"ref_15"},{"unstructured":"Li, C., Farkhoor, H., Liu, R., and Yosinski, J. (2018). Measuring the intrinsic dimension of objective landscapes. arXiv.","key":"ref_16"},{"key":"ref_17","first-page":"12140","article-title":"Improving neural network training in low dimensional random bases","volume":"33","author":"Gressmann","year":"2020","journal-title":"Annu. Conf. Neural Inf. Process. Syst. (NeurIPS)"},{"doi-asserted-by":"crossref","unstructured":"Li, T., Tan, L., Tao, Q., Liu, Y., and Huang, X. (2021). Low dimensional landscape hypothesis is true: DNNs can be trained in tiny subspaces. arXiv.","key":"ref_18","DOI":"10.1109\/TPAMI.2022.3178101"},{"unstructured":"Bachem, O., Lucic, M., and Krause, A. (2017). Practical coreset constructions for machine learning. arXiv.","key":"ref_19"},{"key":"ref_20","first-page":"14879","article-title":"Coresets via bilevel optimization for continual learning and streaming","volume":"33","author":"Borsos","year":"2020","journal-title":"Annu. Conf. Neural Inf. Process. Syst. (NeurIPS)"},{"doi-asserted-by":"crossref","unstructured":"Har-Peled, S., and Kushal, A. (2005, January 6\u20138). Smaller coresets for k-median and k-means clustering. Proceedings of the Twenty-First Annual Symposium on Computational Geometry, Pisa, Italy.","key":"ref_21","DOI":"10.1145\/1064092.1064114"},{"unstructured":"Sener, O., and Savarese, S. (2017). Active learning for convolutional neural networks: A core-set approach. arXiv.","key":"ref_22"},{"key":"ref_23","first-page":"363","article-title":"Core vector machines: Fast SVM training on very large data sets","volume":"6","author":"Tsang","year":"2005","journal-title":"J. Mach. Learn. Res. JMRL"},{"unstructured":"Krizhevsky, A., Vinod, N., and Hinton, G. (2023, August 11). Learning Multiple Layers of Features from Tiny Images. Technical Report. University of Toronto. Available online: https:\/\/www.cs.toronto.edu\/~kriz\/cifar.html.","key":"ref_24"},{"key":"ref_25","first-page":"3","article-title":"Tiny imagenet visual recognition challenge","volume":"7","author":"Le","year":"2015","journal-title":"CS 231N"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet Large Scale Visual Recognition Challenge","volume":"115","author":"Russakovsky","year":"2015","journal-title":"Int. J. Comput. Vis. (IJCV)"},{"unstructured":"Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A.Y. (2011, January 16\u201317). Reading Digits in Natural Images with Unsupervised Feature Learning. Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Granada, Spain.","key":"ref_27"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proc. IEEE"},{"unstructured":"Kingma, D.P., and Ba, J. (2015, January 7\u20139). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations, ICLR, San Diego, CA, USA.","key":"ref_29"},{"doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep residual learning for image recognition. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","key":"ref_30","DOI":"10.1109\/CVPR.2016.90"},{"doi-asserted-by":"crossref","unstructured":"Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21\u201326). Densely connected convolutional networks. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","key":"ref_31","DOI":"10.1109\/CVPR.2017.243"},{"unstructured":"Chen, Y., Welling, M., and Smola, A. (2012). Super-samples from kernel herding. arXiv.","key":"ref_32"},{"doi-asserted-by":"crossref","unstructured":"Rebuffi, S.A., Kolesnikov, A., Sperl, G., and Lampert, C.H. (2017, January 21\u201326). icarl: Incremental classifier and representation learning. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.","key":"ref_33","DOI":"10.1109\/CVPR.2017.587"},{"doi-asserted-by":"crossref","unstructured":"Belouadah, E., and Popescu, A. (2020, January 1\u20135). Scail: Classifier weights scaling for class incremental learning. Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision, Snowmass Village, CO, USA.","key":"ref_34","DOI":"10.1109\/WACV45572.2020.9093562"},{"doi-asserted-by":"crossref","unstructured":"Castro, F.M., Mar\u00edn-Jim\u00e9nez, M.J., Guil, N., Schmid, C., and Alahari, K. (2018, January 8\u201314). End-to-end incremental learning. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","key":"ref_35","DOI":"10.1007\/978-3-030-01258-8_15"},{"key":"ref_36","first-page":"5186","article-title":"Dataset distillation with infinitely wide convolutional networks","volume":"34","author":"Nguyen","year":"2021","journal-title":"Annu. Conf. Neural Inf. Process. Syst. (NeurIPS)"},{"doi-asserted-by":"crossref","unstructured":"Wang, K., Zhao, B., Peng, X., Zhu, Z., Yang, S., Wang, S., Huang, G., Bilen, H., Wang, X., and You, Y. (2022, January 18\u201324). Cafe: Learning to condense dataset by aligning features. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.","key":"ref_37","DOI":"10.1109\/CVPR52688.2022.01188"},{"unstructured":"Kim, J.H., Kim, J., Oh, S.J., Yun, S., Song, H., Jeong, J., Ha, J.W., and Song, H.O. (2022). Dataset Condensation via Efficient Synthetic-Data Parameterization. arXiv.","key":"ref_38"},{"unstructured":"Tan, M., and Le, Q. (2021, January 18\u201324). Efficientnetv2: Smaller models and faster training. Proceedings of the International Conference on Machine Learning, PMLR, Virtual Event.","key":"ref_39"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/19\/8148\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T21:01:14Z","timestamp":1760130074000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/19\/8148"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,28]]},"references-count":39,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2023,10]]}},"alternative-id":["s23198148"],"URL":"https:\/\/doi.org\/10.3390\/s23198148","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2023,9,28]]}}}