{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,10,26]],"date-time":"2024-10-26T04:19:09Z","timestamp":1729916349696,"version":"3.28.0"},"reference-count":52,"publisher":"MIT Press","issue":"10","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,9,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The task of transfer learning using pretrained convolutional neural networks is considered. We propose a convolution-SVD layer to analyze the convolution operators with a singular value decomposition computed in the Fourier domain. Singular vectors extracted from the source domain are transferred to the target domain, whereas the singular values are fine-tuned with a target data set. In this way, dimension reduction is achieved to avoid overfitting, while some flexibility to fine-tune the convolution kernels is maintained. We extend an existing convolution kernel reconstruction algorithm to allow for a reconstruction from an arbitrary set of learned singular values. A generalization bound for a single convolution-SVD layer is devised to show the consistency between training and testing errors. We further introduce a notion of transfer learning gap. We prove that the testing error for a single convolution-SVD layer is bounded in terms of the gap, which motivates us to develop a regularization model with the gap as the regularizer. Numerical experiments are conducted to demonstrate the superiority of the proposed model in solving classification problems and the influence of various parameters. In particular, the regularization is shown to yield a significantly higher prediction accuracy.<\/jats:p>","DOI":"10.1162\/neco_a_01608","type":"journal-article","created":{"date-parts":[[2023,7,31]],"date-time":"2023-07-31T18:06:39Z","timestamp":1690826799000},"page":"1678-1712","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":1,"title":["Transfer Learning With Singular Value Decomposition of Multichannel Convolution Matrices"],"prefix":"10.1162","volume":"35","author":[{"given":"Tak Shing Au","family":"Yeung","sequence":"first","affiliation":[{"name":"NVIDIA AI Technology Center, NVIDIA, Hong Kong 852, China iauyeung@nvidia.com"}]},{"given":"Ka Chun","family":"Cheung","sequence":"additional","affiliation":[{"name":"Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong"},{"name":"NVIDIA AI Technology Center, NVIDIA, Hong Kong 852, China chcheung@nvidia.com"}]},{"given":"Michael K.","family":"Ng","sequence":"additional","affiliation":[{"name":"Institute of Data Science and Department of Mathematics, University of Hong Kong, Hong Kong 852, China michael.ng@hku.hk"}]},{"given":"Simon","family":"See","sequence":"additional","affiliation":[{"name":"NVIDIA AI Technology Center, NVIDIA, Singapore 65"},{"name":"Centre for Computational Science and Mathematical Modelling, Coventry University, Coventry, CV1 2TL, U.K."},{"name":"Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 65, China"},{"name":"Department of Computer Science and Engineering, Mahindra University, Hyderabad 500043, India ssee@nvidia.com"}]},{"given":"Andy","family":"Yip","sequence":"additional","affiliation":[{"name":"Department of Mathematics, University of Hong Kong, Pokfulam Road, Hong Kong 852, China mhyipa@hotmail.com"}]}],"member":"281","published-online":{"date-parts":[[2023,9,8]]},"reference":[{"key":"2023091116080347400_bib1","first-page":"1","article-title":"A new neural network pruning method based on the singular value decomposition and the weight initialisation","volume-title":"Proceedings of the 2002 11th European Signal Processing Conference","author":"Abid","year":"2002"},{"key":"2023091116080347400_bib2","first-page":"254","article-title":"Stronger generalization bounds for deep nets via a compression approach","author":"Arora","year":"2018","journal-title":"Proceedings of the International Conference on Machine Learning"},{"key":"2023091116080347400_bib3","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1016\/j.neunet.2020.04.021","article-title":"Theory of adaptive SVD regularization for deep neural networks","volume":"128","author":"Bejani","year":"2020","journal-title":"Neural Networks"},{"issue":"1","key":"2023091116080347400_bib4","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1016\/S0893-6080(02)00167-3","article-title":"Analysis of Tikhonov regularization for function approximation by neural networks","volume":"16","author":"Burger","year":"2003","journal-title":"Neural Networks"},{"year":"2020","author":"Chen","journal-title":"A simple framework for contrastive learning of visual representations.","key":"2023091116080347400_bib5"},{"key":"2023091116080347400_bib6","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1145\/2959100.2959190","article-title":"Deep neural networks for YouTube recommendations","volume-title":"Proceedings of the 10th ACM Conference on Recommender Systems","author":"Covington","year":"2016"},{"key":"2023091116080347400_bib7","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1109\/CVPR.2009.5206848","article-title":"ImageNet: A large-scale hierarchical image database","volume-title":"Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition","author":"Deng","year":"2009"},{"key":"2023091116080347400_bib8","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Devlin","year":"2019"},{"key":"2023091116080347400_bib9","first-page":"373","article-title":"How many samples are needed to estimate a convolutional neural network?","volume-title":"Advances in neural information processing systems","author":"Du","year":"2018"},{"key":"2023091116080347400_bib10","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3212725.3212733","article-title":"Eficiently combining SVD, pruning, clustering and retraining for enhanced neural network compression","volume-title":"Proceedings of the 2nd International Workshop on Embedded and Mobile Deep Learning","author":"Goetschalckx","year":"2018"},{"key":"2023091116080347400_bib11","article-title":"Caltech-256 object category dataset","author":"Griffin","year":"2007","journal-title":"CalTech Report"},{"key":"2023091116080347400_bib12","first-page":"21271","article-title":"Bootstrap your own latent\u2014a new approach to self-supervised learning","volume-title":"Advances in neural information processing systems","author":"Grill","year":"2020"},{"year":"2015","author":"Hardt","journal-title":"Train faster, generalize better: Stability of stochastic gradient descent.","key":"2023091116080347400_bib13"},{"key":"2023091116080347400_bib14","doi-asserted-by":"crossref","first-page":"770","DOI":"10.1109\/CVPR.2016.90","article-title":"Deep residual learning for image recognition","author":"He","year":"2016","journal-title":"Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"2023091116080347400_bib15","first-page":"245","article-title":"Reshaping deep neural network for fast decoding by node-pruning","volume-title":"Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing","author":"He","year":"2014"},{"key":"2023091116080347400_bib16","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511840371","volume-title":"Topics in matrix analysis","author":"Horn","year":"1991"},{"key":"2023091116080347400_bib17","first-page":"2790","article-title":"Parameter-efficient transfer learning for NLP","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Houlsby","year":"2019"},{"volume-title":"Fundamentals of digital image processing","year":"1988","author":"Jain","key":"2023091116080347400_bib18"},{"key":"2023091116080347400_bib19","first-page":"8513","article-title":"On the stability of graph convolutional neural networks under edge rewiring","volume-title":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing","author":"Kenlay","year":"2021"},{"key":"2023091116080347400_bib20","article-title":"Novel dataset for fine-grained image categorization","volume-title":"Proceedings of the First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition","author":"Khosla","year":"2011"},{"key":"2023091116080347400_bib21","first-page":"6124","article-title":"Quartznet: Deep automatic speech recognition with 1D time-channel separable convolutions","volume-title":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing","author":"Kriman","year":"2020"},{"year":"2012","author":"Krizhevsky","article-title":"Learning multiple layers of features from tiny images","key":"2023091116080347400_bib22"},{"key":"2023091116080347400_bib23","article-title":"Fine-tuning can distort pretrained features and underperform out-of-distribution","author":"Kumar","year":"2022","journal-title":"Proceedings of the International Conference on Learning Representations."},{"year":"2022","author":"Lian","journal-title":"Scaling and shifting your features: A new baseline for efficient model tuning.","key":"2023091116080347400_bib24"},{"issue":"5","key":"2023091116080347400_bib25","doi-asserted-by":"crossref","DOI":"10.3390\/s18051523","article-title":"Planetary gears feature extraction and fault diagnosis method based on VMD and CNN","volume":"18","author":"Liu","year":"2018","journal-title":"Sensors"},{"key":"2023091116080347400_bib26","first-page":"11976","article-title":"A ConvNet for the 2020s","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Liu","year":"2022"},{"key":"2023091116080347400_bib27","article-title":"Generalization bounds for deep convolutional neural networks","volume-title":"Proceedings of the International Conference on Learning Representations.","author":"Long","year":"2019"},{"key":"2023091116080347400_bib28","first-page":"118","article-title":"VGG deep neural network compression via SVD and CUR decomposition techniques","volume-title":"Proceedings of the 2020 7th NAFOSTED Conference on Information and Computer Science","author":"Mai","year":"2020"},{"key":"2023091116080347400_bib29","first-page":"148","article-title":"On the method of bounded differences","volume-title":"Surveys in combinatorics","author":"McDiarmid","year":"1989"},{"key":"2023091116080347400_bib30","doi-asserted-by":"publisher","first-page":"152261","DOI":"10.1109\/ACCESS.2021.3125791","article-title":"Hybrid CNN-SVD based prominent feature extraction and selection for grading diabetic retinopathy using extreme learning machine algorithm","volume":"9","author":"Nahiduzzaman","year":"2021","journal-title":"IEEE Access"},{"key":"2023091116080347400_bib31","doi-asserted-by":"crossref","DOI":"10.1109\/ICVGIP.2008.47","article-title":"Automated flower classification over a large number of classes","volume-title":"Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing","author":"Nilsback","year":"2008"},{"issue":"10","key":"2023091116080347400_bib32","doi-asserted-by":"publisher","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"2023091116080347400_bib33","first-page":"779","article-title":"You only look once: Unified, real-time object detection","volume-title":"Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition","author":"Redmon","year":"2016"},{"key":"2023091116080347400_bib34","article-title":"The singular values of convolutional layers","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Sedghi","year":"2019"},{"key":"2023091116080347400_bib35","first-page":"4779","article-title":"Natural TTS synthesis by conditioning WaveNet on MEL spectrogram predictions","volume-title":"Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing","author":"Shen","year":"2018"},{"key":"2023091116080347400_bib36","article-title":"Very deep convolutional networks for large-scale image recognition","volume-title":"Proceedings of the 3rd International Conference on Learning Representations","author":"Simonyan","year":"2015"},{"key":"2023091116080347400_bib37","doi-asserted-by":"publisher","first-page":"383","DOI":"10.1016\/j.patrec.2020.02.004","article-title":"SVD-based redundancy removal in 1-D CNNs for acoustic scene classification","volume":"131","author":"Singh","year":"2020","journal-title":"Pattern Recognition Letters"},{"key":"2023091116080347400_bib38","doi-asserted-by":"crossref","DOI":"10.1155\/2020\/5343214","article-title":"SVD-CNN: A convolutional neural network model with orthogonal constraints based on SVD for context-aware citation recommendation","author":"Tao","year":"2020","journal-title":"Computational Intelligence and Neuroscience"},{"volume-title":"Solution of ill-posed problems","year":"1977","author":"Tikhonov","key":"2023091116080347400_bib39"},{"key":"2023091116080347400_bib40","doi-asserted-by":"crossref","DOI":"10.1145\/3292500.3330956","article-title":"Stability and generalization of graph convolutional neural networks","volume-title":"Proceedings of the 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","author":"Verma","year":"2019"},{"key":"2023091116080347400_bib41","article-title":"Transfer learning via minimizing the performance gap between domains","volume-title":"Advances in neural information processing systems","author":"Wang","year":"2019"},{"key":"2023091116080347400_bib42","first-page":"390","article-title":"SVD-based channel pruning for convolutional neural network in acoustic scene classification model","volume-title":"Proceedings of the 2019 IEEE International Conference on Multimedia and Expo Workshops","author":"Wang","year":"2019"},{"key":"2023091116080347400_bib43","first-page":"2097","article-title":"ChestX-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Wang","year":"2017"},{"key":"2023091116080347400_bib44","article-title":"Stability of manifold neural networks to deformations","author":"Wang","year":"2021","journal-title":"CoRR"},{"key":"2023091116080347400_bib45","doi-asserted-by":"crossref","DOI":"10.1109\/CVPRW50498.2020.00347","article-title":"Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops","author":"Yang","year":"2020"},{"key":"2023091116080347400_bib46","first-page":"25739","article-title":"Deep model reassembly","volume":"35","author":"Yang","year":"2022","journal-title":"Advances in neural information processing systems"},{"key":"2023091116080347400_bib47","article-title":"Transfer learning or self-supervised learning? A tale of two pretraining paradigms","author":"Yang","year":"2020","journal-title":"CoRR"},{"year":"2020","author":"Yang","article-title":"COVID-CT-dataset: A CT scan dataset about COVID-19","key":"2023091116080347400_bib48"},{"key":"2023091116080347400_bib49","first-page":"73","article-title":"Factorizing knowledge in neural networks","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Yang","year":"2022"},{"key":"2023091116080347400_bib50","first-page":"3320","article-title":"How transferable are features in deep neural networks?","volume-title":"Advances in neural information processing systems","author":"Yosinski","year":"2014"},{"year":"2019","author":"Zhai","journal-title":"A large-scale study of representation learning with the visual task adaptation benchmark","key":"2023091116080347400_bib51"},{"key":"2023091116080347400_bib52","article-title":"Universality of deep convolutional neural networks","author":"Zhou","year":"2018","journal-title":"CoRR"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/35\/10\/1678\/2157851\/neco_a_01608.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/35\/10\/1678\/2157851\/neco_a_01608.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,25]],"date-time":"2024-10-25T11:46:03Z","timestamp":1729856763000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/35\/10\/1678\/117018\/Transfer-Learning-With-Singular-Value"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,8]]},"references-count":52,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2023,9,8]]},"published-print":{"date-parts":[[2023,9,8]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01608","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"type":"print","value":"0899-7667"},{"type":"electronic","value":"1530-888X"}],"subject":[],"published-other":{"date-parts":[[2023,10]]},"published":{"date-parts":[[2023,9,8]]}}}