{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,23]],"date-time":"2025-10-23T21:05:42Z","timestamp":1761253542799,"version":"build-2065373602"},"reference-count":36,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2021,3,27]],"date-time":"2021-03-27T00:00:00Z","timestamp":1616803200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>Transfer learning seeks to improve the generalization performance of a target task by exploiting the knowledge learned from a related source task. Central questions include deciding what information one should transfer and when transfer can be beneficial. The latter question is related to the so-called negative transfer phenomenon, where the transferred source information actually reduces the generalization performance of the target task. This happens when the two tasks are sufficiently dissimilar. In this paper, we present a theoretical analysis of transfer learning by studying a pair of related perceptron learning tasks. Despite the simplicity of our model, it reproduces several key phenomena observed in practice. Specifically, our asymptotic analysis reveals a phase transition from negative transfer to positive transfer as the similarity of the two tasks moves past a well-defined threshold.<\/jats:p>","DOI":"10.3390\/e23040400","type":"journal-article","created":{"date-parts":[[2021,3,28]],"date-time":"2021-03-28T22:09:01Z","timestamp":1616969341000},"page":"400","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Phase Transitions in Transfer Learning for High-Dimensional Perceptrons"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8961-3553","authenticated-orcid":false,"given":"Oussama","family":"Dhifallah","sequence":"first","affiliation":[{"name":"John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5174-2595","authenticated-orcid":false,"given":"Yue M.","family":"Lu","sequence":"additional","affiliation":[{"name":"John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2021,3,27]]},"reference":[{"key":"ref_1","unstructured":"Pratt, L.Y., Mostow, J., and Kamm, C.A. (1991, January 14\u201319). Direct Transfer of Learned Information among Neural Networks. Proceedings of the Ninth National Conference on Artificial Intelligence\u2014Volume 2, AAAI\u201991, Anaheim, CA, USA."},{"key":"ref_2","first-page":"204","article-title":"Discriminability-Based Transfer between Neural Networks","volume":"Volume 5","author":"Hanson","year":"1993","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_3","unstructured":"Perkins, D., and Salomon, G. (1992). Transfer of Learning, Pergamon."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A Survey on Transfer Learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., and Liu, C. (2018). A Survey on Deep Transfer Learning. arXiv.","DOI":"10.1007\/978-3-030-01424-7_27"},{"key":"ref_6","unstructured":"Rosenstein, M.T., Marx, Z., Kaelbling, P.K., and Dietterich, T.G. (2005). To transfer or not to transfer. NIPS Workshop on Transfer Learning, NIPS."},{"key":"ref_7","first-page":"83","article-title":"Task Clustering and Gating for Bayesian Multitask Learning","volume":"4","author":"Bakker","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Sch\u00f6lkopf, B., and Warmuth, M.K. (2003). Exploiting Task Relatedness for Multiple Task Learning. Learning Theory and Kernel Machines, Springer.","DOI":"10.1007\/b12006"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Kornblith, S., Shlens, J., and Le, Q.V. (2019). Do Better ImageNet Models Transfer Better?. arXiv.","DOI":"10.1109\/CVPR.2019.00277"},{"key":"ref_10","unstructured":"Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. (2014). How Transferable Are Features in Deep Neural Networks?. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"928","DOI":"10.1109\/TPAMI.2013.197","article-title":"Learning Categories From Few Examples With Multi Model Knowledge Transfer","volume":"36","author":"Tommasi","year":"2014","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Yang, J., Yan, R., and Hauptmann, A.G. (2007, January 28\u201331). Adapting SVM Classifiers to Data with Shifted Distributions. Proceedings of the Seventh IEEE International Conference on Data Mining Workshops (ICDMW 2007), Omaha, NE, USA.","DOI":"10.1109\/ICDMW.2007.37"},{"key":"ref_13","unstructured":"Lampinen, A.K., and Ganguli, S. (2019). An Analytic Theory of Generalization Dynamics and Transfer Learning in Deep Linear Networks. arXiv."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Dar, Y., and Baraniuk, R.G. (2021). Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Tasks. arXiv.","DOI":"10.1137\/22M1469559"},{"key":"ref_15","unstructured":"Saglietti, L., and Zdeborov\u00e1, L. (2020). Solvable Model for Inheriting the Regularization through Knowledge Distillation. arXiv."},{"key":"ref_16","unstructured":"Stojnic, M. (2013). A Framework to Characterize Performance of LASSO Algorithms. arXiv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Thrampoulidis, C., Abbasi, E., and Hassibi, B. (October, January 29). Precise high-dimensional error analysis of regularized M-estimators. Proceedings of the 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA.","DOI":"10.1109\/ALLERTON.2015.7447033"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Lindenstrauss, J., and Milman, V.D. (1988). On Milman\u2019s inequality and random subspaces which escape through a mesh in Rn. Geometric Aspects of Functional Analysis, Springer.","DOI":"10.1007\/BFb0081732"},{"key":"ref_19","unstructured":"Dhifallah, O., Thrampoulidis, C., and Lu, Y.M. (2018). Phase Retrieval via Polytope Optimization: Geometry, Phase Transitions, and New Algorithms. arXiv."},{"key":"ref_20","unstructured":"Dhifallah, O., and Lu, Y.M. (2020). A Precise Performance Analysis of Learning with Random Features. arXiv."},{"key":"ref_21","unstructured":"Salehi, F., Abbasi, E., and Hassibi, B. (2019). The Impact of Regularization on High-dimensional Logistic Regression. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kammoun, A., and Alouini, M.S. (2020). On the Precise Error Analysis of Support Vector Machines. arXiv.","DOI":"10.1109\/OJSP.2021.3051849"},{"key":"ref_23","unstructured":"Mignacco, F., Krzakala, F., Lu, Y.M., and Zdeborov\u00e1, L. (2020). The Role of Regularization in Classification of High-Dimensional Noisy Gaussian Mixture. arXiv."},{"key":"ref_24","unstructured":"Aubin, B., Krzakala, F., Lu, Y.M., and Zdeborov\u00e1, L. (2020). Generalization Error in High-Dimensional Perceptrons: Approaching Bayes Error with Convex Optimization. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Rockafellar, R.T., and Wets, R.J.B. (1998). Variational Analysis, Springer.","DOI":"10.1007\/978-3-642-02431-3"},{"key":"ref_26","first-page":"1683","article-title":"Regularized Linear Regression: A Precise Analysis of the Estimation Error","volume":"Volume 40","author":"Hazan","year":"2015","journal-title":"Proceedings of the 28th Conference on Learning Theory"},{"key":"ref_27","unstructured":"Rudelson, M., and Vershynin, R. (2010). Non-Asymptotic Theory of Random Matrices: Extreme Singular Values. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"269","DOI":"10.1137\/16M1058200","article-title":"Solving the Trust-Region Subproblem By a Generalized Eigenvalue Problem","volume":"27","author":"Adachi","year":"2017","journal-title":"SIAM J. Optim."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Shapiro, A., Dentcheva, D., and Ruszczy\u0144ski, A. (2014). Lectures on Stochastic Programming: Modeling and Theory, Society for Industrial and Applied Mathematics. [2nd ed.].","DOI":"10.1137\/1.9781611973433"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"1100","DOI":"10.1214\/aos\/1176345976","article-title":"Cox\u2019s Regression Model for Counting Processes: A Large Sample Study","volume":"10","author":"Andersen","year":"1982","journal-title":"Ann. Statist."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Newey, W.K., and Mcfadden, D. (1994). Chapter 36 Large sample estimation and hypothesis testing. Handbook of Econometrics, Elsevier.","DOI":"10.1016\/S1573-4412(05)80005-4"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Schilling, R.L. (2005). Measures, Integrals and Martingales, Cambridge University Press.","DOI":"10.1017\/CBO9780511810886"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Shor, N. (1985). Minimization Methods for Non-Differentiable Functions, Springer.","DOI":"10.1007\/978-3-642-82118-9"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1293","DOI":"10.1109\/TIT.2003.810641","article-title":"MMSE analysis of certain large isometric random precoded systems","volume":"49","author":"Debbah","year":"2003","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Boyd, S., and Vandenberghe, L. (2004). Convex Optimization, Cambridge University Press.","DOI":"10.1017\/CBO9780511804441"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"171","DOI":"10.2140\/pjm.1958.8.171","article-title":"On general minimax theorems","volume":"8","author":"Sion","year":"1958","journal-title":"Pac. J. Math."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/4\/400\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T13:53:10Z","timestamp":1760363590000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/23\/4\/400"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,27]]},"references-count":36,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2021,4]]}},"alternative-id":["e23040400"],"URL":"https:\/\/doi.org\/10.3390\/e23040400","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2021,3,27]]}}}