{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,19]],"date-time":"2026-09-19T18:40:07Z","timestamp":1789843207507,"version":"4.0.1"},"reference-count":67,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2022,3,17]],"date-time":"2022-03-17T00:00:00Z","timestamp":1647475200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The deep neural network suffers from many fundamental issues in machine learning. For example, it often gets trapped into a local minimum in training, and its prediction uncertainty is hard to be assessed. To address these issues, we propose the so-called kernel-expanded stochastic neural network (K-StoNet) model, which incorporates support vector regression as the first hidden layer and reformulates the neural network as a latent variable model. The former maps the input vector into an infinite dimensional feature space via a radial basis function kernel, ensuring the absence of local minima on its training loss surface. The latter breaks the high-dimensional non-convex neural network training problem into a series of low-dimensional convex optimization problems, and enables its prediction uncertainty easily assessed. The K-StoNet can be easily trained using the imputation-regularized optimization algorithm. Compared to traditional deep neural networks, K-StoNet possesses a theoretical guarantee to asymptotically converge to the global optimum and enables the prediction uncertainty easily assessed. The performances of the new model in training, prediction and uncertainty quantification are illustrated by simulated and real data examples.<\/jats:p>","DOI":"10.1111\/rssb.12496","type":"journal-article","created":{"date-parts":[[2022,3,18]],"date-time":"2022-03-18T05:54:34Z","timestamp":1647582874000},"page":"547-578","source":"Crossref","is-referenced-by-count":7,"title":["A Kernel-Expanded Stochastic Neural Network"],"prefix":"10.1093","volume":"84","author":[{"given":"Yan","family":"Sun","sequence":"first","affiliation":[{"name":"Department of Statistics , , West Lafayette, IN , USA"},{"name":"Purdue University , , West Lafayette, IN , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Faming","family":"Liang","sequence":"additional","affiliation":[{"name":"Department of Statistics , , West Lafayette, IN , USA"},{"name":"Purdue University , , West Lafayette, IN , USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,3,17]]},"reference":[{"key":"2023022408355723200_","article-title":"A convergence theory for deep learning via over-parameterization","volume-title":"ICML","author":"Allen-Zhu","year":"2019"},{"key":"2023022408355723200_","first-page":"67","article-title":"Lagrangian support vector regression via unconstrained convex minimization","volume":"51C","author":"Balasundaram","year":"2013","journal-title":"Neural Networks"},{"issue":"64","key":"2023022408355723200_","first-page":"1","article-title":"A representer theorem for deep kernel learning","volume":"20","author":"Bohn","year":"2019","journal-title":"Journal of Machine Learning Research"},{"key":"2023022408355723200_","first-page":"73","article-title":"The sem algorithm: a probabilistic teacher algorithm derived from the em algorithm for the mixture problem","volume":"2","author":"Celeux","year":"1985","journal-title":"Computational Statistics Quarterly"},{"key":"2023022408355723200_","first-page":"300","article-title":"Underdamped langevin mcmc: a non-asymptotic analysis","volume-title":"Proceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research","author":"Cheng","year":"2018"},{"key":"2023022408355723200_","volume-title":"Advances in neural information processing systems","author":"Cho","year":"2009"},{"issue":"3","key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"799","DOI":"10.3150\/07-BEJ5102","article-title":"Consistency and robustness of kernel-based regression in convex risk minimization","volume":"13","author":"Christmann","year":"2007","journal-title":"Bernoulli"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"999","DOI":"10.1109\/BigData.2015.7363851","article-title":"Large scale support vector regression for aviation safety","volume-title":"2015 IEEE International Conference on Big Data (Big Data)","author":"Das","year":"2015"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1111\/j.2517-6161.1977.tb01600.x","article-title":"Maximum likelihood from incomplete data via the em algorithm","volume":"39","author":"Dempster","year":"1977","journal-title":"Journal of the Royal Statistical Society, Series B"},{"key":"2023022408355723200_","article-title":"Gradient descent finds global minima of deep neural networks","volume-title":"ICML","author":"Du","year":"2019"},{"issue":"2","key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"216","DOI":"10.1016\/0370-2693(87)91197-X","article-title":"Hybrid monte carlo","volume":"195","author":"Duane","year":"1987","journal-title":"Physics Letters B"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"1348","DOI":"10.1198\/016214501753382273","article-title":"Variable selection via nonconcave penalized likelihood and its oracle properties","volume":"96","author":"Fan","year":"2001","journal-title":"Journal of the American Statistical Association"},{"key":"2023022408355723200_","first-page":"1050","article-title":"Dropout as a Bayesian approximation: Representing model uncertainty in deep learning","volume-title":"Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48","author":"Gal","year":"2016"},{"issue":"1","key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1023\/A:1012494009640","article-title":"A probabilistic framework for SVM regression and error bar estimation","volume":"46","author":"Gao","year":"2002","journal-title":"Machine Learning"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"721","DOI":"10.1109\/TPAMI.1984.4767596","article-title":"Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images","volume":"6","author":"Geman","year":"1984","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"issue":"1","key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1109\/34.107014","article-title":"On the problem of local minima in backpropagation","volume":"14","author":"Gori","year":"1992","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"2023022408355723200_","first-page":"2348","article-title":"Practical variational inference for neural networks","volume-title":"Advances in neural information processing systems","author":"Graves","year":"2011"},{"key":"2023022408355723200_","article-title":"Noisy activation functions","volume-title":"ICML","author":"G\u00fcl\u00e7ehre","year":"2016"},{"key":"2023022408355723200_","first-page":"1321","article-title":"On calibration of modern neural networks","volume-title":"Proceedings of the 34th International Conference on Machine Learning - Volume 70","author":"Guo","year":"2017"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1023\/A:1022936519097","article-title":"A note on the universal approximation capability of support vector machines","volume":"17","author":"Hammer","year":"2003","journal-title":"Neural Processing Letters"},{"key":"2023022408355723200_","first-page":"1861","article-title":"Probabilistic backpropagation for scalable learning of Bayesian neural networks","volume-title":"International Conference on Machine Learning","author":"Hernndez-Lobato","year":"2015"},{"issue":"10","key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"428","DOI":"10.1016\/j.tics.2007.09.004","article-title":"Learning multiple layers of representation","volume":"11","author":"Hinton","year":"2007","journal-title":"Trends in Cognitive Sciences"},{"key":"2023022408355723200_","first-page":"448","article-title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","volume-title":"Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37","author":"Ioffe","year":"2015"},{"key":"2023022408355723200_","article-title":"Adam: A method for stochastic optimization","volume-title":"arXiv preprint arXiv:1412.6980","author":"Kingma","year":"2014"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4614-6849-3","volume-title":"Applied predictive modeling","author":"Kuhn","year":"2013"},{"key":"2023022408355723200_","first-page":"6405","article-title":"Simple and scalable predictive uncertainty estimation using deep ensembles","volume-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems","author":"Lakshminarayanan","year":"2017"},{"key":"2023022408355723200_","first-page":"1909","article-title":"Incremental support vector learning: analysis, implementation and applications","volume":"7","author":"Laskov","year":"2006","journal-title":"Journal of Machine Learning Research"},{"issue":"11","key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"2278","DOI":"10.1109\/5.726791","article-title":"Gradient-based learning applied to document recognition","volume":"86","author":"LeCun","year":"1998","journal-title":"Proceedings of the IEEE"},{"issue":"5","key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"899","DOI":"10.1111\/rssb.12279","article-title":"An imputation-regularized optimization algorithm for high-dimensional missing data problems and beyond","volume":"80","author":"Liang","year":"2018","journal-title":"Journal of the Royal Statistical Society, Series B"},{"issue":"2","key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"866","DOI":"10.1214\/16-AOS1471","article-title":"Statistical consistency and asymptotic normality for high-dimensional robust m-estimators","volume":"45","author":"Loh","year":"2017","journal-title":"Annals of Statistics"},{"issue":"6","key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"2455","DOI":"10.1214\/16-AOS1530","article-title":"Support recovery without incoherence: a case for nonconvex regularization","volume":"45","author":"Loh","year":"2017","journal-title":"The Annals of Statistics"},{"issue":"11","key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"2683","DOI":"10.1162\/089976603322385117","article-title":"Accurate on-line support vector regression","volume":"15","author":"Ma","year":"2003","journal-title":"Neural Computation"},{"key":"2023022408355723200_","article-title":"Convolutional kernel networks","volume-title":"arXiv preprint arXiv:1406.3332","author":"Mairal","year":"2014"},{"key":"2023022408355723200_","first-page":"2651","article-title":"Universal kernels","volume":"7","author":"Micchelli","year":"2006","journal-title":"Journal of Machine Learning Research"},{"key":"2023022408355723200_","article-title":"Deep double descent: Where bigger models and more data hurt","volume-title":"International Conference on Learning Representations","author":"Nakkiran","year":"2020"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1201\/b10905-6","volume-title":"Handbook of Markov Chain Monte Carlo","author":"Neal","year":"2011"},{"key":"2023022408355723200_","article-title":"Adding gradient noise improves learning for very deep networks","volume-title":"ArXiv, abs\/1511.06807","author":"Neelakantan","year":"2017"},{"key":"2023022408355723200_","volume-title":"The loss surface of deep and wide neural networks","author":"Nguyen","year":"2017"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"457","DOI":"10.2307\/3318671","article-title":"The stochastic EM algorithm: estimation and asymptotic results","volume":"6","author":"Nielsen","year":"2000","journal-title":"Bernoulli"},{"key":"2023022408355723200_","article-title":"Regularizing deep neural networks by noise: Its interpretation and optimization","volume-title":"ArXiv, abs\/1710.05179","author":"Noh","year":"2017"},{"issue":"2","key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1162\/neco.1991.3.2.246","article-title":"Universal approximation using radial-basis-function networks","volume":"3","author":"Park","year":"1991","journal-title":"Neural Computation"},{"issue":"85","key":"2023022408355723200_","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"Journal of Machine Learning Research"},{"issue":"1","key":"2023022408355723200_","first-page":"356","article-title":"Asymptotic behavior of likelihood methods for exponential families when the number of parameters tend to infinity","volume":"16","author":"Portnoy","year":"1989","journal-title":"Annals of Statistics"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"2305","DOI":"10.1007\/s00521-015-2066-x","article-title":"Deep multilayer multiple kernel learning","volume":"27","author":"Rebai","year":"2016","journal-title":"Neural Computing & Applications"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"4628","DOI":"10.1063\/1.436415","article-title":"Brownian dynamics as smart Monte Carlo simulation","volume":"69","author":"Rossky","year":"1978","journal-title":"Journal of Chemical Physics"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"7","DOI":"10.15439\/2019F364","article-title":"Greedy incremental support vector regression","volume-title":"2019 Federated Conference on Computer Science and Information Systems (FedCSIS)","author":"Ruta","year":"2019"},{"key":"2023022408355723200_","first-page":"448","article-title":"Deep Boltzmann machines","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics","author":"Salakhutdinov","year":"2009"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1007\/3-540-44581-1_27","volume-title":"Computational learning theory","author":"Sch\u00f6lkopf","year":"2001"},{"issue":"3","key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1023\/B:STCO.0000035301.49549.88","article-title":"A tutorial on support vector regression","volume":"14","author":"Smola","year":"2004","journal-title":"Statistics and Computing"},{"issue":"1","key":"2023022408355723200_","first-page":"1929","article-title":"Dropout: a simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"The Journal of Machine Learning Research"},{"issue":"1","key":"2023022408355723200_","first-page":"67","article-title":"On the influence of the kernel on the consistency of support vector machines","volume":"2","author":"Steinwart","year":"2002","journal-title":"Journal of Machine Learning Research"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1109\/ICMLA.2013.84","article-title":"Deep multiple kernel learning","volume-title":"2013 12th International Conference on Machine Learning and Applications","author":"Strobl","year":"2013"},{"key":"2023022408355723200_","article-title":"Consistent sparse deep learning: Theory and computation","author":"Sun","year":"2021","journal-title":"Journal of the American Statistical Association"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression shrinkage and selection via the lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"Journal of the Royal Statistical Society, Series B"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4757-3264-1","volume-title":"The nature of statistical learning theory","author":"Vapnik","year":"2000"},{"key":"2023022408355723200_","first-page":"774","article-title":"A note on one class of perceptrons","volume":"25","author":"Vapnik","year":"1964","journal-title":"Automation and Remote Control"},{"key":"2023022408355723200_","first-page":"774","article-title":"Pattern recognition using generalized portrait method","volume":"24","author":"Vapnik","year":"1963","journal-title":"Automation and Remote Control"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","DOI":"10.1137\/1.9781611970128","volume-title":"Splines models for observational data","author":"Wahba","year":"1990"},{"key":"2023022408355723200_","article-title":"Uncertainty quantification for sparse deep learning","volume-title":"AISTATS","author":"Wang","year":"2020"},{"key":"2023022408355723200_","first-page":"797","article-title":"ThunderSVM: a fast SVM library on GPUs and CPUs","volume":"19","author":"Wen","year":"2018","journal-title":"Journal of Machine Learning Research"},{"key":"2023022408355723200_","first-page":"370","article-title":"Deep kernel learning","volume-title":"19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016","author":"Wilson","year":"2016"},{"key":"2023022408355723200_","first-page":"1","article-title":"A new svr incremental algorithm based on boundary vector","volume-title":"2010 International Conference on Computational Intelligence and Software Engineering","author":"Xu","year":"2010"},{"key":"2023022408355723200_","first-page":"909","article-title":"Adversarial noise layer: Regularize neural network by adding noise","volume-title":"2019 IEEE International Conference on Image Processing (ICIP)","author":"You","year":"2018"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"894","DOI":"10.1214\/09-AOS729","article-title":"Nearly unbiased variable selection under minimax concave penalty","volume":"38","author":"Zhang","year":"2010","journal-title":"Annals of Statistics"},{"key":"2023022408355723200_","first-page":"909","volume-title":"Two-layer multiple kernel learning","author":"Zhuang","year":"2011"},{"key":"2023022408355723200_","article-title":"An improved analysis of training over-parameterized deep neural networks","volume-title":"NuerIPS","author":"Zou","year":"2019"},{"key":"2023022408355723200_","doi-asserted-by":"crossref","first-page":"467","DOI":"10.1007\/s10994-019-05839-6","article-title":"Gradient descent optimizes overparameterized deep relu networks","volume":"109","author":"Zou","year":"2020","journal-title":"Machine Learning"}],"container-title":["Journal of the Royal Statistical Society Series B: Statistical Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1111\/rssb.12496","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1111\/rssb.12496","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/84\/2\/547\/49321430\/jrsssb_84_2_547.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jrsssb\/article-pdf\/84\/2\/547\/49321430\/jrsssb_84_2_547.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,20]],"date-time":"2024-09-20T14:28:03Z","timestamp":1726842483000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jrsssb\/article\/84\/2\/547\/7056090"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,17]]},"references-count":67,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,3,17]]},"published-print":{"date-parts":[[2022,4,1]]}},"URL":"https:\/\/doi.org\/10.1111\/rssb.12496","relation":{},"ISSN":["1369-7412","1467-9868"],"issn-type":[{"value":"1369-7412","type":"print"},{"value":"1467-9868","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,4]]},"published":{"date-parts":[[2022,3,17]]}}}