{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,9,10]],"date-time":"2024-09-10T06:43:36Z","timestamp":1725950616644},"reference-count":62,"publisher":"MIT Press","issue":"5","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,4,13]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This letter introduces a new framework for quantifying predictive uncertainty for both data and models that relies on projecting the data into a gaussian reproducing kernel Hilbert space (RKHS) and transforming the data probability density function (PDF) in a way that quantifies the flow of its gradient as a topological potential field (quantified at all points in the sample space). This enables the decomposition of the PDF gradient flow by formulating it as a moment decomposition problem using operators from quantum physics, specifically Schr\u00f6dinger's formulation. We experimentally show that the higher-order moments systematically cluster the different tail regions of the PDF, thereby providing unprecedented discriminative resolution of data regions having high epistemic uncertainty. In essence, this approach decomposes local realizations of the data PDF in terms of uncertainty moments. We apply this framework as a surrogate tool for predictive uncertainty quantification of point-prediction neural network models, overcoming various limitations of conventional Bayesian-based uncertainty quantification methods. Experimental comparisons with some established methods illustrate performance advantages that our framework exhibits.<\/jats:p>","DOI":"10.1162\/neco_a_01372","type":"journal-article","created":{"date-parts":[[2021,2,23]],"date-time":"2021-02-23T00:17:58Z","timestamp":1614039478000},"page":"1164-1198","update-policy":"http:\/\/dx.doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":1,"title":["Toward a Kernel-Based Uncertainty Decomposition Framework for Data and Models"],"prefix":"10.1162","volume":"33","author":[{"given":"Rishabh","family":"Singh","sequence":"first","affiliation":[{"name":"Computational NeuroEngineering Laboratory, Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL 32611, U.S.A. rish283@ufl.edu"}]},{"given":"Jose C.","family":"Principe","sequence":"additional","affiliation":[{"name":"Computational NeuroEngineering Laboratory, Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL 32611, U.S.A. principe@cnel.ufl.edu"}]}],"member":"281","published-online":{"date-parts":[[2021,4,13]]},"reference":[{"issue":"3","key":"2021041321535864900_B1","doi-asserted-by":"crossref","DOI":"10.1209\/0295-5075\/120\/38003","article-title":"Modeling stock return distributions with a quantum harmonic oscillator","volume":"120","author":"Ahn","year":"2018","journal-title":"Europhysics Letters"},{"issue":"3","key":"2021041321535864900_B2","doi-asserted-by":"crossref","first-page":"337","DOI":"10.1090\/S0002-9947-1950-0051437-7","article-title":"Theory of reproducing kernels","volume":"68","author":"Aronszajn","year":"1950","journal-title":"Transactions of the American Mathematical Society"},{"issue":"7\u20138","key":"2021041321535864900_B3","doi-asserted-by":"crossref","first-page":"355","DOI":"10.1016\/0375-9601(89)90066-2","article-title":"A new wave equation for a continuous nondemolition measurement","volume":"140","author":"Belavkin","year":"1989","journal-title":"Physics Letters A"},{"journal-title":"The kernel function and conformal mapping","year":"1970","author":"Bergman","key":"2021041321535864900_B4"},{"journal-title":"Reproducing kernel Hilbert spaces in probability and statistics","year":"2011","author":"Berlinet","key":"2021041321535864900_B5"},{"key":"2021041321535864900_B6","doi-asserted-by":"crossref","DOI":"10.1093\/oso\/9780198538493.001.0001","author":"Bishop","year":"1995","journal-title":"Neural networks for pattern recognition."},{"journal-title":"\u00dcber die beziehung zwischen dem zweiten hauptsatze des mechanischen w\u00e4rmetheorie und der wahrscheinlichkeitsrechnung, respective den s\u00e4tzen \u00fcber das w\u00e4rmegleichgewicht","year":"1877","author":"Boltzmann","key":"2021041321535864900_B7"},{"issue":"1","key":"2021041321535864900_B8","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1109\/TNNLS.2011.2178446","article-title":"Quantized kernel least mean square algorithm","volume":"23","author":"Chen","year":"2011","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"journal-title":"Deep learning for classical Japanese literature","year":"2018","author":"Clanuwat","key":"2021041321535864900_B9"},{"key":"2021041321535864900_B10","doi-asserted-by":"crossref","first-page":"248","DOI":"10.1109\/CVPR.2009.5206848","article-title":"Imagenet: A large-scale hierarchical image database","author":"Deng","year":"2009","journal-title":"Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"2021041321535864900_B11","doi-asserted-by":"crossref","DOI":"10.1201\/9781420034899","author":"Fang","year":"2005","journal-title":"Design and modeling for computer experiments"},{"issue":"594\u2013604","key":"2021041321535864900_B12","first-page":"309","article-title":"On the mathematical foundations of theoretical statistics","volume":"222","author":"Fisher","year":"1922","journal-title":"Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character"},{"key":"2021041321535864900_B13","doi-asserted-by":"crossref","DOI":"10.1002\/9780470770801","author":"Forrester","year":"2008","journal-title":"Engineering design via surrogate modelling: A practical guide"},{"key":"2021041321535864900_B14","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511616907","author":"Frieden","year":"2004","journal-title":"Science from Fisher information: A unification"},{"issue":"6","key":"2021041321535864900_B15","doi-asserted-by":"crossref","DOI":"10.1103\/PhysRevE.82.066117","article-title":"Quantifying system order for full and partial coarse graining","volume":"82","author":"Frieden","year":"2010","journal-title":"Physical Review E"},{"key":"2021041321535864900_B16","first-page":"1050","article-title":"Dropout as a Bayesian approximation: Representing model uncertainty in deep learning","author":"Gal","year":"2016","journal-title":"Proceedings of the International Conference on Machine Learning"},{"key":"2021041321535864900_B17","first-page":"2348","volume-title":"Advances in neural information processing systems","author":"Graves","year":"2011"},{"issue":"1","key":"2021041321535864900_B18","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1137\/0912004","article-title":"The fast gauss transform","volume":"12","author":"Greengard","year":"1991","journal-title":"SIAM Journal on Scientific and Statistical Computing"},{"issue":"3","key":"2021041321535864900_B19","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1007\/BF02590638","article-title":"Stochastic processes and statistical inference","volume":"1","author":"Grenander","year":"1950","journal-title":"Arkiv f\u00f6r matematik"},{"journal-title":"Statistical models in engineering","year":"1967","author":"Hahn","key":"2021041321535864900_B20"},{"journal-title":"Sur un nouveau d\u00e9veloppement en s\u00e9rie des fonctions","year":"1864","author":"Hermite","key":"2021041321535864900_B21"},{"key":"2021041321535864900_B22","first-page":"1861","article-title":"Probabilistic backpropagation for scalable learning of Bayesian neural networks","author":"Hern\u00e1ndez-Lobato","year":"2015","journal-title":"Proceedings of the International Conference on Machine Learning"},{"issue":"1","key":"2021041321535864900_B23","first-page":"1303","article-title":"Stochastic variational inference","volume":"14","author":"Hoffman","year":"2013","journal-title":"Journal of Machine Learning Research"},{"key":"2021041321535864900_B24","doi-asserted-by":"crossref","first-page":"1171","DOI":"10.1214\/009053607000000677","article-title":"Kernel methods in machine learning","volume":"36","author":"Hofmann","year":"2008","journal-title":"Annals of Statistics"},{"key":"2021041321535864900_B25","article-title":"Zur spektraltheorie stochastischer prozesse","volume":"AI","author":"Karhunen","year":"1946","journal-title":"Ann. Acad. Sci. Fennicae"},{"key":"2021041321535864900_B26","first-page":"12","article-title":"Mauna Loa atmospheric CO2 modern record","volume":"91","author":"Keeling","year":"1991","journal-title":"Trends"},{"issue":"1","key":"2021041321535864900_B27","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1214\/aoms\/1177729694","article-title":"On information and sufficiency","volume":"22","author":"Kullback","year":"1951","journal-title":"Annals of Mathematical Statistics"},{"key":"2021041321535864900_B28","first-page":"6402","volume-title":"Advances in neural information processing systems","author":"Lakshminarayanan","year":"2017"},{"issue":"7553","key":"2021041321535864900_B29","doi-asserted-by":"crossref","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"journal-title":"The MNIST database of handwritten digits, 1998","year":"1998","author":"LeCun","key":"2021041321535864900_B30"},{"issue":"11","key":"2021041321535864900_B31","doi-asserted-by":"crossref","first-page":"5286","DOI":"10.1109\/TSP.2007.896065","article-title":"Correntropy: Properties and applications in non-gaussian signal processing","volume":"55","author":"Liu","year":"2007","journal-title":"IEEE Transactions on Signal Processing"},{"journal-title":"Kernel adaptive filtering: A comprehensive introduction","year":"2011","author":"Liu","key":"2021041321535864900_B32"},{"key":"2021041321535864900_B33","first-page":"159","article-title":"Fonctions al\u00e9atoires \u00e0 d\u00e9composition orthogonale exponentielle","volume":"84","author":"Lo\u00e8ve","year":"1946","journal-title":"La Revue Scientifique"},{"issue":"3","key":"2021041321535864900_B34","doi-asserted-by":"crossref","first-page":"448","DOI":"10.1162\/neco.1992.4.3.448","article-title":"A practical Bayesian framework for backpropagation networks","volume":"4","author":"MacKay","year":"1992","journal-title":"Neural Computation"},{"key":"2021041321535864900_B35","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1016\/j.physa.2016.02.026","article-title":"Quantum Brownian motion model for the stock market","volume":"452","author":"Meng","year":"2016","journal-title":"Physica A: Statistical Mechanics and Its Applications"},{"issue":"1\u20132","key":"2021041321535864900_B36","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/2200000060","article-title":"Kernel mean embedding of distributions: A review and beyond","volume":"10","author":"Muandet","year":"2017","journal-title":"Foundations and Trends in Machine Learning"},{"journal-title":"Bayesian techniques for inverse uncertainty quantification","year":"2017","author":"Nagel","key":"2021041321535864900_B37"},{"journal-title":"Bayesian learning for neural networks","year":"2012","author":"Neal","key":"2021041321535864900_B38"},{"key":"2021041321535864900_B40","first-page":"427","article-title":"Deep neural networks are easily fooled: High confidence predictions for unrecognizable images","author":"Nguyen","year":"2015","journal-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition"},{"key":"2021041321535864900_B41","first-page":"4026","volume-title":"Advances in neural information processing systems","author":"Osband","year":"2016"},{"journal-title":"Variational Bayesian inference with stochastic search.","year":"2012","author":"Paisley","key":"2021041321535864900_B42"},{"issue":"3","key":"2021041321535864900_B43","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1214\/aoms\/1177704472","article-title":"On estimation of a probability density function and mode","volume":"33","author":"Parzen","year":"1962","journal-title":"Annals of Mathematical Statistics"},{"journal-title":"Statistical inference on time series by RKHS methods","year":"1970","author":"Parzen","key":"2021041321535864900_B44"},{"journal-title":"Uncertainty in neural networks: Bayesian ensembling","year":"2018","author":"Pearce","key":"2021041321535864900_B45"},{"journal-title":"Latent projection BNNs: Avoiding weight-space pathologies by learning latent representations of neural network weights","year":"2018","author":"Pradier","key":"2021041321535864900_B46"},{"key":"2021041321535864900_B47","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4419-1570-2","author":"Principe","year":"2010","journal-title":"Information theoretic learning: Renyi's entropy and kernel perspectives"},{"key":"2021041321535864900_B48","first-page":"265","article-title":"Information theoretic learning","volume":"1","author":"Principe","year":"2000","journal-title":"Unsupervised Adaptive Filtering"},{"issue":"1","key":"2021041321535864900_B49","doi-asserted-by":"crossref","first-page":"1","DOI":"10.2307\/2527726","article-title":"Some statistical methods for comparison of growth curves","volume":"14","author":"Rao","year":"1958","journal-title":"Biometrics"},{"key":"2021041321535864900_B50","article-title":"On measures of entropy and information","volume":"1","author":"R\u00e9nyi","year":"1961","journal-title":"Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability"},{"key":"2021041321535864900_B51","doi-asserted-by":"crossref","DOI":"10.1201\/9781315140919","author":"Silverman","year":"2018","journal-title":"Density estimation for statistics and data analysis"},{"journal-title":"Very deep convolutional networks for large-scale image recognition.","year":"2014","author":"Simonyan","key":"2021041321535864900_B52"},{"journal-title":"Uncertainty quantification: Theory, implementation, and applications","year":"2013","author":"Smith","key":"2021041321535864900_B53"},{"journal-title":"Learning with kernels","year":"1998","author":"Smola","key":"2021041321535864900_B54"},{"key":"2021041321535864900_B55","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1016\/B978-044450871-3\/50105-4","volume-title":"Landmark writings in Western mathematics","author":"Stigler","year":"2005"},{"key":"2021041321535864900_B56","doi-asserted-by":"crossref","first-page":"828","DOI":"10.1109\/TEVC.2019.2890858","article-title":"One pixel attack for fooling deep neural networks","volume":"23","author":"Su","year":"2019","journal-title":"IEEE Transactions on Evolutionary Computation"},{"key":"2021041321535864900_B57","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-23395-6","author":"Sullivan","year":"2015","journal-title":"Introduction to uncertainty quantification"},{"issue":"4","key":"2021041321535864900_B58","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1016\/0165-1765(80)90128-7","article-title":"Simultaneous equation estimation based on maximum entropy moments","volume":"5","author":"Theil","year":"1980","journal-title":"Economics Letters"},{"issue":"1","key":"2021041321535864900_B59","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1162\/neco.1996.8.1.152","article-title":"A comparison of some error estimates for neural network models","volume":"8","author":"Tibshirani","year":"1996","journal-title":"Neural Computation"},{"journal-title":"Matrix computations","year":"1983","author":"Van Loan","key":"2021041321535864900_B60"},{"journal-title":"The nature of statistical learning theory","year":"2013","author":"Vapnik","key":"2021041321535864900_B61"},{"journal-title":"Mathematical foundations of quantum mechanics","year":"1955","author":"von Neumann","key":"2021041321535864900_B39"},{"journal-title":"Gaussian processes for machine learning.","year":"2006","author":"Williams","key":"2021041321535864900_B62"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/neco\/article-pdf\/33\/5\/1164\/1909045\/neco_a_01372.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/neco\/article-pdf\/33\/5\/1164\/1909045\/neco_a_01372.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,24]],"date-time":"2024-08-24T17:43:31Z","timestamp":1724521411000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/33\/5\/1164\/97481\/Toward-a-Kernel-Based-Uncertainty-Decomposition"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,13]]},"references-count":62,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2021,4,13]]},"published-print":{"date-parts":[[2021,4,13]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01372","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"type":"print","value":"0899-7667"},{"type":"electronic","value":"1530-888X"}],"subject":[],"published-other":{"date-parts":[[2021,5]]},"published":{"date-parts":[[2021,4,13]]}}}