Posterior consistency for Gaussian process approximations of Bayesian posterior distributions

Stuart, Andrew; Teckentrup, Aretha

doi:10.1090/mcom/3244

Posterior consistency for Gaussian process approximations of Bayesian posterior distributions
HTML articles powered by AMS MathViewer

by Andrew M. Stuart and Aretha L. Teckentrup PDF

Math. Comp. 87 (2018), 721-753 Request permission

Abstract:

We study the use of Gaussian process emulators to approximate the parameter-to-observation map or the negative log-likelihood in Bayesian inverse problems. We prove error bounds on the Hellinger distance between the true posterior distribution and various approximations based on the Gaussian process emulator. Our analysis includes approximations based on the mean of the predictive process, as well as approximations based on the full Gaussian process emulator. Our results show that the Hellinger distance between the true posterior and its approximations can be bounded by moments of the error in the emulator. Numerical results confirm our theoretical findings.

References

Robert J. Adler, The geometry of random fields, Wiley Series in Probability and Mathematical Statistics, John Wiley & Sons, Ltd., Chichester, 1981. MR 611857
Christophe Andrieu and Gareth O. Roberts, The pseudo-marginal approach for efficient Monte Carlo computations, Ann. Statist. 37 (2009), no. 2, 697–725. MR 2502648, DOI 10.1214/07-AOS574
N. Aronszajn, Theory of reproducing kernels, Trans. Amer. Math. Soc. 68 (1950), 337–404. MR 51437, DOI 10.1090/S0002-9947-1950-0051437-7
S. R. Arridge, J. P. Kaipio, V. Kolehmainen, M. Schweiger, E. Somersalo, T. Tarvainen, and M. Vauhkonen, Approximation errors and model reduction with an application in optical diffusion tomography, Inverse Problems 22 (2006), no. 1, 175–195. MR 2194190, DOI 10.1088/0266-5611/22/1/010
Ivo Babuška, Fabio Nobile, and Raúl Tempone, A stochastic collocation method for elliptic partial differential equations with random input data, SIAM Rev. 52 (2010), no. 2, 317–355. MR 2646806, DOI 10.1137/100786356
I. Bilionis, N. Zabaras, B. A. Konomi, and G. Lin, Multi-output separable Gaussian process: Towards an efficient, fully Bayesian paradigm for uncertainty quantification, Journal of Computational Physics 241 (2013), 212–239.
Nikolay Bliznyuk, David Ruppert, Christine Shoemaker, Rommel Regis, Stefan Wild, and Pradeep Mugunthan, Bayesian calibration and uncertainty analysis for computationally expensive models using optimization and radial basis function approximation, J. Comput. Graph. Statist. 17 (2008), no. 2, 270–294. MR 2439960, DOI 10.1198/106186008X320681
Vladimir I. Bogachev, Gaussian measures, Mathematical Surveys and Monographs, vol. 62, American Mathematical Society, Providence, RI, 1998. MR 1642391, DOI 10.1090/surv/062
T. Bui-Thanh, K. Willcox, and O. Ghattas, Model reduction for large-scale systems with high-dimensional parametric input space, SIAM J. Sci. Comput. 30 (2008), no. 6, 3270–3288. MR 2452388, DOI 10.1137/070694855
Albert Cohen, Ronald Devore, and Christoph Schwab, Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDE’s, Anal. Appl. (Singap.) 9 (2011), no. 1, 11–47. MR 2763359, DOI 10.1142/S0219530511001728
P. R. Conrad, Y. M. Marzouk, N. S. Pillai, and A. Smith, Asymptotically exact MCMC algorithms via local approximations of computationally intensive models, J. Amer. Statist. Assoc. 111 (2016), 1591–1607.
Paul G. Constantine, Active subspaces, SIAM Spotlights, vol. 2, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2015. Emerging ideas for dimension reduction in parameter studies. MR 3486165, DOI 10.1137/1.9781611973860
Paul G. Constantine, Eric Dow, and Qiqi Wang, Active subspace methods in theory and practice: applications to kriging surfaces, SIAM J. Sci. Comput. 36 (2014), no. 4, A1500–A1524. MR 3233940, DOI 10.1137/130916138
S. L. Cotter, G. O. Roberts, A. M. Stuart, and D. White, MCMC methods for functions: modifying old algorithms to make them faster, Statist. Sci. 28 (2013), no. 3, 424–446. MR 3135540, DOI 10.1214/13-STS421
M. Dashti and A.M.Stuart, The Bayesian Approach to Inverse Problems, Handbook of Uncertainty Quantification (R. Ghanem, D. Higdon, and H. Owhadi, eds.), Springer.
Mark Girolami and Ben Calderhead, Riemann manifold Langevin and Hamiltonian Monte Carlo methods, J. R. Stat. Soc. Ser. B Stat. Methodol. 73 (2011), no. 2, 123–214. With discussion and a reply by the authors. MR 2814492, DOI 10.1111/j.1467-9868.2010.00765.x
Markus Hansen and Christoph Schwab, Sparse adaptive approximation of high dimensional parametric initial value problems, Vietnam J. Math. 41 (2013), no. 2, 181–215. MR 3089816, DOI 10.1007/s10013-013-0011-9
W. K. Hastings, Monte Carlo sampling methods using Markov chains and their applications, Biometrika 57 (1970), no. 1, 97–109. MR 3363437, DOI 10.1093/biomet/57.1.97
Dave Higdon, Marc Kennedy, James C. Cavendish, John A. Cafeo, and Robert D. Ryne, Combining field data and computer simulations for calibration and prediction, SIAM J. Sci. Comput. 26 (2004), no. 2, 448–466. MR 2116355, DOI 10.1137/S1064827503426693
Jari Kaipio and Erkki Somersalo, Statistical and computational inverse problems, Applied Mathematical Sciences, vol. 160, Springer-Verlag, New York, 2005. MR 2102218
Marc C. Kennedy and Anthony O’Hagan, Bayesian calibration of computer models, J. R. Stat. Soc. Ser. B Stat. Methodol. 63 (2001), no. 3, 425–464. MR 1858398, DOI 10.1111/1467-9868.00294
Youssef Marzouk and Dongbin Xiu, A stochastic collocation approach to Bayesian inference in inverse problems, Commun. Comput. Phys. 6 (2009), no. 4, 826–847. MR 2672325, DOI 10.4208/cicp.2009.v6.p826
Youssef M. Marzouk, Habib N. Najm, and Larry A. Rahn, Stochastic spectral methods for efficient Bayesian solution of inverse problems, J. Comput. Phys. 224 (2007), no. 2, 560–586. MR 2330284, DOI 10.1016/j.jcp.2006.10.010
Bertil Matérn, Spatial variation, 2nd ed., Lecture Notes in Statistics, vol. 36, Springer-Verlag, Berlin, 1986. With a Swedish summary. MR 867886, DOI 10.1007/978-1-4615-7892-5
J. Mercer, Functions of positive and negative type, and their connection with the theory of integral equations, Philosophical Transactions of the Royal Society of London, Series A 209 (1909), 415–446.
N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller, Equation of state calculations by fast computing machines, J. Chemical Physics 21 (1953), 1087.
Francis J. Narcowich, Joseph D. Ward, and Holger Wendland, Sobolev bounds on functions with scattered zeros, with applications to radial basis function surface fitting, Math. Comp. 74 (2005), no. 250, 743–763. MR 2114646, DOI 10.1090/S0025-5718-04-01708-9
Francis J. Narcowich, Joseph D. Ward, and Holger Wendland, Sobolev error estimates and a Bernstein inequality for scattered data interpolation via radial basis functions, Constr. Approx. 24 (2006), no. 2, 175–186. MR 2239119, DOI 10.1007/s00365-005-0624-7
Harald Niederreiter, Random number generation and quasi-Monte Carlo methods, CBMS-NSF Regional Conference Series in Applied Mathematics, vol. 63, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 1992. MR 1172997, DOI 10.1137/1.9781611970081
A. O’Hagan, Bayesian analysis of computer code outputs: a tutorial, Reliability Engineering & System Safety 91 (2006), no. 10, 1290–1300.
Giuseppe Da Prato and Jerzy Zabczyk, Stochastic equations in infinite dimensions, Encyclopedia of Mathematics and its Applications, vol. 44, Cambridge University Press, Cambridge, 1992. MR 1207136, DOI 10.1017/CBO9780511666223
Carl Edward Rasmussen and Christopher K. I. Williams, Gaussian processes for machine learning, Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, 2006. MR 2514435
Patrick Rebeschini and Ramon van Handel, Can local particle filters beat the curse of dimensionality?, Ann. Appl. Probab. 25 (2015), no. 5, 2809–2866. MR 3375889, DOI 10.1214/14-AAP1061
Christian P. Robert and George Casella, Monte Carlo statistical methods, Springer Texts in Statistics, Springer-Verlag, New York, 1999. MR 1707311, DOI 10.1007/978-1-4757-3071-5
Walter Rudin, Principles of mathematical analysis, 2nd ed., McGraw-Hill Book Co., New York, 1964. MR 0166310
Jerome Sacks, William J. Welch, Toby J. Mitchell, and Henry P. Wynn, Design and analysis of computer experiments, Statist. Sci. 4 (1989), no. 4, 409–435. With comments and a rejoinder by the authors. MR 1041765
M. Scheuerer, R. Schaback, and M. Schlather, Interpolation of spatial data—a stochastic or a deterministic problem?, European J. Appl. Math. 24 (2013), no. 4, 601–629. MR 3082868, DOI 10.1017/S0956792513000016
Cl. Schillings and Ch. Schwab, Sparsity in Bayesian inversion of parametric operator equations, Inverse Problems 30 (2014), no. 6, 065007, 30. MR 3224127, DOI 10.1088/0266-5611/30/6/065007
M. Sinsbeck and W. Nowak, Sequential design of computer experiments for the solution of Bayesian inverse problems with process emulators, SIAM/ASA Journal on Uncertainty Quantification, to appear
Michael L. Stein, Interpolation of spatial data, Springer Series in Statistics, Springer-Verlag, New York, 1999. Some theory for Kriging. MR 1697409, DOI 10.1007/978-1-4612-1494-6
A. M. Stuart, Inverse problems: a Bayesian perspective, Acta Numer. 19 (2010), 451–559. MR 2652785, DOI 10.1017/S0962492910000061
Wolfgang Walter, Ordinary differential equations, Graduate Texts in Mathematics, vol. 182, Springer-Verlag, New York, 1998. Translated from the sixth German (1996) edition by Russell Thompson; Readings in Mathematics. MR 1629775, DOI 10.1007/978-1-4612-0601-9
Holger Wendland, Scattered data approximation, Cambridge Monographs on Applied and Computational Mathematics, vol. 17, Cambridge University Press, Cambridge, 2005. MR 2131724
Zong Min Wu and Robert Schaback, Local error estimates for radial basis function interpolation of scattered data, IMA J. Numer. Anal. 13 (1993), no. 1, 13–27. MR 1199027, DOI 10.1093/imanum/13.1.13
Dongbin Xiu and George Em Karniadakis, Modeling uncertainty in flow simulations via generalized polynomial chaos, J. Comput. Phys. 187 (2003), no. 1, 137–167. MR 1977783, DOI 10.1016/S0021-9991(03)00092-5