Remote Access Mathematics of Computation
Green Open Access

Mathematics of Computation

ISSN 1088-6842(online) ISSN 0025-5718(print)

Request Permissions   Purchase Content 


Posterior consistency for Gaussian process approximations of Bayesian posterior distributions

Authors: Andrew M. Stuart and Aretha L. Teckentrup
Journal: Math. Comp. 87 (2018), 721-753
MSC (2010): Primary 60G15, 62G08, 65D05, 65D30, 65J22
Published electronically: August 3, 2017
Full-text PDF

Abstract | References | Similar Articles | Additional Information

Abstract: We study the use of Gaussian process emulators to approximate the parameter-to-observation map or the negative log-likelihood in Bayesian inverse problems. We prove error bounds on the Hellinger distance between the true posterior distribution and various approximations based on the Gaussian process emulator. Our analysis includes approximations based on the mean of the predictive process, as well as approximations based on the full Gaussian process emulator. Our results show that the Hellinger distance between the true posterior and its approximations can be bounded by moments of the error in the emulator. Numerical results confirm our theoretical findings.

References [Enhancements On Off] (What's this?)

  • [1] Robert J. Adler, The Geometry of Random Fields, John Wiley & Sons, Ltd., Chichester, 1981. Wiley Series in Probability and Mathematical Statistics. MR 611857
  • [2] Christophe Andrieu and Gareth O. Roberts, The pseudo-marginal approach for efficient Monte Carlo computations, Ann. Statist. 37 (2009), no. 2, 697-725. MR 2502648,
  • [3] N. Aronszajn, Theory of reproducing kernels, Trans. Amer. Math. Soc. 68 (1950), 337-404. MR 0051437
  • [4] S. R. Arridge, J. P. Kaipio, V. Kolehmainen, M. Schweiger, E. Somersalo, T. Tarvainen, and M. Vauhkonen, Approximation errors and model reduction with an application in optical diffusion tomography, Inverse Problems 22 (2006), no. 1, 175-195. MR 2194190,
  • [5] Ivo Babuška, Fabio Nobile, and Raúl Tempone, A stochastic collocation method for elliptic partial differential equations with random input data, SIAM Rev. 52 (2010), no. 2, 317-355. MR 2646806,
  • [6] I. Bilionis, N. Zabaras, B. A. Konomi, and G. Lin, Multi-output separable Gaussian process: Towards an efficient, fully Bayesian paradigm for uncertainty quantification, Journal of Computational Physics 241 (2013), 212-239.
  • [7] Nikolay Bliznyuk, David Ruppert, Christine Shoemaker, Rommel Regis, Stefan Wild, and Pradeep Mugunthan, Bayesian calibration and uncertainty analysis for computationally expensive models using optimization and radial basis function approximation, J. Comput. Graph. Statist. 17 (2008), no. 2, 270-294. MR 2439960,
  • [8] Vladimir I. Bogachev, Gaussian Measures, Mathematical Surveys and Monographs, vol. 62, American Mathematical Society, Providence, RI, 1998. MR 1642391
  • [9] T. Bui-Thanh, K. Willcox, and O. Ghattas, Model reduction for large-scale systems with high-dimensional parametric input space, SIAM J. Sci. Comput. 30 (2008), no. 6, 3270-3288. MR 2452388,
  • [10] Albert Cohen, Ronald Devore, and Christoph Schwab, Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDE's, Anal. Appl. (Singap.) 9 (2011), no. 1, 11-47. MR 2763359,
  • [11] P. R. Conrad, Y. M. Marzouk, N. S. Pillai, and A. Smith, Asymptotically exact MCMC algorithms via local approximations of computationally intensive models, J. Amer. Statist. Assoc. 111 (2016), 1591-1607.
  • [12] Paul G. Constantine, Active Subspaces: Emerging Ideas for Dimension Reduction in Parameter Studies, SIAM Spotlights, vol. 2, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2015. MR 3486165
  • [13] Paul G. Constantine, Eric Dow, and Qiqi Wang, Active subspace methods in theory and practice: applications to kriging surfaces, SIAM J. Sci. Comput. 36 (2014), no. 4, A1500-A1524. MR 3233940,
  • [14] S. L. Cotter, G. O. Roberts, A. M. Stuart, and D. White, MCMC methods for functions: modifying old algorithms to make them faster, Statist. Sci. 28 (2013), no. 3, 424-446. MR 3135540,
  • [15] M. Dashti and A.M.Stuart, The Bayesian Approach to Inverse Problems, Handbook of Uncertainty Quantification (R. Ghanem, D. Higdon, and H. Owhadi, eds.), Springer.
  • [16] Mark Girolami and Ben Calderhead, Riemann manifold Langevin and Hamiltonian Monte Carlo methods, J. R. Stat. Soc. Ser. B Stat. Methodol. 73 (2011), no. 2, 123-214. MR 2814492,
  • [17] Markus Hansen and Christoph Schwab, Sparse adaptive approximation of high dimensional parametric initial value problems, Vietnam J. Math. 41 (2013), no. 2, 181-215. MR 3089816,
  • [18] W. K. Hastings, Monte Carlo sampling methods using Markov chains and their applications, Biometrika 57 (1970), no. 1, 97-109. MR 3363437,
  • [19] Dave Higdon, Marc Kennedy, James C. Cavendish, John A. Cafeo, and Robert D. Ryne, Combining field data and computer simulations for calibration and prediction, SIAM J. Sci. Comput. 26 (2004), no. 2, 448-466. MR 2116355,
  • [20] Jari Kaipio and Erkki Somersalo, Statistical and Computational Inverse Problems, Applied Mathematical Sciences, vol. 160, Springer-Verlag, New York, 2005. MR 2102218
  • [21] Marc C. Kennedy and Anthony O'Hagan, Bayesian calibration of computer models, J. R. Stat. Soc. Ser. B Stat. Methodol. 63 (2001), no. 3, 425-464. MR 1858398,
  • [22] Youssef Marzouk and Dongbin Xiu, A stochastic collocation approach to Bayesian inference in inverse problems, Commun. Comput. Phys. 6 (2009), no. 4, 826-847. MR 2672325,
  • [23] Youssef M. Marzouk, Habib N. Najm, and Larry A. Rahn, Stochastic spectral methods for efficient Bayesian solution of inverse problems, J. Comput. Phys. 224 (2007), no. 2, 560-586. MR 2330284,
  • [24] Bertil Matérn, Spatial variation, 2nd ed., Lecture Notes in Statistics, vol. 36, Springer-Verlag, Berlin, 1986. MR 867886
  • [25] J. Mercer, Functions of positive and negative type, and their connection with the theory of integral equations, Philosophical Transactions of the Royal Society of London, Series A 209 (1909), 415-446.
  • [26] N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller, Equation of state calculations by fast computing machines, J. Chemical Physics 21 (1953), 1087.
  • [27] Francis J. Narcowich, Joseph D. Ward, and Holger Wendland, Sobolev bounds on functions with scattered zeros, with applications to radial basis function surface fitting, Math. Comp. 74 (2005), no. 250, 743-763. MR 2114646,
  • [28] Francis J. Narcowich, Joseph D. Ward, and Holger Wendland, Sobolev error estimates and a Bernstein inequality for scattered data interpolation via radial basis functions, Constr. Approx. 24 (2006), no. 2, 175-186. MR 2239119,
  • [29] Harald Niederreiter, Random Number Generation and Quasi-Monte Carlo Methods, CBMS-NSF Regional Conference Series in Applied Mathematics, vol. 63, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 1992. MR 1172997
  • [30] A. O'Hagan, Bayesian analysis of computer code outputs: a tutorial, Reliability Engineering & System Safety 91 (2006), no. 10, 1290-1300.
  • [31] Giuseppe Da Prato and Jerzy Zabczyk, Stochastic Equations in Infinite Dimensions, Encyclopedia of Mathematics and its Applications, vol. 44, Cambridge University Press, Cambridge, 1992. MR 1207136
  • [32] Carl Edward Rasmussen and Christopher K. I. Williams, Gaussian Processes for Machine Learning, Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, 2006. MR 2514435
  • [33] Patrick Rebeschini and Ramon van Handel, Can local particle filters beat the curse of dimensionality?, Ann. Appl. Probab. 25 (2015), no. 5, 2809-2866. MR 3375889,
  • [34] Christian P. Robert and George Casella, Monte Carlo statistical methods, Springer Texts in Statistics, Springer-Verlag, New York, 1999. MR 1707311
  • [35] Walter Rudin, Principles of Mathematical Analysis, Second edition, McGraw-Hill Book Co., New York, 1964. MR 0166310
  • [36] Jerome Sacks, William J. Welch, Toby J. Mitchell, and Henry P. Wynn, Design and analysis of computer experiments, Statist. Sci. 4 (1989), no. 4, 409-435. MR 1041765
  • [37] M. Scheuerer, R. Schaback, and M. Schlather, Interpolation of spatial data--a stochastic or a deterministic problem?, European J. Appl. Math. 24 (2013), no. 4, 601-629. MR 3082868,
  • [38] Cl. Schillings and Ch. Schwab, Sparsity in Bayesian inversion of parametric operator equations, Inverse Problems 30 (2014), no. 6, 065007, 30. MR 3224127,
  • [39] M. Sinsbeck and W. Nowak, Sequential design of computer experiments for the solution of Bayesian inverse problems with process emulators, SIAM/ASA Journal on Uncertainty Quantification, to appear
  • [40] Michael L. Stein, Interpolation of Spatial Data: Some Theory for Kriging, Springer Series in Statistics, Springer-Verlag, New York, 1999. MR 1697409
  • [41] A. M. Stuart, Inverse problems: a Bayesian perspective, Acta Numer. 19 (2010), 451-559. MR 2652785,
  • [42] Wolfgang Walter, Ordinary differential equations, Graduate Texts in Mathematics, vol. 182, Springer-Verlag, New York, 1998. MR 1629775
  • [43] Holger Wendland, Scattered Data Approximation, Cambridge Monographs on Applied and Computational Mathematics, vol. 17, Cambridge University Press, Cambridge, 2005. MR 2131724
  • [44] Zong Min Wu and Robert Schaback, Local error estimates for radial basis function interpolation of scattered data, IMA J. Numer. Anal. 13 (1993), no. 1, 13-27. MR 1199027,
  • [45] Dongbin Xiu and George Em Karniadakis, Modeling uncertainty in flow simulations via generalized polynomial chaos, J. Comput. Phys. 187 (2003), no. 1, 137-167. MR 1977783,

Similar Articles

Retrieve articles in Mathematics of Computation with MSC (2010): 60G15, 62G08, 65D05, 65D30, 65J22

Retrieve articles in all journals with MSC (2010): 60G15, 62G08, 65D05, 65D30, 65J22

Additional Information

Andrew M. Stuart
Affiliation: Mathematics Institute, Zeeman Building, University of Warwick, Coventry, CV4 7AL, England
Address at time of publication: Computing and Mathematical Sciences, Caltech, Pasadena, California 91125

Aretha L. Teckentrup
Affiliation: Mathematics Institute, Zeeman Building, University of Warwick, Coventry, CV4 7AL, England
Address at time of publication: School of Mathematics, James Clerk Maxwell Building, University of Edinburgh, EH9 3FD, Edinburgh, Scotland

Keywords: Inverse problem, Bayesian approach, surrogate model, Gaussian process regression, posterior consistency
Received by editor(s): March 7, 2016
Received by editor(s) in revised form: September 26, 2016
Published electronically: August 3, 2017
Article copyright: © Copyright 2017 American Mathematical Society

American Mathematical Society