Non-centered parametric variational Bayes’ approach for hierarchical inverse problems of partial differential equations

Sui, Jiaming; Jia, Junxiong

doi:10.1090/mcom/3906

Non-centered parametric variational Bayes’ approach for hierarchical inverse problems of partial differential equations
HTML articles powered by AMS MathViewer

by Jiaming Sui and Junxiong Jia

Math. Comp. 93 (2024), 1715-1760

DOI: https://doi.org/10.1090/mcom/3906

Published electronically: November 15, 2023

HTML | PDF | Request permission

Abstract:

This paper proposes a non-centered parameterization based infinite-dimensional mean-field variational inference (NCP-iMFVI) approach for solving the hierarchical Bayesian inverse problems. This method can generate available estimates from the approximated posterior distribution efficiently. To avoid the mutually singular obstacle that occurred in the infinite-dimensional hierarchical approach, we propose a rigorous theory of the non-centered variational Bayesian approach. Since the non-centered parameterization weakens the connection between the parameter and the hyper-parameter, we can introduce the hyper-parameter to all terms of the eigendecomposition of the prior covariance operator. We also show the relationships between the NCP-iMFVI and infinite-dimensional hierarchical approaches with centered parameterization. The proposed algorithm is applied to three inverse problems governed by the simple smooth equation, the Helmholtz equation, and the steady-state Darcy flow equation. Numerical results confirm our theoretical findings, illustrate the efficiency of solving the iMFVI problem formulated by large-scale linear and non-linear statistical inverse problems, and verify the mesh-independent property.

References

Sergios Agapiou, Johnathan M. Bardsley, Omiros Papaspiliopoulos, and Andrew M. Stuart, Analysis of the Gibbs sampler for hierarchical inverse problems, SIAM/ASA J. Uncertain. Quantif. 2 (2014), no. 1, 511–544. MR 3283919, DOI 10.1137/130944229
Sergios Agapiou, Stig Larsson, and Andrew M. Stuart, Posterior contraction rates for the Bayesian approach to linear ill-posed inverse problems, Stochastic Process. Appl. 123 (2013), no. 10, 3828–3860. MR 3084161, DOI 10.1016/j.spa.2013.05.001
S. Agapiou, O. Papaspiliopoulos, D. Sanz-Alonso, and A. M. Stuart, Importance sampling: intrinsic dimension and computational cost, Statist. Sci. 32 (2017), no. 3, 405–431. MR 3696003, DOI 10.1214/17-STS611
Simon Arridge, Peter Maass, Ozan Öktem, and Carola-Bibiane Schönlieb, Solving inverse problems using data-driven models, Acta Numer. 28 (2019), 1–174. MR 3963505, DOI 10.1017/s0962492919000059
Gang Bao, Shui-Nee Chow, Peijun Li, and Haomin Zhou, Numerical solution of an inverse medium scattering problem with a stochastic source, Inverse Probl. 26 (2010), no. 7, 074014.
Gang Bao, Peijun Li, Junshan Lin, and Faouzi Triki, Inverse scattering problems with multi-frequencies, Inverse Problems 31 (2015), no. 9, 093001, 21. MR 3401991, DOI 10.1088/0266-5611/31/9/093001
Alexandros Beskos, Mark Girolami, Shiwei Lan, Patrick E. Farrell, and Andrew M. Stuart, Geometric MCMC for infinite-dimensional inverse problems, J. Comput. Phys. 335 (2017), 327–351. MR 3612501, DOI 10.1016/j.jcp.2016.12.041
Alexandros Beskos, Ajay Jasra, Ege A. Muzaffer, and Andrew M. Stuart, Sequential Monte Carlo methods for Bayesian elliptic inverse problems, Stat. Comput. 25 (2015), no. 4, 727–737. MR 3360488, DOI 10.1007/s11222-015-9556-7
Christopher M. Bishop, Pattern recognition and machine learning, Information Science and Statistics, Springer, New York, 2006. MR 2247587, DOI 10.1007/978-0-387-45528-0
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe, Variational inference: a review for statisticians, J. Amer. Statist. Assoc. 112 (2017), no. 518, 859–877. MR 3671776, DOI 10.1080/01621459.2017.1285773
Tan Bui-Thanh and Omar Ghattas, Analysis of the Hessian for inverse scattering problems: I. Inverse shape scattering of acoustic waves, Inverse Problems 28 (2012), no. 5, 055001, 32. MR 2915289, DOI 10.1088/0266-5611/28/5/055001
Tan Bui-Thanh, Omar Ghattas, James Martin, and Georg Stadler, A computational framework for infinite-dimensional Bayesian inverse problems Part I: The linearized case, with application to global seismic inversion, SIAM J. Sci. Comput. 35 (2013), no. 6, A2494–A2523. MR 3126997, DOI 10.1137/12089586X
Tan Bui-Thanh and Quoc P. Nguyen, FEM-based discretization-invariant MCMC methods for PDE-constrained Bayesian inverse problems, Inverse Probl. Imaging 10 (2016), no. 4, 943–975. MR 3610747, DOI 10.3934/ipi.2016028
Daniela Calvetti, Matthew Dunlop, Erkki Somersalo, and Andrew Stuart, Iterative updating of model error for Bayesian inversion, Inverse Problems 34 (2018), no. 2, 025008, 38. MR 3751052, DOI 10.1088/1361-6420/aaa34d
Daniela Calvetti, Harri Hakula, Sampsa Pursiainen, and Erkki Somersalo, Conditionally Gaussian hypermodels for cerebral source localization, SIAM J. Imaging Sci. 2 (2009), no. 3, 879–909. MR 2540174, DOI 10.1137/080723995
Daniela Calvetti and Erkki Somersalo, Hypermodels in the Bayesian imaging framework, Inverse Problems 24 (2008), no. 3, 034013, 20. MR 2421950, DOI 10.1088/0266-5611/24/3/034013
Peng Chen and Omar Ghattas, Stein variational reduced basis Bayesian inversion, SIAM J. Sci. Comput. 43 (2021), no. 2, A1163–A1193. MR 4235177, DOI 10.1137/20M1321589
Peng Chen, Keyi Wu, Joshua Chen, Tom O’Leary-Roseberry, and Omar Ghattas, Projected Stein Variational Newton: A Fast and Scalable Bayesian Inference Method in High Dimensions, Advances in Neural Information Processing Systems, vol. 32, 2019, pp. 15104–15113.
Victor Chen, Matthew M. Dunlop, Omiros Papaspiliopoulos, and Andrew M. Stuart, Dimension-robust MCMC in Bayesian inverse problems, arXiv preprint arXiv:1803.03344 (2018).
S. L. Cotter, M. Dashti, J. C. Robinson, and A. M. Stuart, Bayesian inverse problems for functions and applications to fluid mechanics, Inverse Problems 25 (2009), no. 11, 115008, 43. MR 2558668, DOI 10.1088/0266-5611/25/11/115008
S. L. Cotter, G. O. Roberts, A. M. Stuart, and D. White, MCMC methods for functions: modifying old algorithms to make them faster, Statist. Sci. 28 (2013), no. 3, 424–446. MR 3135540, DOI 10.1214/13-STS421
Giuseppe Da Prato, An introduction to infinite-dimensional analysis, Universitext, Springer-Verlag, Berlin, 2006. Revised and extended from the 2001 original by Da Prato. MR 2244975, DOI 10.1007/3-540-29021-4
Giuseppe Da Prato and Jerzy Zabczyk, Stochastic equations in infinite dimensions, 2nd ed., Encyclopedia of Mathematics and its Applications, vol. 152, Cambridge University Press, Cambridge, 2014. MR 3236753, DOI 10.1017/CBO9781107295513
M. Dashti, K. J. H. Law, A. M. Stuart, and J. Voss, MAP estimators and their consistency in Bayesian nonparametric inverse problems, Inverse Problems 29 (2013), no. 9, 095017, 27. MR 3104933, DOI 10.1088/0266-5611/29/9/095017
Masoumeh Dashti and Andrew M. Stuart, The Bayesian approach to inverse problems, Handbook of uncertainty quantification. Vol. 1, 2, 3, Springer, Cham, 2017, pp. 311–428. MR 3839555
Matthew M. Dunlop, Tapio Helin, and Andrew M. Stuart, Hyperparameter estimation in Bayesian MAP estimation: parameterizations and consistency, SMAI J. Comput. Math. 6 (2020), 69–100. MR 4100532, DOI 10.5802/smai-jcm.62
Matthew M. Dunlop, Marco A. Iglesias, and Andrew M. Stuart, Hierarchical Bayesian level set inversion, Stat. Comput. 27 (2017), no. 6, 1555–1584. MR 3687326, DOI 10.1007/s11222-016-9704-8
Heinz W. Engl, Martin Hanke, and Andreas Neubauer, Regularization of inverse problems, Mathematics and its Applications, vol. 375, Kluwer Academic Publishers Group, Dordrecht, 1996. MR 1408680, DOI 10.1007/978-94-009-1740-8
Zhe Feng and Jinglai Li, An adaptive independence sampler MCMC algorithm for Bayesian inferences of functions, SIAM J. Sci. Comput. 40 (2018), no. 3, A1301–A1321. MR 3799061, DOI 10.1137/15M1021751
Andreas Fichtner, Full Seismic Waveform Modelling and Inversion, Springer, Heidelberg, 2010.
Omar Ghattas and Karen Willcox, Learning physics-based models from data: perspectives from inverse problems and model reduction, Acta Numer. 30 (2021), 445–554. MR 4298221, DOI 10.1017/S0962492921000064
Nilabja Guha, Xiaoqing Wu, Yalchin Efendiev, Bangti Jin, and Bani K. Mallick, A variational Bayesian approach for inverse problems with skew-t error distributions, J. Comput. Phys. 301 (2015), 377–393. MR 3402736, DOI 10.1016/j.jcp.2015.07.062
M. Hinze, R. Pinnau, M. Ulbrich, and S. Ulbrich, Optimization with PDE constraints, Mathematical Modelling: Theory and Applications, vol. 23, Springer, New York, 2009. MR 2516528, DOI 10.1007/978-1-4020-8839-1_{3}
Junxiong Jia, Peijun Li, and Deyu Meng, Stein variational gradient descent on infinite-dimensional space and applications to statistical inverse problems, SIAM J. Numer. Anal. 60 (2022), no. 4, 2225–2252. MR 4469515, DOI 10.1137/21M1440773
Junxiong Jia, Bangyu Wu, Jigen Peng, and Jinghuai Gao, Recursive linearization method for inverse medium scattering problems with complex mixture Gaussian error learning, Inverse Problems 35 (2019), no. 7, 075003, 29. MR 3975379, DOI 10.1088/1361-6420/ab08f2
Junxiong Jia, Yanni Wu, Peijun Li, and Deyu Meng, Variational inverting network for statistical inverse problems of partial differential equations, J. Mach. Learn. Res. 24 (2023), Paper No. [201], 60. MR 4633590, DOI 10.1515/ijnsns-2020-0155
Junxiong Jia, Qian Zhao, Zongben Xu, Deyu Meng, and Yee Leung, Variational Bayes’ method for functions with applications to some inverse problems, SIAM J. Sci. Comput. 43 (2021), no. 1, A355–A383. MR 4205093, DOI 10.1137/19M130409X
Bangti Jin, A variational Bayesian method to inverse problems with impulsive noise, J. Comput. Phys. 231 (2012), no. 2, 423–435. MR 2872083, DOI 10.1016/j.jcp.2011.09.009
Bangti Jin and Jun Zou, Hierarchical Bayesian inference for ill-posed problems via variational method, J. Comput. Phys. 229 (2010), no. 19, 7317–7343. MR 2677781, DOI 10.1016/j.jcp.2010.06.016
Jari Kaipio and Erkki Somersalo, Statistical and computational inverse problems, Applied Mathematical Sciences, vol. 160, Springer-Verlag, New York, 2005. MR 2102218, DOI 10.1007/b138659
Andreas Kirsch, An introduction to the mathematical theory of inverse problems, 2nd ed., Applied Mathematical Sciences, vol. 120, Springer, New York, 2011. MR 3025302, DOI 10.1007/978-1-4419-8474-6
Matti Lassas and Samuli Siltanen, Can one use total variation prior for edge-preserving Bayesian inversion?, Inverse Problems 20 (2004), no. 5, 1537–1563. MR 2109134, DOI 10.1088/0266-5611/20/5/013
Omiros Papaspiliopoulos and Gareth Roberts, Stability of the Gibbs sampler for Bayesian hierarchical models, Ann. Statist. 36 (2008), no. 1, 95–117. MR 2387965, DOI 10.1214/009053607000000749
Omiros Papaspiliopoulos, Gareth O. Roberts, and Martin Sköld, Non-centered parameterizations for hierarchical models and data augmentation, Bayesian statistics, 7 (Tenerife, 2002) Oxford Univ. Press, New York, 2003, pp. 307–326. With a discussion by Alan E. Gelfand, Ole F. Christensen and Darren J. Wilkinson, and a reply by the authors. MR 2003180
Omiros Papaspiliopoulos, Gareth O. Roberts, and Martin Sköld, A general framework for the parametrization of hierarchical models, Statist. Sci. 22 (2007), no. 1, 59–73. MR 2408661, DOI 10.1214/088342307000000014
Noemi Petra, James Martin, Georg Stadler, and Omar Ghattas, A computational framework for infinite-dimensional Bayesian inverse problems, Part II: Stochastic Newton MCMC with application to ice sheet flow inverse problems, SIAM J. Sci. Comput. 36 (2014), no. 4, A1525–A1555. MR 3233941, DOI 10.1137/130934805
Natesh S. Pillai, Andrew M. Stuart, and Alexandre H. Thiéry, Noisy gradient flow from a random walk in Hilbert space, Stoch. Partial Differ. Equ. Anal. Comput. 2 (2014), no. 2, 196–232. MR 3249584, DOI 10.1007/s40072-014-0029-3
F. J. Pinski, G. Simpson, A. M. Stuart, and H. Weber, Algorithms for Kullback-Leibler approximation of probability measures in infinite dimensions, SIAM J. Sci. Comput. 37 (2015), no. 6, A2733–A2757. MR 3424069, DOI 10.1137/14098171X
F. J. Pinski, G. Simpson, A. M. Stuart, and H. Weber, Kullback-Leibler approximation for probability measures on infinite dimensional spaces, SIAM J. Math. Anal. 47 (2015), no. 6, 4091–4122. MR 3419882, DOI 10.1137/140962802
Michael Reed and Barry Simon, Methods of modern mathematical physics. IV. Analysis of operators, Academic Press [Harcourt Brace Jovanovich, Publishers], New York-London, 1978. MR 493421
Arvind K. Saibaba, Jonghyun Lee, and Peter K. Kitanidis, Randomized algorithms for generalized Hermitian eigenvalue problems with application to computing Karhunen-Loève expansion, Numer. Linear Algebra Appl. 23 (2016), no. 2, 314–339. MR 3457529, DOI 10.1002/nla.2026
A. M. Stuart, Inverse problems: a Bayesian perspective, Acta Numer. 19 (2010), 451–559. MR 2652785, DOI 10.1017/S0962492910000061
Dustin Tran, Rajesh Ranganath, and David M. Blei, The variational Gaussian process, arXiv preprint arXiv:1511.06499 (2015).
Umberto Villa, Noemi Petra, and Omar Ghattas, HIPPYlib: an extensible software framework for large-scale inverse problems governed by PDEs: Part I: Deterministic inversion and linearized Bayesian inference, ACM Trans. Math. Software 47 (2021), no. 2, Art. 16, 34. MR 4254158, DOI 10.1145/3428447
Yixin Wang and David M. Blei, Frequentist consistency of variational Bayes, J. Amer. Statist. Assoc. 114 (2019), no. 527, 1147–1161. MR 4011769, DOI 10.1080/01621459.2018.1473776
Arthur B. Weglein, Fernanda V. Araújo, Paulo M. Carvalho, Robert H. Stolt, Kenneth H. Matson, Richard T. Coates, Dennis Corrigan, Douglas J. Foster, Simon A. Shaw, and Haiyan Zhang, Inverse scattering series and seismic exploration, Inverse Problems 19 (2003), no. 6, R27–R83. MR 2036527, DOI 10.1088/0266-5611/19/6/R01
Yau Shu Wong and Guangrui Li, Exact finite difference schemes for solving Helmholtz equation at any wavenumber, Int. J. Numer. Anal. Model. Ser. B 2 (2011), no. 1, 91–108. MR 2866996
Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt, Advances in variational inference, IEEE Trans. Pattern Anal. Mach. Intelligence 41 (2018), no. 8, 2008–2026.
Fengshuo Zhang and Chao Gao, Convergence rates of variational posterior distributions, Ann. Statist. 48 (2020), no. 4, 2180–2207. MR 4134791, DOI 10.1214/19-AOS1883
Qingping Zhou, Tengchao Yu, Xiaoqun Zhang, and Jinglai Li, Bayesian inference and uncertainty quantification for medical image reconstruction with Poisson data, SIAM J. Imaging Sci. 13 (2020), no. 1, 29–52. MR 4048008, DOI 10.1137/19M1248352