Non-centered parametric variational Bayes’ approach for hierarchical inverse problems of partial differential equations
HTML articles powered by AMS MathViewer
- by Jiaming Sui and Junxiong Jia
- Math. Comp. 93 (2024), 1715-1760
- DOI: https://doi.org/10.1090/mcom/3906
- Published electronically: November 15, 2023
- HTML | PDF | Request permission
Abstract:
This paper proposes a non-centered parameterization based infinite-dimensional mean-field variational inference (NCP-iMFVI) approach for solving the hierarchical Bayesian inverse problems. This method can generate available estimates from the approximated posterior distribution efficiently. To avoid the mutually singular obstacle that occurred in the infinite-dimensional hierarchical approach, we propose a rigorous theory of the non-centered variational Bayesian approach. Since the non-centered parameterization weakens the connection between the parameter and the hyper-parameter, we can introduce the hyper-parameter to all terms of the eigendecomposition of the prior covariance operator. We also show the relationships between the NCP-iMFVI and infinite-dimensional hierarchical approaches with centered parameterization. The proposed algorithm is applied to three inverse problems governed by the simple smooth equation, the Helmholtz equation, and the steady-state Darcy flow equation. Numerical results confirm our theoretical findings, illustrate the efficiency of solving the iMFVI problem formulated by large-scale linear and non-linear statistical inverse problems, and verify the mesh-independent property.References
- Sergios Agapiou, Johnathan M. Bardsley, Omiros Papaspiliopoulos, and Andrew M. Stuart, Analysis of the Gibbs sampler for hierarchical inverse problems, SIAM/ASA J. Uncertain. Quantif. 2 (2014), no. 1, 511–544. MR 3283919, DOI 10.1137/130944229
- Sergios Agapiou, Stig Larsson, and Andrew M. Stuart, Posterior contraction rates for the Bayesian approach to linear ill-posed inverse problems, Stochastic Process. Appl. 123 (2013), no. 10, 3828–3860. MR 3084161, DOI 10.1016/j.spa.2013.05.001
- S. Agapiou, O. Papaspiliopoulos, D. Sanz-Alonso, and A. M. Stuart, Importance sampling: intrinsic dimension and computational cost, Statist. Sci. 32 (2017), no. 3, 405–431. MR 3696003, DOI 10.1214/17-STS611
- Simon Arridge, Peter Maass, Ozan Öktem, and Carola-Bibiane Schönlieb, Solving inverse problems using data-driven models, Acta Numer. 28 (2019), 1–174. MR 3963505, DOI 10.1017/s0962492919000059
- Gang Bao, Shui-Nee Chow, Peijun Li, and Haomin Zhou, Numerical solution of an inverse medium scattering problem with a stochastic source, Inverse Probl. 26 (2010), no. 7, 074014.
- Gang Bao, Peijun Li, Junshan Lin, and Faouzi Triki, Inverse scattering problems with multi-frequencies, Inverse Problems 31 (2015), no. 9, 093001, 21. MR 3401991, DOI 10.1088/0266-5611/31/9/093001
- Alexandros Beskos, Mark Girolami, Shiwei Lan, Patrick E. Farrell, and Andrew M. Stuart, Geometric MCMC for infinite-dimensional inverse problems, J. Comput. Phys. 335 (2017), 327–351. MR 3612501, DOI 10.1016/j.jcp.2016.12.041
- Alexandros Beskos, Ajay Jasra, Ege A. Muzaffer, and Andrew M. Stuart, Sequential Monte Carlo methods for Bayesian elliptic inverse problems, Stat. Comput. 25 (2015), no. 4, 727–737. MR 3360488, DOI 10.1007/s11222-015-9556-7
- Christopher M. Bishop, Pattern recognition and machine learning, Information Science and Statistics, Springer, New York, 2006. MR 2247587, DOI 10.1007/978-0-387-45528-0
- David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe, Variational inference: a review for statisticians, J. Amer. Statist. Assoc. 112 (2017), no. 518, 859–877. MR 3671776, DOI 10.1080/01621459.2017.1285773
- Tan Bui-Thanh and Omar Ghattas, Analysis of the Hessian for inverse scattering problems: I. Inverse shape scattering of acoustic waves, Inverse Problems 28 (2012), no. 5, 055001, 32. MR 2915289, DOI 10.1088/0266-5611/28/5/055001
- Tan Bui-Thanh, Omar Ghattas, James Martin, and Georg Stadler, A computational framework for infinite-dimensional Bayesian inverse problems Part I: The linearized case, with application to global seismic inversion, SIAM J. Sci. Comput. 35 (2013), no. 6, A2494–A2523. MR 3126997, DOI 10.1137/12089586X
- Tan Bui-Thanh and Quoc P. Nguyen, FEM-based discretization-invariant MCMC methods for PDE-constrained Bayesian inverse problems, Inverse Probl. Imaging 10 (2016), no. 4, 943–975. MR 3610747, DOI 10.3934/ipi.2016028
- Daniela Calvetti, Matthew Dunlop, Erkki Somersalo, and Andrew Stuart, Iterative updating of model error for Bayesian inversion, Inverse Problems 34 (2018), no. 2, 025008, 38. MR 3751052, DOI 10.1088/1361-6420/aaa34d
- Daniela Calvetti, Harri Hakula, Sampsa Pursiainen, and Erkki Somersalo, Conditionally Gaussian hypermodels for cerebral source localization, SIAM J. Imaging Sci. 2 (2009), no. 3, 879–909. MR 2540174, DOI 10.1137/080723995
- Daniela Calvetti and Erkki Somersalo, Hypermodels in the Bayesian imaging framework, Inverse Problems 24 (2008), no. 3, 034013, 20. MR 2421950, DOI 10.1088/0266-5611/24/3/034013
- Peng Chen and Omar Ghattas, Stein variational reduced basis Bayesian inversion, SIAM J. Sci. Comput. 43 (2021), no. 2, A1163–A1193. MR 4235177, DOI 10.1137/20M1321589
- Peng Chen, Keyi Wu, Joshua Chen, Tom O’Leary-Roseberry, and Omar Ghattas, Projected Stein Variational Newton: A Fast and Scalable Bayesian Inference Method in High Dimensions, Advances in Neural Information Processing Systems, vol. 32, 2019, pp. 15104–15113.
- Victor Chen, Matthew M. Dunlop, Omiros Papaspiliopoulos, and Andrew M. Stuart, Dimension-robust MCMC in Bayesian inverse problems, arXiv preprint arXiv:1803.03344 (2018).
- S. L. Cotter, M. Dashti, J. C. Robinson, and A. M. Stuart, Bayesian inverse problems for functions and applications to fluid mechanics, Inverse Problems 25 (2009), no. 11, 115008, 43. MR 2558668, DOI 10.1088/0266-5611/25/11/115008
- S. L. Cotter, G. O. Roberts, A. M. Stuart, and D. White, MCMC methods for functions: modifying old algorithms to make them faster, Statist. Sci. 28 (2013), no. 3, 424–446. MR 3135540, DOI 10.1214/13-STS421
- Giuseppe Da Prato, An introduction to infinite-dimensional analysis, Universitext, Springer-Verlag, Berlin, 2006. Revised and extended from the 2001 original by Da Prato. MR 2244975, DOI 10.1007/3-540-29021-4
- Giuseppe Da Prato and Jerzy Zabczyk, Stochastic equations in infinite dimensions, 2nd ed., Encyclopedia of Mathematics and its Applications, vol. 152, Cambridge University Press, Cambridge, 2014. MR 3236753, DOI 10.1017/CBO9781107295513
- M. Dashti, K. J. H. Law, A. M. Stuart, and J. Voss, MAP estimators and their consistency in Bayesian nonparametric inverse problems, Inverse Problems 29 (2013), no. 9, 095017, 27. MR 3104933, DOI 10.1088/0266-5611/29/9/095017
- Masoumeh Dashti and Andrew M. Stuart, The Bayesian approach to inverse problems, Handbook of uncertainty quantification. Vol. 1, 2, 3, Springer, Cham, 2017, pp. 311–428. MR 3839555
- Matthew M. Dunlop, Tapio Helin, and Andrew M. Stuart, Hyperparameter estimation in Bayesian MAP estimation: parameterizations and consistency, SMAI J. Comput. Math. 6 (2020), 69–100. MR 4100532, DOI 10.5802/smai-jcm.62
- Matthew M. Dunlop, Marco A. Iglesias, and Andrew M. Stuart, Hierarchical Bayesian level set inversion, Stat. Comput. 27 (2017), no. 6, 1555–1584. MR 3687326, DOI 10.1007/s11222-016-9704-8
- Heinz W. Engl, Martin Hanke, and Andreas Neubauer, Regularization of inverse problems, Mathematics and its Applications, vol. 375, Kluwer Academic Publishers Group, Dordrecht, 1996. MR 1408680, DOI 10.1007/978-94-009-1740-8
- Zhe Feng and Jinglai Li, An adaptive independence sampler MCMC algorithm for Bayesian inferences of functions, SIAM J. Sci. Comput. 40 (2018), no. 3, A1301–A1321. MR 3799061, DOI 10.1137/15M1021751
- Andreas Fichtner, Full Seismic Waveform Modelling and Inversion, Springer, Heidelberg, 2010.
- Omar Ghattas and Karen Willcox, Learning physics-based models from data: perspectives from inverse problems and model reduction, Acta Numer. 30 (2021), 445–554. MR 4298221, DOI 10.1017/S0962492921000064
- Nilabja Guha, Xiaoqing Wu, Yalchin Efendiev, Bangti Jin, and Bani K. Mallick, A variational Bayesian approach for inverse problems with skew-t error distributions, J. Comput. Phys. 301 (2015), 377–393. MR 3402736, DOI 10.1016/j.jcp.2015.07.062
- M. Hinze, R. Pinnau, M. Ulbrich, and S. Ulbrich, Optimization with PDE constraints, Mathematical Modelling: Theory and Applications, vol. 23, Springer, New York, 2009. MR 2516528, DOI 10.1007/978-1-4020-8839-1_{3}
- Junxiong Jia, Peijun Li, and Deyu Meng, Stein variational gradient descent on infinite-dimensional space and applications to statistical inverse problems, SIAM J. Numer. Anal. 60 (2022), no. 4, 2225–2252. MR 4469515, DOI 10.1137/21M1440773
- Junxiong Jia, Bangyu Wu, Jigen Peng, and Jinghuai Gao, Recursive linearization method for inverse medium scattering problems with complex mixture Gaussian error learning, Inverse Problems 35 (2019), no. 7, 075003, 29. MR 3975379, DOI 10.1088/1361-6420/ab08f2
- Junxiong Jia, Yanni Wu, Peijun Li, and Deyu Meng, Variational inverting network for statistical inverse problems of partial differential equations, J. Mach. Learn. Res. 24 (2023), Paper No. [201], 60. MR 4633590, DOI 10.1515/ijnsns-2020-0155
- Junxiong Jia, Qian Zhao, Zongben Xu, Deyu Meng, and Yee Leung, Variational Bayes’ method for functions with applications to some inverse problems, SIAM J. Sci. Comput. 43 (2021), no. 1, A355–A383. MR 4205093, DOI 10.1137/19M130409X
- Bangti Jin, A variational Bayesian method to inverse problems with impulsive noise, J. Comput. Phys. 231 (2012), no. 2, 423–435. MR 2872083, DOI 10.1016/j.jcp.2011.09.009
- Bangti Jin and Jun Zou, Hierarchical Bayesian inference for ill-posed problems via variational method, J. Comput. Phys. 229 (2010), no. 19, 7317–7343. MR 2677781, DOI 10.1016/j.jcp.2010.06.016
- Jari Kaipio and Erkki Somersalo, Statistical and computational inverse problems, Applied Mathematical Sciences, vol. 160, Springer-Verlag, New York, 2005. MR 2102218, DOI 10.1007/b138659
- Andreas Kirsch, An introduction to the mathematical theory of inverse problems, 2nd ed., Applied Mathematical Sciences, vol. 120, Springer, New York, 2011. MR 3025302, DOI 10.1007/978-1-4419-8474-6
- Matti Lassas and Samuli Siltanen, Can one use total variation prior for edge-preserving Bayesian inversion?, Inverse Problems 20 (2004), no. 5, 1537–1563. MR 2109134, DOI 10.1088/0266-5611/20/5/013
- Omiros Papaspiliopoulos and Gareth Roberts, Stability of the Gibbs sampler for Bayesian hierarchical models, Ann. Statist. 36 (2008), no. 1, 95–117. MR 2387965, DOI 10.1214/009053607000000749
- Omiros Papaspiliopoulos, Gareth O. Roberts, and Martin Sköld, Non-centered parameterizations for hierarchical models and data augmentation, Bayesian statistics, 7 (Tenerife, 2002) Oxford Univ. Press, New York, 2003, pp. 307–326. With a discussion by Alan E. Gelfand, Ole F. Christensen and Darren J. Wilkinson, and a reply by the authors. MR 2003180
- Omiros Papaspiliopoulos, Gareth O. Roberts, and Martin Sköld, A general framework for the parametrization of hierarchical models, Statist. Sci. 22 (2007), no. 1, 59–73. MR 2408661, DOI 10.1214/088342307000000014
- Noemi Petra, James Martin, Georg Stadler, and Omar Ghattas, A computational framework for infinite-dimensional Bayesian inverse problems, Part II: Stochastic Newton MCMC with application to ice sheet flow inverse problems, SIAM J. Sci. Comput. 36 (2014), no. 4, A1525–A1555. MR 3233941, DOI 10.1137/130934805
- Natesh S. Pillai, Andrew M. Stuart, and Alexandre H. Thiéry, Noisy gradient flow from a random walk in Hilbert space, Stoch. Partial Differ. Equ. Anal. Comput. 2 (2014), no. 2, 196–232. MR 3249584, DOI 10.1007/s40072-014-0029-3
- F. J. Pinski, G. Simpson, A. M. Stuart, and H. Weber, Algorithms for Kullback-Leibler approximation of probability measures in infinite dimensions, SIAM J. Sci. Comput. 37 (2015), no. 6, A2733–A2757. MR 3424069, DOI 10.1137/14098171X
- F. J. Pinski, G. Simpson, A. M. Stuart, and H. Weber, Kullback-Leibler approximation for probability measures on infinite dimensional spaces, SIAM J. Math. Anal. 47 (2015), no. 6, 4091–4122. MR 3419882, DOI 10.1137/140962802
- Michael Reed and Barry Simon, Methods of modern mathematical physics. IV. Analysis of operators, Academic Press [Harcourt Brace Jovanovich, Publishers], New York-London, 1978. MR 493421
- Arvind K. Saibaba, Jonghyun Lee, and Peter K. Kitanidis, Randomized algorithms for generalized Hermitian eigenvalue problems with application to computing Karhunen-Loève expansion, Numer. Linear Algebra Appl. 23 (2016), no. 2, 314–339. MR 3457529, DOI 10.1002/nla.2026
- A. M. Stuart, Inverse problems: a Bayesian perspective, Acta Numer. 19 (2010), 451–559. MR 2652785, DOI 10.1017/S0962492910000061
- Dustin Tran, Rajesh Ranganath, and David M. Blei, The variational Gaussian process, arXiv preprint arXiv:1511.06499 (2015).
- Umberto Villa, Noemi Petra, and Omar Ghattas, HIPPYlib: an extensible software framework for large-scale inverse problems governed by PDEs: Part I: Deterministic inversion and linearized Bayesian inference, ACM Trans. Math. Software 47 (2021), no. 2, Art. 16, 34. MR 4254158, DOI 10.1145/3428447
- Yixin Wang and David M. Blei, Frequentist consistency of variational Bayes, J. Amer. Statist. Assoc. 114 (2019), no. 527, 1147–1161. MR 4011769, DOI 10.1080/01621459.2018.1473776
- Arthur B. Weglein, Fernanda V. Araújo, Paulo M. Carvalho, Robert H. Stolt, Kenneth H. Matson, Richard T. Coates, Dennis Corrigan, Douglas J. Foster, Simon A. Shaw, and Haiyan Zhang, Inverse scattering series and seismic exploration, Inverse Problems 19 (2003), no. 6, R27–R83. MR 2036527, DOI 10.1088/0266-5611/19/6/R01
- Yau Shu Wong and Guangrui Li, Exact finite difference schemes for solving Helmholtz equation at any wavenumber, Int. J. Numer. Anal. Model. Ser. B 2 (2011), no. 1, 91–108. MR 2866996
- Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt, Advances in variational inference, IEEE Trans. Pattern Anal. Mach. Intelligence 41 (2018), no. 8, 2008–2026.
- Fengshuo Zhang and Chao Gao, Convergence rates of variational posterior distributions, Ann. Statist. 48 (2020), no. 4, 2180–2207. MR 4134791, DOI 10.1214/19-AOS1883
- Qingping Zhou, Tengchao Yu, Xiaoqun Zhang, and Jinglai Li, Bayesian inference and uncertainty quantification for medical image reconstruction with Poisson data, SIAM J. Imaging Sci. 13 (2020), no. 1, 29–52. MR 4048008, DOI 10.1137/19M1248352
Bibliographic Information
- Jiaming Sui
- Affiliation: School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, People’s Republic of China
- ORCID: 0009-0006-2033-4800
- Email: sjming1997327@stu.xjtu.edu.cn
- Junxiong Jia
- Affiliation: School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, People’s Republic of China
- MR Author ID: 947692
- ORCID: 0000-0002-0917-2381
- Email: jjx323@xjtu.edu.cn
- Received by editor(s): November 19, 2022
- Received by editor(s) in revised form: July 30, 2023, and August 25, 2023
- Published electronically: November 15, 2023
- Additional Notes: This work was supported by the NSFC grants 12271428, 12322116 and the Major projects of the NSFC grants 12090021, 12090020 and in part by the Natural Science Basic Research Plan in Shaanxi Province of China under Grant 2023-JC-QN-0035.
The second author is the corresponding author. - © Copyright 2023 American Mathematical Society
- Journal: Math. Comp. 93 (2024), 1715-1760
- MSC (2020): Primary 65L09, 35R30, 49N45, 62F15
- DOI: https://doi.org/10.1090/mcom/3906