Applications of Laplace’s method in Bayesian analysis and related topics

Miyata, Yoichi

doi:10.1090/suga/472

Applications of Laplace’s method in Bayesian analysis and related topics
HTML articles powered by AMS MathViewer

by Yoichi Miyata
Translated by: the author

Sugaku Expositions 35 (2022), 197-220

DOI: https://doi.org/10.1090/suga/472

Published electronically: August 9, 2022

PDF | Request permission

Abstract:

This article reviews some second-order approximations to the posterior mean, and the marginal distribution of observations called a marginal likelihood, where “second-order” means that the approximation has asymptotic errors of order $O(n^{-2})$ as the sample size $n$ goes to infinity. In addition, the Laplace approximations using asymptotic modes instead of the exact mode of integrand, and higher-order asymptotic expansions are presented.

References

O. E. Barndorff-Nielsen and D. R. Cox, Asymptotic techniques for use in statistics, Monographs on Statistics and Applied Probability, Chapman & Hall, London, 1989. MR 1010226, DOI 10.1007/978-1-4899-3424-6
Jose-M. Bernardo, Reference posterior distributions for Bayesian inference, J. Roy. Statist. Soc. Ser. B 41 (1979), no. 2, 113–147. With discussion. MR 547240, DOI 10.1111/j.2517-6161.1979.tb01066.x
Peter J. Bickel and J. K. Ghosh, A decomposition for the likelihood ratio statistic and the Bartlett correction—a Bayesian argument, Ann. Statist. 18 (1990), no. 3, 1070–1090. MR 1062699, DOI 10.1214/aos/1176347740
Tim Bollerslev, Generalized autoregressive conditional heteroskedasticity, J. Econometrics 31 (1986), no. 3, 307–327. MR 853051, DOI 10.1016/0304-4076(86)90063-1
Arijit Chakrabarti and Jayanta K. Ghosh, A generalization of BIC for the general exponential family, J. Statist. Plann. Inference 136 (2006), no. 9, 2847–2872. MR 2281234, DOI 10.1016/j.jspi.2005.01.005
Bertrand S. Clarke and Andrew R. Barron, Information-theoretic asymptotics of Bayes methods, IEEE Trans. Inform. Theory 36 (1990), no. 3, 453–471. MR 1053841, DOI 10.1109/18.54897
Bertrand S. Clarke and Andrew R. Barron, Jeffreys’ prior is asymptotically least favorable under entropy risk, J. Statist. Plann. Inference 41 (1994), no. 1, 37–60. MR 1292146, DOI 10.1016/0378-3758(94)90153-8
Martin Crowder, Asymptotic expansions of posterior expectations, distributions and densities for stochastic processes, Ann. Inst. Statist. Math. 40 (1988), no. 2, 297–309. MR 947362, DOI 10.1007/BF00052346
Gauri Sankar Datta and Rahul Mukerjee, Probability matching priors: higher order asymptotics, Lecture Notes in Statistics, vol. 178, Springer-Verlag, New York, 2004. MR 2053794, DOI 10.1007/978-1-4612-2036-7
Thomas J. DiCiccio, Robert E. Kass, Adrian Raftery, and Larry Wasserman, Computing Bayes factors by combining simulation and asymptotic approximations, J. Amer. Statist. Assoc. 92 (1997), no. 439, 903–915. MR 1482122, DOI 10.2307/2965554
Thomas J. DiCiccio and Steven E. Stern, On Bartlett adjustments for approximate Bayesian inference, Biometrika 80 (1993), no. 4, 731–740. MR 1282782, DOI 10.1093/biomet/80.4.731
Sylvia Frühwirth-Schnatter, Finite mixture and Markov switching models, Springer Series in Statistics, Springer, New York, 2006. MR 2265601
Dani Gamerman and Hedibert Freitas Lopes, Markov chain Monte Carlo, 2nd ed., Texts in Statistical Science Series, Chapman & Hall/CRC, Boca Raton, FL, 2006. Stochastic simulation for Bayesian inference. MR 2260716, DOI 10.1201/9781482296426
John Geweke, Bayesian inference in econometric models using Monte Carlo integration, Econometrica 57 (1989), no. 6, 1317–1339. MR 1035115, DOI 10.2307/1913710
J. K. Ghosh, Higher Order Asymptotics, NSF-CBMS Regional Conf. Ser. Probab. Statist., 4, Inst. Math. Statist., Hayward, CA, 1994.
Jayanta K. Ghosh, Mohan Delampady, and Tapas Samanta, An introduction to Bayesian analysis, Springer Texts in Statistics, Springer, New York, 2006. Theory and methods. MR 2247439
J. K. Ghosh and Rahul Mukerjee, Characterization of priors under which Bayesian and frequentist Bartlett corrections are equivalent in the multiparameter case, J. Multivariate Anal. 38 (1991), no. 2, 385–393. MR 1131727, DOI 10.1016/0047-259X(91)90052-4
J. K. Ghosh and Rahul Mukerjee, Bayesian and frequentist Bartlett corrections for likelihood ratio and conditional likelihood ratio tests, J. Roy. Statist. Soc. Ser. B 54 (1992), no. 3, 867–875. MR 1185228, DOI 10.1111/j.2517-6161.1992.tb01458.x
J. K. Ghosh and R. V. Ramamoorthi, Bayesian nonparametrics, Springer Series in Statistics, Springer-Verlag, New York, 2003. MR 1992245
J. K. Ghosh, B. K. Sinha, and S. N. Joshi, Expansions for posterior probability and integrated Bayes risk, Statistical decision theory and related topics, III, Vol. 1 (West Lafayette, Ind., 1981) Academic Press, New York-London, 1982, pp. 403–456. MR 705299
Chantal Guihenneuc-Jouyaux and Judith Rousseau, Laplace expansions in Markov chain Monte Carlo algorithms, J. Comput. Graph. Statist. 14 (2005), no. 1, 75–94. MR 2137891, DOI 10.1198/106186005X25727
Atsushi Inoue and Mototsugu Shintani, Quasi-Bayesian model selection, Quant. Econ. 9 (2018), no. 3, 1265–1297. MR 3891567, DOI 10.3982/QE587
R. A. Johnson, An asymptotic expansion for posterior distributions, Ann. Math. Statist. 38 (1967), 1899–1906. MR 219161, DOI 10.1214/aoms/1177698624
Richard A. Johnson, Asymptotic expansions associated with posterior distributions, Ann. Math. Statist. 41 (1970), 851–864. MR 263198, DOI 10.1214/aoms/1177696963
R. E. Kass, L. Tierney, and J. B. Kadane, The validity of posterior expansions based on Laplace’s method, Bayesian and Likelihood Methods in Statistics and Econometrics: Essays in Honor of Georrge A. Barnard, (eds. S. Geisser, J. S. Hodges, S. J. Press, and A. Zellner), Stud. Bayesian Econometrics Statist., 7, North-Holland, Amsterdam, 1990, pp.473–488.
Sadanori Konishi, Tomohiro Ando, and Seiya Imoto, Bayesian information criteria and smoothing parameter selection in radial basis function networks, Biometrika 91 (2004), no. 1, 27–43. MR 2050458, DOI 10.1093/biomet/91.1.27
Dennis Kristensen and Oliver Linton, A closed-form estimator for the Garch $(1,1)$ model, Econometric Theory 22 (2006), no. 2, 323–337. MR 2230392, DOI 10.1017/S0266466606060142
P. S. De. Laplace, Oeuvres de Laplace, 7, Imprimerie Royale, Paris, 1847.
T. Leonard, Comment on ’A simple predictive density function’, J. Amer. Statist. Assoc., 77, (1982), 657–685.
D. V. Lindley, The use of prior probability distributions in statistical inference and decisions, Proc. 4th Berkeley Sympos. Math. Statist. and Prob., Vol. I, Univ. California Press, Berkeley, Calif., 1961, pp. 453–468. MR 0156437
D. V. Lindley, Approximate Bayesian methods, Bayesian statistics (Valencia, 1979) Univ. Press, Valencia, 1980, pp. 223–245. With discussion. MR 638879
S. Lipschutz, Theory and problems of discrete mathematics, Tokyo, Ohmsha, 1984 (Translated into Japanese by H. Narishima).
Peter McCullagh, Tensor methods in statistics, Monographs on Statistics and Applied Probability, Chapman & Hall, London, 1987. MR 907286
P. McCullagh and J. A. Nelder, Generalized linear models, Monographs on Statistics and Applied Probability, Chapman & Hall, London, 1989. Second edition [of MR0727836]. MR 3223057, DOI 10.1007/978-1-4899-3242-6
Alexander J. McNeil, Rüdiger Frey, and Paul Embrechts, Quantitative risk management, Princeton Series in Finance, Princeton University Press, Princeton, NJ, 2005. Concepts, techniques and tools. MR 2175089
Yoichi Miyata, Fully exponential Laplace approximations using asymptotic modes, J. Amer. Statist. Assoc. 99 (2004), no. 468, 1037–1049. MR 2109493, DOI 10.1198/016214504000001673
Yoichi Miyata, Higher order expansions for posterior distributions using posterior modes, J. Japan Statist. Soc. 38 (2008), no. 3, 415–429. MR 2510947, DOI 10.14490/jjss.38.415
Yoichi Miyata, Laplace approximations to means and variances with asymptotic modes, J. Statist. Plann. Inference 140 (2010), no. 2, 382–392. MR 2558371, DOI 10.1016/j.jspi.2009.06.015
Yoichi Miyata, Laplace approximations using $n^\alpha$-consistent estimators, J. Statist. Plann. Inference 194 (2018), 25–31. MR 3725786, DOI 10.1016/j.jspi.2017.10.003
Helen Ogden, On the error in Laplace approximations of high-dimensional integrals, Stat 10 (2021), Paper No. e380, 13. MR 4276039, DOI 10.1002/sta4.380
P. C. B. Phillips, Marginal densities of instrumental variable estimators in the general single equation case. Adv. in Econom., 2, (1983), 1-24.
Dimitris Rizopoulos, Geert Verbeke, and Emmanuel Lesaffre, Fully exponential Laplace approximations for the joint modelling of survival and longitudinal data, J. R. Stat. Soc. Ser. B Stat. Methodol. 71 (2009), no. 3, 637–654. MR 2749911, DOI 10.1111/j.1467-9868.2008.00704.x
Gideon Schwarz, Estimating the dimension of a model, Ann. Statist. 6 (1978), no. 2, 461–464. MR 468014
Zhenming Shun and Peter McCullagh, Laplace approximation of high-dimensional integrals, J. Roy. Statist. Soc. Ser. B 57 (1995), no. 4, 749–760. MR 1354079, DOI 10.1111/j.2517-6161.1995.tb02060.x
Trevor J. Sweeting, A framework for Bayesian and likelihood approximations in statistics, Biometrika 82 (1995), no. 1, 1–23. MR 1332836, DOI 10.1093/biomet/82.1.1
T. J. Sweeting, Approximate Bayesian computation based on signed roots of log-density ratios, Bayesian statistics, 5 (Alicante, 1994) Oxford Sci. Publ., Oxford Univ. Press, New York, 1996, pp. 427–444. MR 1425418
Trevor J. Sweeting and Samer A. Kharroubi, Some new formulae for posterior expectations and Bartlett corrections, Test 12 (2003), no. 2, 497–521. MR 2044322, DOI 10.1007/BF02595727
Luke Tierney and Joseph B. Kadane, Accurate approximations for posterior moments and marginal densities, J. Amer. Statist. Assoc. 81 (1986), no. 393, 82–86. MR 830567, DOI 10.1080/01621459.1986.10478240
Luke Tierney, Robert E. Kass, and Joseph B. Kadane, Fully exponential Laplace approximations to expectations and variances of nonpositive functions, J. Amer. Statist. Assoc. 84 (1989), no. 407, 710–716. MR 1132586, DOI 10.1080/01621459.1989.10478824
Abraham Wald, Note on the consistency of the maximum likelihood estimate, Ann. Math. Statistics 20 (1949), 595–601. MR 32169, DOI 10.1214/aoms/1177729952
S. Watanabe, Daisu Kika to Gakushu Riron, Morikita Shuppan, 2006 (in Japanese).
B. L. Welch and H. W. Peers, On formulae for confidence points based on integrals of weighted likelihoods, J. Roy. Statist. Soc. Ser. B 25 (1963), 318–329. MR 173309, DOI 10.1111/j.2517-6161.1963.tb00512.x