A structure-preserving kernel method for learning Hamiltonian systems

Hu, Jianyu; Ortega, Juan-Pablo; Yin, Daiying

doi:10.1090/mcom/4106

A structure-preserving kernel method for learning Hamiltonian systems
HTML articles powered by AMS MathViewer

by Jianyu Hu, Juan-Pablo Ortega and Daiying Yin;

Math. Comp.

DOI: https://doi.org/10.1090/mcom/4106

Published electronically: June 16, 2025

PDF | Request permission

Abstract:

A structure-preserving kernel ridge regression method is presented that allows the recovery of nonlinear Hamiltonian functions out of datasets made of noisy observations of Hamiltonian vector fields. The method proposes a closed-form solution that yields excellent numerical performances that surpass other techniques proposed in the literature in this setup. From the methodological point of view, the paper extends kernel regression methods to problems in which loss functions involving linear functions of gradients are required and, in particular, a differential reproducing property and a Representer Theorem are proved in this context. The relation between the structure-preserving kernel estimator and the Gaussian posterior mean estimator is analyzed. A full error analysis is conducted that provides convergence rates using fixed and adaptive regularization parameters. The good performance of the proposed estimator together with the convergence rate is illustrated with various numerical experiments.

References

R. Abraham and J. E. Marsden, Foundations of Mechanics, American Mathematical Soc., (364), 2008.
V. I. Arnol′d, Mathematical methods of classical mechanics, Graduate Texts in Mathematics, vol. 60, Springer-Verlag, New York-Heidelberg, 1978. Translated from the Russian by K. Vogtmann and A. Weinstein. MR 690288
N. Aronszajn, Theory of reproducing kernels, Trans. Amer. Math. Soc. 68 (1950), 337–404. MR 51437, DOI 10.1090/S0002-9947-1950-0051437-7
Viorel Barbu, Differential equations, Springer Undergraduate Mathematics Series, Springer, Cham, 2016. Translated from the 1985 Romanian original by Liviu Nicolaescu. MR 3585801, DOI 10.1007/978-3-319-45261-6
Frank Bauer, Sergei Pereverzev, and Lorenzo Rosasco, On regularization algorithms in learning theory, J. Complexity 23 (2007), no. 1, 52–72. MR 2297015, DOI 10.1016/j.jco.2006.07.001
Martin Benning and Martin Burger, Modern regularization methods for inverse problems, Acta Numer. 27 (2018), 1–111. MR 3826506, DOI 10.1017/s0962492918000016
Tom Bertalan, Felix Dietrich, Igor Mezić, and Ioannis G. Kevrekidis, On learning Hamiltonian systems from data, Chaos 29 (2019), no. 12, 121107, 9. MR 4045468, DOI 10.1063/1.5128231
Jake Bouvrie and Boumediene Hamzi, Kernel methods for the approximation of nonlinear systems, SIAM J. Control Optim. 55 (2017), no. 4, 2460–2492. MR 3683690, DOI 10.1137/14096815X
Jake Bouvrie and Boumediene Hamzi, Kernel methods for the approximation of some key quantities of nonlinear systems, J. Comput. Dyn. 4 (2017), no. 1-2, 1–19. MR 3808250, DOI 10.3934/jcd.2017001
Martin Burger and Stanley Osher, Convergence rates of convex variational regularization, Inverse Problems 20 (2004), no. 5, 1411–1421. MR 2109126, DOI 10.1088/0266-5611/20/5/005
A. Caponnetto and E. De Vito, Fast rates for regularized least-squares algorithm, Tech. Rep., MIT, 2005.
Z. Chen, J. Zhang, M. Arjovsky, and L. Bottou, Symplectic recurrent neural networks, Preprint, arXiv:1909.13334, 2019.
R. Chen and M. Tao, Data-driven prediction of general Hamiltonian dynamics via learning exactly-symplectic maps, International Conference on Machine Learning, PMLR, 2021, pp. 1717–1727.
Xiaoli Chen, Jinqiao Duan, Jianyu Hu, and Dongfang Li, Data-driven method to learn the most probable transition pathway and stochastic differential equation, Phys. D 443 (2023), Paper No. 133559, 15. MR 4503825, DOI 10.1016/j.physd.2022.133559
Pascal Chossat, Debra Lewis, Juan-Pablo Ortega, and Tudor S. Ratiu, Bifurcation of relative equilibria in mechanical systems with symmetry, Adv. in Appl. Math. 31 (2003), no. 1, 10–45. MR 1985819, DOI 10.1016/S0196-8858(02)00503-1
A. Christmann and I. Steinwart, Support Vector Machines. Springer, 2008.
John B. Conway, A course in functional analysis, 2nd ed., Graduate Texts in Mathematics, vol. 96, Springer-Verlag, New York, 1990. MR 1070713
M. Cranmer, S. Greydanus, S. Hoyer, P. Battaglia, D. Spergel, and S. Ho, Lagrangian neural networks, Preprint, arXiv:2003.04630, 2020.
S. Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli, Scientific machine learning through physics–informed neural networks: where we are and what’s next, J. Scientific Comput. 92 (2022), no. 3, 88.
G. Dal Maso, An Introduction to $\Gamma$-convergence, Springer Science and Business Media, 1993.
Marco David and Florian Méhats, Symplectic learning for Hamiltonian neural networks, J. Comput. Phys. 494 (2023), Paper No. 112495, 13. MR 4652800, DOI 10.1016/j.jcp.2023.112495
Ernesto De Vito, Lorenzo Rosasco, Andrea Caponnetto, Umberto De Giovannini, and Francesca Odone, Learning from examples as an inverse problem, J. Mach. Learn. Res. 6 (2005), 883–904. MR 2249842
H. W. Engl, M. Hanke, and A. Neubauer, Regularization of Inverse Problems, Vol. 375, Springer Science and Business Media, 1996.
Jinchao Feng, Charles Kulick, Yunxiang Ren, and Sui Tang, Learning particle swarming models from data with Gaussian processes, Math. Comp. 93 (2024), no. 349, 2391–2437. MR 4759379, DOI 10.1090/mcom/3915
José C. Ferreira and Valdir A. Menegatto, Reproducing properties of differentiable Mercer-like kernels, Math. Nachr. 285 (2012), no. 8-9, 959–973. MR 2928393, DOI 10.1002/mana.201100072
L. Gonon, L. Grigoryeva, and J.-P. Ortega, Reservoir kernels and Volterra series, arXiv:2212.14641, 2022.
S. Greydanus, M. Dzamba, and J. Yosinski, Hamiltonian neural networks, Advances in Neural Information Processing Systems, vol. 32, 2019.
Boumediene Hamzi and Houman Owhadi, Learning dynamical systems from data: a simple cross-validation perspective, part I: Parametric kernel flows, Phys. D 421 (2021), Paper No. 132817, 10. MR 4233447, DOI 10.1016/j.physd.2020.132817
C.-D. Han, B. Glaz, M. Haile, and Y.-C. Lai, Adaptable Hamiltonian neural networks, Phys. Rev. Res. 3 (2021), no. 2, 023156.
Michiel Hermans and Benjamin Schrauwen, Recurrent kernel machines: computing with infinite echo state networks, Neural Comput. 24 (2012), no. 1, 104–133. MR 2907166, DOI 10.1162/NECO_{a}_{0}0200
Lars Hörmander, The analysis of linear partial differential operators. III, Classics in Mathematics, Springer, Berlin, 2007. Pseudo-differential operators; Reprint of the 1994 edition. MR 2304165, DOI 10.1007/978-3-540-49938-1
B. Hou, S. Sanjari, N. Dahlin, S. Bose, and U. Vaidya, Sparse learning of dynamical systems in RKHS: an operator-theoretic approach, Proceedings of the 40th International Conference on Machine Learning, PMLR, July 23–29, 2023, pp. 13325–13352.
F. Huszár and D. Duvenaud, Optimally-weighted herding is Bayesian quadrature, Preprint, arXiv:1204.1664, 2012.
P. Jin, Z. Zhang, A. Zhu, Y. Tang, and G. E. Karniadakis, SympNets: intrinsic structure-preserving symplectic networks for identifying Hamiltonian systems, Neural Netw. 132 (2020), 166–179.
M. Kanagawa, P. Hennig, D. Sejdinovic, and B. K. Sriperumbudur, Gaussian processes and kernel methods: a review on connections and equivalences, Preprint, arXiv:1807.02582, 2018.
George S. Kimeldorf and Grace Wahba, A correspondence between Bayesian estimation on stochastic processes and smoothing by splines, Ann. Math. Statist. 41 (1970), 495–502. MR 254999, DOI 10.1214/aoms/1177697089
A. Lederer, J. Umlauft, and S. Hirche, Uniform error bounds for Gaussian process regression with application to safe control, Advances in Neural Information Processing Systems, vol. 32, 2019.
Tzon-Tzer Lu and Sheng-Hua Shiou, Inverses of $2\times 2$ block matrices, Comput. Math. Appl. 43 (2002), no. 1-2, 119–129. MR 1873248, DOI 10.1016/S0898-1221(01)00278-4
Fei Lu, Ming Zhong, Sui Tang, and Mauro Maggioni, Nonparametric inference of interaction laws in systems of agents from trajectory data, Proc. Natl. Acad. Sci. USA 116 (2019), no. 29, 14424–14433. MR 3984488, DOI 10.1073/pnas.1822012116
J. E. Marsden and M. West, Discrete mechanics and variational integrators, Acta Numer. 10 (2001), 357–514. MR 2009697, DOI 10.1017/S096249290100006X
Jerrold E. Marsden and Tudor S. Ratiu, Introduction to mechanics and symmetry, 2nd ed., Texts in Applied Mathematics, vol. 17, Springer-Verlag, New York, 1999. A basic exposition of classical mechanical systems. MR 1723696, DOI 10.1007/978-0-387-21792-5
Charles A. Micchelli, Yuesheng Xu, and Haizhang Zhang, Universal kernels, J. Mach. Learn. Res. 7 (2006), 2651–2667. MR 2274454
Ha Quang Minh, Some properties of Gaussian reproducing kernel Hilbert spaces and their implications for function approximation and learning theory, Constr. Approx. 32 (2010), no. 2, 307–338. MR 2677883, DOI 10.1007/s00365-009-9080-0
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar, Foundations of machine learning, Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, 2012. MR 3057769
C. H. Nadungodage, Y. Xia, F. Li, J. J. Lee, and J. Ge, StreamFitter: a real time linear regression analysis system for continuous data streams, Database Systems for Advanced Applications, Springer Berlin Heidelberg, Berlin, Heidelberg, 2011, pp. 458–461.
Erich Novak, Mario Ullrich, Henryk Woźniakowski, and Shun Zhang, Reproducing kernels of Sobolev spaces on $\Bbb R^d$ and applications to embedding constants and tractability, Anal. Appl. (Singap.) 16 (2018), no. 5, 693–715. MR 3850092, DOI 10.1142/S0219530518500094
A. O’Hagan, Bayes-Hermite quadrature, J. Statist. Plann. Inference 29 (1991), no. 3, 245–260. MR 1144171, DOI 10.1016/0378-3758(91)90002-V
Juan-Pablo Ortega and Tudor S. Ratiu, Momentum maps and Hamiltonian reduction, Progress in Mathematics, vol. 222, Birkhäuser Boston, Inc., Boston, MA, 2004. MR 2021152, DOI 10.1007/978-1-4757-3811-7
Juan-Pablo Ortega, Víctor Planas-Bielsa, and Tudor S. Ratiu, Asymptotic and Lyapunov stability of constrained and Poisson equilibria, J. Differential Equations 214 (2005), no. 1, 92–127. MR 2143513, DOI 10.1016/j.jde.2004.09.016
Juan-Pablo Ortega and Daiying Yin, Learnability of linear port-Hamiltonian systems, J. Mach. Learn. Res. 25 (2024), Paper No. [68], 56. MR 4749104
Houman Owhadi and Clint Scovel, Separability of reproducing kernel spaces, Proc. Amer. Math. Soc. 145 (2017), no. 5, 2131–2138. MR 3611326, DOI 10.1090/proc/13354
M. Pförtner, I. Steinwart, P. Hennig, and J. Wenger, Physics-informed Gaussian process regression generalizes linear PDE solvers, Preprint, arXiv:2212.12474, 2022.
Robert Plato, Peter Mathé, and Bernd Hofmann, Optimal rates for Lavrentiev regularization with adjoint source conditions, Math. Comp. 87 (2018), no. 310, 785–801. MR 3739217, DOI 10.1090/mcom/3237
Alberto Racca and Luca Magri, Automatic-differentiated Physics-Informed Echo State Network (API-ESN), Computational science—ICCS 2021. Part V, Lecture Notes in Comput. Sci., vol. 12746, Springer, Cham, [2021] ©2021, pp. 323–329. MR 4371823, DOI 10.1007/978-3-030-77977-1_{2}5
M. Raissi, P. Perdikaris, and G. E. Karniadakis, Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys. 378 (2019), 686–707. MR 3881695, DOI 10.1016/j.jcp.2018.10.045
Mark Rudelson and Roman Vershynin, Hanson-Wright inequality and sub-Gaussian concentration, Electron. Commun. Probab. 18 (2013), no. 82, 9. MR 3125258, DOI 10.1214/ECP.v18-2865
Robert Schaback and Holger Wendland, Kernel techniques: from machine learning to meshless methods, Acta Numer. 15 (2006), 543–639. MR 2269747, DOI 10.1017/S0962492906270016
A. P. Buslaev and O. V. Seleznjev, On certain extremal problems in the theory of approximation of random processes, East J. Approx. 5 (1999), no. 4, 467–481. MR 1738482
Yeonjong Shin, Jérôme Darbon, and George Em Karniadakis, On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEs, Commun. Comput. Phys. 28 (2020), no. 5, 2042–2074. MR 4188529, DOI 10.4208/cicp.oa-2020-0193
Ingo Steinwart, On the influence of the kernel on the consistency of support vector machines, J. Mach. Learn. Res. 2 (2002), no. 1, 67–93. MR 1883281, DOI 10.1162/153244302760185252
Michael Stein, Uniform asymptotic optimality of linear predictions of a random field using an incorrect second-order structure, Ann. Statist. 18 (1990), no. 2, 850–872. MR 1056340, DOI 10.1214/aos/1176347629
Artur P. Toshev, Gianluca Galletti, Johannes Brandstetter, Stefan Adami, and Nikolaus A. Adams, Learning Lagrangian fluid mechanics with $\rm E(3)$-equivariant graph neural networks, Geometric science of information. Part II, Lecture Notes in Comput. Sci., vol. 14072, Springer, Cham, [2023] ©2023, pp. 332–341. MR 4656142, DOI 10.1007/978-3-031-38299-4_{3}5
P. Toth, D. J. Rezende, A. Jaegle, S. Racanière, A. Botev, and I. Higgins, Hamiltonian generative networks, Preprint, arXiv:1909.13789, 2019.
Rui Tuo and Wenjia Wang, Kriging prediction with isotropic Matérn correlations: robustness and experimental designs, J. Mach. Learn. Res. 21 (2020), Paper No. 187, 38. MR 4209473
R. Valperga, K. Webster, D. Turaev, V. Klein, and J. Lamb, Learning reversible symplectic dynamics, Learning for Dynamics and Control Conference, PMLR, 2022, pp. 906–916.
Aad van der Vaart and Harry van Zanten, Information rates of nonparametric Gaussian process methods, J. Mach. Learn. Res. 12 (2011), 2095–2119. MR 2819028
Steven Van Vaerenbergh and Ignacio Santamaría, Online regression with kernels, Regularization, optimization, kernels, and support vector machines, Chapman & Hall/CRC Mach. Learn. Pattern Recogn. Ser., CRC Press, Boca Raton, FL, 2015, pp. 477–501. MR 3380651
Miguel Vaquero, Jorge Cortés, and David Martín de Diego, Symmetry preservation in Hamiltonian systems: simulation and learning, J. Nonlinear Sci. 34 (2024), no. 6, Paper No. 115, 29. MR 4813269, DOI 10.1007/s00332-024-10089-5
Wenjia Wang and Bing-Yi Jing, Gaussian process regression: optimality, robustness, and relationship with kernel ridge regression, J. Mach. Learn. Res. 23 (2022), Paper No. [193], 67. MR 4577146
Carl Edward Rasmussen and Christopher K. I. Williams, Gaussian processes for machine learning, Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, 2006. MR 2514435
S. J. Yakowitz and F. Szidarovszky, A comparison of Kriging with nonparametric regression methods, J. Multivariate Anal. 16 (1985), no. 1, 21–53. MR 778488, DOI 10.1016/0047-259X(85)90050-8
X. Yang, G. Tartakovsky, and A. Tartakovsky. Physics-informed kriging: A physics-informed Gaussian process regression method for data-model convergence, 2018, arXiv preprint, arXiv:1809.03461.
Vadim Yurinsky, Sums and Gaussian vectors, Lecture Notes in Mathematics, vol. 1617, Springer-Verlag, Berlin, 1995. MR 1442713, DOI 10.1007/BFb0092599
Zhen Zhang, Yeonjong Shin, and George Em Karniadakis, GFINNs: GENERIC formalism informed neural networks for deterministic and stochastic dynamical systems, Philos. Trans. Roy. Soc. A 380 (2022), no. 2229, Paper No. 20210207, 21. MR 4460492, DOI 10.1098/rsta.2021.0207
Ding-Xuan Zhou, Derivative reproducing properties for kernel methods in learning theory, J. Comput. Appl. Math. 220 (2008), no. 1-2, 456–463. MR 2444183, DOI 10.1016/j.cam.2007.08.023

AMS Website Down

Mathematics of Computation

A structure-preserving kernel method for learning Hamiltonian systems
HTML articles powered by AMS MathViewer

Abstract: